Communication in Support of Software Sharingand Collaborative Development
Martin J. KnottArgonne National Laboratory
Argonne, Illinois 60439
ABSTRACT
To be successful, software which is shared among several users requires a means of reporting trouble andreceiving help. This is even more critical in the case of a collaborative software development effort. The
Experimental Physics and Industrial Control System (EPICS) collaboration uses the Internet as its major
communication medium.
In addition to conventional electronic mail and occasional use of MBONE
teleconferencing [1], a distributed listserver system is used to announce releases, ask for aid, announce thediscovery and disposal of bugs, and to converse generally about the future development directions of EPICS toolsand methods. The EPICS listservers are divided into several subject categories, and since all questions, answers,and announcements are archived for future reference, some statistics can be gleaned from these records. Suchstatistics and information from the collaborators show that they make use of this system and find it helpful. As amanager, I have found that the system gives reassuring evidence that the collaboration is alive, responsive to callsfor aid, and helpful even to those not actively participating in the question and answer activity.

THE EPICS COLLABORATIONThe collaboration is now six years old and while recently the number of facilities using EPICS has grown
rapidly, most development of new tools and all development of the "core" software and central tools still resides infour laboratories: Argonne National Laboratory (ANL), Los Alamos National Laboratory (LANL), the Continuous
Electron Beam Accelerator Facility (CEBAF), and Lawrence Berkeley Laboratory (LBL). In addition, due to thetool-based nature of EPICS, application-specific tools are being developed at several other sites [2]. Figure 1
shows the worldwide nature of EPICS, which is now used at about 50 locations including laboratories, universities,and industries.

.creighton.edu.anl.gov.iit.edu.nwu.edu.uchicago.edu.umich.edu.uiuc.edu.kscorp.com.lbl.gov.caltech.edu.stanford.edu.kent.edu.mit.edu.bnl.gov.desy.de.bessy.de.mpg.de.ariadne-t.gr.ihep.su.roe.ac.uk.cam.ac.uk.cea.fr.kek.jp.isas.ac.jp.gemini.edu.noao.edu.cebaf.gov.tate.com.lanl.gov.duke.edu.wpafb.af.mil.keck.hawaii.edu.jach.hawaii.edu

Figure 1 World map showing the current EPICS collaborators and users.

.ssrc.gov.tw

Virtually all EPICS documentation is posted to servers on the World Wide Web (WWW), enabling instant,worldwide revision and availability. Although EPICS documentation is extensive and quite good when comparedwith that normally produced for a control system developed and used at a single laboratory, it is not extensive
enough to provide total guidance to a tool developer. Nor is it tailored to the point that it can help troubleshootersin tracking down a bug.
Like any multi-site development collaboration, EPICS depends heavily on convenient communications. Since
EPICS is available for use in non-profit (and largely public domain) organizations, the collaboration is free fromconcerns about protection of proprietary information. From the beginning of the collaboration, the benefits of fully
open communication were realized, and so concerns about unfavorable publicity generated when a bug isdiscovered are also absent.
One feature of EPICS that makes it an attractive alternative to software developed in-house, or even heavilysupported and high quality commercial software, is the availability of the source code along with one or moreexperts on the deep internals of the code. These experts can solve most problems quickly and when a work-aroundor correction is developed, the new code can be posted immediately for general usage and then incorporated into thenext release.
All of the above factors make communication by open exchange via e-mail exploders, on-line documentation,and non-secure MBONE teleconferencing very attractive. These methods are economical, fast, and universallyavailable worldwide.

COMMUNICATION METHODS
Although MBONE, the workstation-based teleconferencing system based on slow-scan video and multicast
network packets, has been used for a few collaboration meetings and some single-topic few-person meetings, thismethod is still somewhat of a novelty, but does show promise.
As implied above, the principle communication methods are various forms of e-mail. Even among thoseworking on related tasks, telephone calls are rare and only used when those involved need that special "face-toface" exchange of ideas. Even point-to-point e-mail is somewhat rare since most questions or requests forassistance are initially thrown out to the largest audience.
By a large margin, the most used and usefulcommunication method is that provided by listservers or "e-mail exploders."

LISTSERVERS
There are six listservers related to EPICS work: four at ANL, one at CEBAF, and one operated at theUniversity of Hawaii for the telescope users. The names, functions, and number of entries for the ANL listservers
as of early October 1995 are as follows:
EPICS_Applications (574 entries) -- Descriptions of applications using EPICS, the purpose being to offer
EPICS solutions that have been developed. Also included are methods used to solve tricky problems. Often,
EPICS can be used to solve a problem in multiple ways, and this forum offers the benefit of experience tonewcomers.
EPICS_Suggestions (61 entries) -- As the title implies, this listserver collects suggestions for future upgradesor corrections to current methods.
Tech_Talk (957 entries) -- This listserver provides a forum for discussion of problems, bugs, future
development directions, and sometimes serves as an e-mail "round table" for open discussion and argument.
Beamline_Controls (46 entries) -- A listserver dedicated to those who use EPICS for control of synchrotron
radiation beamlines. Actually, a fortunate cross-fertilization has developed between members of two of thecommunities that subscribe to this listserver—synchrotron radiation experimenters and builders of astronomical
telescopes—since both use CCD image capture devices and lots of stepping motors.
The first three listservers were started in 1992 when the EPICS collaboration grew with the addition of twonew collaborators, CEBAF and LBL; and the Beamline_Controls listserver was added in May 1995 when the APS
users began getting organized. According to the total number of entries and the current rate of activity, the mostused listserver is obviously Tech_Talk. I have studied and extracted information from the archive of this listserver
to learn how it is used and how responsive the collaboration is to requests for help and advice (see below).

Listserver Archives
To provide for future reference for trouble reports and advice given, archives are maintained.
Web-based ArchiveAll Argonne-based listservers are archived and can be accessed from the WWW [3]. This archiving system
was set up in early 1994 when the volume of listserver entries began a period of explosive growth. In keeping withthe spirit of completely open and public disclosure, these archives are not restricted to collaborators and can beaccessed freely worldwide. The archive system produces a hyperlink list of each day's entries (in real time) so that
rapidly answered queries and their answers are displayed on the same page. As soon as an entry is received for thenext month, a new hyperlink is created to reference the previous month's (relocated) hyperlink list. All of this work
is done automatically by clocked scripts.
The month-long hyperlink entries include the subject, date/time, and the originator's e-mail address so that a
visual search can usually locate a recent topic in a short time. A search tool is provided to help in locating a topic
or reference by searching for a specific string or for strings with logical constructs.
This archive is used principally by new EPICS users searching for a problem solution that may apply to theirsituation. Even so, it takes a while to get accustomed to the nomenclature and EPICS-related terms andmethodology, and the archives serve more as repositories of recent help requests, trouble reports, and thecorresponding responses. The fact that virtually all such calls for help are responded to does give confidence toprospective EPICS users that help will be available when needed.
NOTES-based Archive
At Argonne an additional archiving system, used for several controls and administrative notification
categories, is used to collect, index, and organize the listserver entries. This software, intended for automatic
handling of e-mail, can be set to send short e-mail alerts to a list of local usernames listing the topic, date/time,subject, and originator. Pull-down menus bring up a window which lists the latest 13 entries and adds an entry
number to the previous basic information. It also indicates the number of entries with the same subject if multiple
such entries are received. Keying in the entry number will display the original e-mail with a compact and uniform
heading. Single-key commands are used to read, scan, and search for author or subject.
Private E-mail Folders
In spite of the convenience offered by the above two methods, most active EPICS software experts use theirown e-mail folders to keep track of active help requests. The above archives are used occasionally to find an old
trouble reference, but the visibility of e-mailed trouble reports among their own e-mail is viewed as a better
reminder of the uncompleted tasks.

UTILIZATION STATISTICS FOR TECH_TALKThe popularity and long running period of the Tech_Talk listserver makes it the best subject for study. It isused for several purposes: trouble reports and the corresponding solutions, notifications, requests for advice on
EPICS techniques and hardware component selection, and open discussions of development directions. The largest
category of entries was requests for help or clarification regarding EPICS itself. Another common message type isquestions concerning hardware devices supported (or not supported) by EPICS. Some devices are quite
complicated and the experience of others who have solved the sometimes tricky installation and programming
issues of these devices can be of great help to others. A third common type of message is requests for advice on
which hardware devices to purchase in order to address a particular class of problems. The result is that theTech_Talk listserver is used to address a number of control system issues only incidentally related to the EPICS
software collaboration.
Enrollment
The enrollment breakdown, including the number of subscribers at each location, is as follows:Principal EPICS development locations:
ANL
17CEBAF
20
LANL
20LBL
11
Other EPICS user locations:
DESY
SaclayU. of Athens, Greece
IHEP
Gemini Observatory
CaltechUniversity of Michigan
Northwestern University
U. of Ill. at Chicago
Kent University
Max Planck Institute6214211221
1BESSY
Cambridge Observatory
KEKNOAO
University of Hawaii
Duke University
MITUniversity of Chicago
Stanford University
Creighton University

1
102
44
34382

Added to these are 16 "observer" locations and 5 commercial locations. These statistics indicate that the four
principal development labs have relatively large staffs who participate in this listserver, whereas the otherparticipant locations have relatively small implementation teams. This participant ratio gives evidence that, with asmall team of implementers, advantage can be taken of a well-developed and supported set of control systemsoftware.

60

End of this data - September 1995

50Entries / MonthSeveral EPICS users added
February 199440

30

Tech_Talk starts
September 1992

20

1001992

1993

1994

1995

Figure 2 Usage history of the EPICS Tech_Talk listserver.
Usage History
Figure 2 shows the growth of Tech_Talk traffic from its inception in September 1992 through September
1995. The jump in traffic in early 1994 corresponded to several new EPICS users being added to the collaboration.

I have taken the entries beginning in August 1994 and extracted information concerning the type of messages andthe time elapsed until responses are seen.

Response TimeNot answered
3 days - long
1 day - 3 days
8 hrs. - 1 day
1 - 8 hrs.
30 min. - 1 hr.
15 - 30 min.
0 - 15 min.
010 20 30 40 50 60 70 80 90 100

Tech_Talk Queries
Figure 3 Response time distribution for Tech_Talk queries.
Response TimeAn important measure for the collaboration to consider is the time elapsed between the request for help andthe posting of the answer. Figure 3 shows how the responses to such requests fall into various time slots. The binsize (horizontal scale) is non-linear and is intended to show how the response time might be viewed by therequestor. It shows that most responses are provided within one day of the request with a substantial number within
the same work shift or even sooner. With the collaborators spread around the world the way they are, there is avery real time-zone effect with many responses having to wait until the responding site sees the request in themorning. Another, similar, effect can be seen by the number of requests that wait over a weekend to receive aresponse. In fact, large time-zone differences can make telephone-based help difficult. The heavy use of e-mailand the listserver does tend, however, to mask the international and time-zone differences.
The "not answered" bin may seem well filled with nearly 40 entries. However, many of these are addressed"out of band" with e-mail or telephone conversations serving to gain additional detail. Interviews with those whohandle many of the help requests at Argonne indicate that this is sometimes the case, although the vast majority arehandled via the listserver.

ORGANIZATION
It must be pointed out that the EPICS collaboration is almost organization-free; that is, no formal set of rules
or procedures governs how the collaboration develops or how the work is assigned. (The development work isposted and tracked [4] so that the developers can be sent suggestions.) The reason the collaboration works, in myopinion, is because all of the involved personnel have a desire to help their counterparts at other locations, and there
is a certain "pride of ownership" involved. The listservers are a reflection of this same philosophy—there is nocentral organization or assignment. The questions are seen by all and the first user who is able to address the issue
does so. Often, this sparks additional solutions and comments from the other listeners.

BENEFITS
The benefits to an EPICS newcomer must be obvious—access to EPICS experts, advice on how to select anduse interface devices, help with designing custom extensions, and even complete solutions to similar problems arehigh on the list. A survey of new EPICS users indicates that they watch the listservers continuously and use themoften. Since they start out without knowing who the experts are, the listserver, by presenting their question to theentire EPICS community, is likely to reach the proper expert while at the same time giving to all the benefit of bothquestion and answer.
A less obvious, but very real and important, benefit is that of having a large and diverse user base to challenge
and stress the EPICS software. Many more bugs are discovered and eliminated by the collective development ofapplications. Additionally, they are discovered faster than if all development and usage were left to a single
laboratory. At my own facility, the Argonne Advanced Photon Source, we were blessed throughout thecommissioning period with a high performance, flexible, and quite stable control system already proven in othersmaller, shorter-time-frame facilities.

CONCLUSION
Most collaborations come into being because they benefit all parties, especially when the parties all havecommon objectives and tools. Software collaborations are perhaps the best suited to electronic communication,
documentation, and distribution. (What rf engineer would not give his right arm to be able to download animproved HOM damper design and have it incorporated into his rf cavity minutes later for testing?) The EPICS
collaboration in particular, since it is completely open with the software available to qualified organizations who
have signed a license agreement, is able to make maximum advantage of tools like listservers for help disseminationand the WWW for documentation.

REFERENCES[1] Macedonia, M. R., Brutzman, D. P., "MBone Provides Audio and Video Across the Internet," IEEE Computer,Vol. 27 #4, April 1994, pp. 30-36. This paper also available at URL http://taurus.cs.nps.navy.mil/pub/mbmg/mbone.html.
[2] Kraimer, M. R., Dalesio, L. R., McDowell, W. P., Anderson, J. B., "EPICS: Distributed Development,Distribution, and Support," EPICS Online Documentation, 1995, URL: http://epics.aps.anl.gov/asd/controls/epicsDevDisSup.html.
[3] EPICS Collaboration, et al, 1994 - 1995, URL: http://epics.aps.anl.gov/asd/controls/epics/maillists.html.
[4] Dalesio, L. R., Kraimer, M. R., Watson, W., Clausen, M., "Distributed Software Development in the EPICS
Collaboration," this conference.