![](media/image1.png){width="6.5in" height="1.2777777777777777in"}

**Draft Workshop Report\
**\
Dr. Stuart W. Shulman\
University of Pittsburgh\
121 University Place, Suite 600\
Pittsburgh, PA 15260\
412.624.3776 (voice) 412.624.4810 (fax)\
<Shulman@pitt.edu> (email)\
<http://shulman.ucsur.pitt.edu/> (home page)\
<http://erulemaking.ucsur.pitt.edu/> (project home page)

*Last modified on November 13, 2023*

**Acknowledgements**

This workshop report was made possible with a small grant for
exploratory research from the National Science Foundation (NSF**).**[^1]
Any opinions, findings and conclusions, or recommendations expressed in
this material are those of the author and do not necessarily reflect
those of the National Science Foundation.

Dr. Jamie Callan graciously made six plenary presentations over the
course of three days. These talks were vitally important for guiding
workshop participants to think critically and knowledgeably about how
human language technologies might help, or possibly hurt, the electronic
rulemaking process. Also, special thanks go to the dozen invited
speakers who made valuable presentations to help spur the small group
discussions. They were: Bob Carlitz, Devin Certas, Neil Eisner, Rosemary
Gunn, Jay LaGarde, Andrew Langer, Bruce Lundegren, Tom Mattzie,
Christine Reed, Seth Schoen, Peter Strauss, and Jim Tozzi. This workshop
also greatly benefited from the wide range of agency personnel and
nongovernmental stakeholders who took the time to attend and actively
participate in the sessions.

The author also wishes to acknowledge the assistance of Dr. Lisa Thrane,
who served as the moderator for half the focus groups and oversaw the
recording and transcription of the focus group data. Mark Perry, Dmitri
Shanin, and Jonah Katz served as effective recorders during the focus
group sessions. Jonah Katz, Dawn Harms, and Karla Stevens were tireless
and articulate advocates for wider stakeholder representation, as they
made hundreds of phone calls during May 2004 to ensure that there would
be enough participants. Karla Stevens and Dawn Harms provided timely and
effective transcription service.

Staff at the federal eRulemaking Initiative provided high-level guidance
as the workshop was being designed. The eRulemaking Initiative Director,
Oscar Morales, and Deputy Director, John Moses, were especially welcome
as lunch speakers each day. Kevin Landy, Minority Counsel for the Senate
Government Affairs Committee, convened a useful stakeholder outreach
meeting on Capitol Hill prior to the workshop and delivered
well-received remarks at lunch on June 3. Larry Brandt, Program Manager
in the NSF's Digital Government program, has been instrumental raising
the profile of eRulemaking research for the last three years by
supporting this effort and several other workshops, and we were grateful
for his lunch remarks on June 2.

Dr. Steven J. Balla, of The George Washington University School of
Public Policy and Public Administration (SPPPA), secured the SPPPA
sponsorship, as well as a location for the event, and he welcomed the
participants at lunch on June 4. Kathryn Newcomer, Director of the
SPPPA, welcomed the participants at lunch on the first two days. Clark
Mercer and Pam Turner of the SPPPA provided invaluable help with local
arrangements.

Dr. Mack C. Shelley and Dr. Lisa Thrane made meticulous edits after the
first draft, saving readers from the tangle of my late night prose. Dr.
Charli Carpenter and Sheila Shulman contributed skillfully to the
conclusion. Finally, the author assumes all responsibility for any
errors or oversights that might appear in this workshop report.

**Overview**

The "eRulemaking Research Group" was formed at a January 2003 National
Science Foundation-sponsored workshop, entitled "E-Rulemaking: New
Directions for Technology and Regulation," held at the John F. Kennedy
School of Government, at Harvard University.[^2]  Following the Harvard
workshop, computer scientists Eduard Hovy (University of Southern
California) and Jamie Callan (Carnegie Mellon University) collaborated
with social scientists Stuart Shulman (University of Pittsburgh) and
Stephen Zavestoski (University of San Francisco), to successfully submit
four linked small exploratory grants to the National Science Foundation
(NSF). As part of the exploratory study,[^3] the eRulemaking Research
Group convened a series of six workshop sessions at The George
Washington University, in Washington, DC, June 2-4, 2004.[^4]

Since its formation, the eRulemaking Research Group has participated in
and organized workshops, made presentations to federal agencies,
launched an eRulemaking text data testbed,[^5] and collaborated with
five federal agencies (DOT, EPA, USDA, BLM, and USFS) in the successful
submission of a larger proposal to the NSF's Digital Government program.
This four-year study will expand and upgrade the scope and function of
the eRulemaking testbed and systematically feedback information from
users in and out of government via workshops, focus groups, web surveys,
reports, and presentations. The following report summarizes the major
themes and issues discussed by workshop participants at the June 2-4,
2004 workshop. Full-transcripts from the sessions are available
online.[^6]

In conjunction with the workshop, a web survey was launched in early
June, 2004. This survey instrument will remain open over the multi-year
life of the project, allowing for structured longitudinal studies of
attitudes about electronic rulemaking as well as open-ended feedback to
guide both researchers and developers of electronic rulemaking
applications. Results from the survey are regularly updated and posted
to the web.[^7]

Advertised as "eRulemaking Workshop 2.0," this set of meetings was a
follow up to a two-day eRulemaking Research Group workshop held
September 4--5, 2003, at the NSF headquarters, in Arlington, VA. In
preparation for the fall 2003 meeting, agency personnel and interest
group representatives were invited to come on separate days to hear
about and discuss the research and collaborative partnerships our group
was exploring. Data generated through focus groups at those sessions
became the basis for a March 2004 grant submission to the NSF.[^8] The
fall 2003 session also made clear the need to do two things differently
at the June 2004 workshop. As a result, we expanded and diversified the
non-governmental stakeholder participants, and we allowed governmental
and non-governmental representatives the opportunity to interact during
the same sessions as well as over lunch.

As this report and the transcripts demonstrate, the workshop discussions
were wide ranging and in most instances did not result in a consensus.
The single overarching finding growing out of these meetings was that
there is very little agreement and much uncertainty about how rulemaking
procedures or outcomes will change as electronic rulemaking matures.
Indeed, after reviewing these sessions, it was tempting to entitle this
report "The Internet Changes Nothing," as a rejoinder to Stephen
Johnson's compelling 1998 argument in the *Administrative Law Review*
article, "The Internet Changes Everything."[^9] Both titles err on the
side of oversimplification. The scope and nature of public participation
in regulatory rulemaking is shifting, perhaps significantly. A number of
new opportunities exist for using information technology to create and
manage large and heterogeneous datasets. The question remaining for many
observers is: Does it matter?

This report does not answer that question. It is suggestive, however, of
a number of scenarios worthy of greater attention. A growing eRulemaking
research and practice community is just starting to shed some light on
this, yet the ultimate answer in many respects will remain
subjective.[^10] Nonetheless, scholars are still predicting a
"revolution" as a result of electronic rulemaking.[^11] To understand
better the actual impact new metrics will be developed and implemented
over time. Our research group, informed by these and other discussions,
will contribute to that effort.

In part, these new metrics will be performance measures that try to
capture indications of improved agency efficiency, effectiveness,
transparency, and responsiveness. Other baseline data will emerge on how
public users and agency officials interact with electronic dockets.[^12]
E-Government initiatives at the federal level are predicated on
"citizen-centric" changes in service delivery.[^13] Information
technology and e-commerce techniques, in particular, offer interesting
possibilities for tracking the development of public comments and rules.
Some early hunches about why this indeed may matter (e.g., a more
powerful democratic corrective mechanism via increased transparency or a
deliberative enabler of better comments, communication, and more durable
rules, thereby bridging divides between experts and lay persons or
traditionally adversarial interests) may well prove naïve, or at least
overly optimistic.[^14] Rulemaking is embedded in politics that can
trump all other factors. Most likely, the answer to the question ("Does
it matter?") will appear slowly in several unexpected ways, as citizens,
groups, and government innovate in response to the parameters of the new
electronic rulemaking environment.

Despite the lack of consensus on this and other important questions, a
meaningful dialogue took place about the present and future implications
of electronic rulemaking at each of the workshop sessions. Critical
issues for research and future policy deliberations were highlighted.
Participants came away from the workshop with a better idea of how
organized interests and the public at large will interface with
government agency personnel in rulemakings in an era of rapid
technological change. Agency rule writers learned more about how and why
public participation in rulemaking is changing. Nongovernmental
organizations were able to envision uses of information technology that
would increase the efficacy of their memberships' engagement with the
rulemaking process. The hope is that continued dialogue on these issues
will result in an electronic regulatory rulemaking system that respects
the traditions and rules of administrative procedure while enhancing the
sense of legitimacy for all involved.

**Outreach**

In April 2004, an electronic call for participants was issued via IT-
and E-Government-related listservs and through our professional
eRulemaking network. It was designed to encourage input from a diverse
set of actors with a clear stake in the development of the federal
eRulemaking Initiative.[^15] Participants were invited from six broadly
defined stakeholder constituencies: 1) larger businesses, 2) smaller
businesses, 3) labor and environmental advocacy groups, 4) good
government and public participation groups, 5) state and local
government, and 6) the legal and lobbyist professions. We asked for
participants who were willing to attend the specific half-day session
devoted to their constituency and that only individuals committed to
contributing to a serious dialogue register.

Initially, the response to the electronic call for participants was
cause for concern. At one point in May, reducing or even canceling the
workshop was a distinct possibility. A number of more direct measures
were undertaken to generate an appropriate number of participants from
the six constituencies. Three full-time students were hired to identify
and telephone directly the appropriate representatives in
non-governmental organizations over a two-week period in May. In
addition, e-mail lists of state and local officials (about 1,000) and
DC-area groups (about 1,700) were purchased, and a mass email of the
workshop call for participants was distributed. As a result of these
efforts, a sufficient number of participants registered, and the
workshop went ahead as planned.

**Participants**

In total, counting the 2 Principal Investigators and the 4 student
assistants, 64 individuals participated over the course of the 3 days. A
total of 10 focus groups were conducted, ranging in size from 7 to 11
participants. Not counting the researchers and research assistants, a
total of 7 individuals participated in more than one session. There were
33 participants who chose to keep their identity confidential, while 31
indicated attribution of their comments in future publications was
permissible. None of the comments presented in this report are
attributed to their speaker.

Federal agency personnel and academics were welcome at any of the
sessions Participation was free, and lunch was provided. Our expressed
intent was to share the findings, via this report, with government
agency managers, members of Congress, IT contractors, academics, and the
general public.

**\
Workshop Proceedings**

Participants heard approximately 90 minutes of presentations. At each
workshop plenary session, participants were welcomed and provided an
overview of the eRulemaking Research Group's activities, as well as some
of the trends and issues in electronic rulemaking and the history of the
workshops.[^16] In May 2001, the first workshop of this kind brought
together a group of about 50 academics, students, and personnel from
seven federal agencies at the Council for Excellence in Government (CEG)
with funding from the NSF and Drake University. At that time, the
E-Government Act of 2002 was a fairly obscure bill in Congress, and the
idea of a consolidated Federal Docket Management System (FDMS) would
have struck many as unrealistic due to the variegated contours of the
administrative landscape and the territoriality of the major rulemaking
agencies.[^17]

At the May 2001 CEG workshop, we proposed linking government, the IT
industry, and academia to promote technology transfer that would improve
rulemaking under the auspices of Digital Government research. Our
research questions ranged over the possible uses of IT to serve
democratic ideals to the impact of different IT enhancements on the
analytical ability of citizens and agency personnel. We proposed to
leverage new research funding opportunities at the NSF to set up annual
end-user and technologist meetings that would build social capital
around the idea of improved rulemaking.

A great deal has changed since then. With approval of the E-Government
Act in 2002 and a far-reaching Presidential Management Agenda,[^18] the
eRulemaking Initiative and its centerpiece, the government-wide FDMS,
became an imminent fact of regulatory life for agency personnel and
stakeholders. Concurrently, the nongovernmental organizations most
attuned to rulemaking have been developing new strategies (or at least
digitizing their old strategies) to take advantage of information and
communications technologies. Indeed, a new sub-sector of the economy is
taking root to provide electronic advocacy services for trade
associations and public interest groups.[^19] Of particular interest to
our research group are those innovations that may increase the
likelihood and manageability of massive stakeholder responses in
controversial rulemakings.[^20] The fear of information overload in this
context is multifaceted, encompassing concerns about ossification of the
rulemaking process as a result of what some describe as "spam-like"
submissions as well as a denigration of the role of public input in
regulatory decision-making, to name just a few.

At every workshop session, Dr. Jamie Callan (Carnegie Mellon University)
showed simple text analysis techniques that can be used to organize and
summarize the contents of large public comment databases. He asked that
attendees share their responses about what tools might or might not be
useful in the future. Public comments, Dr. Callan noted, are expressed
in human language. Although computers cannot understand human language
the way people do, they can still be useful in helping people make sense
of large public comment databases.

Text analysis software can identify letters that are exact duplicates
(e.g., form letters from a letter-writing campaign) and near-duplicates
(e.g., form letters that people have modified to represent their
opinions better).[^21] Simple phrase recognition techniques can identify
concepts that people mention frequently, which can serve as a starting
point for \"drill down\" activities that examine comments addressing
particular topics or points of view. People often identify their roles
with respect to a particular regulation---for example, \"As a mother, I
believe \...,\" or \"I have been a truck driver for 25 years and \....\"
Relatively simple techniques can find and organize such references,
enabling policy makers, rule writers, and other interested parties to
understand better who commented on a particular aspect of the rule.

These and a wide variety of similar techniques are possible in the near
future. Today regulatory agencies are struggling with basic IT issues
related to capturing public comments electronically. Soon these will be
mastered, and attention will turn to better use of language analysis and
text mining software. At present there is an opportunity to provide
better tools for rapidly analyzing large public comment databases, and
consequently for increasing transparency and efficacy in the comment
submission and analysis process.

At lunch each day, the morning and afternoon groups together heard from
and questioned Oscar Morales, the Director of the federal eRulemaking
Initiative. Mr. Morales stressed the importance of feedback from
agencies and stakeholders in the development of requirements for the
FDMS. He described a three-module development approach. Module One,
which is complete and operational, created a web portal that allows
users to search for and comment on any open federal rulemaking.[^22] He
provided an overview of the commercial off-the-shelf (COTS)-based
architecture (Oracle and Documentum) for the FDMS, which will be Module
Two. User testing of Module Two was projected for fall 2004, with the
first agencies migrating to the system early in 2005. Mr. Morales then
provided a broad outline of Module Three plans for an "Integrated
Federal Rulemaking System," which will blend IT tools with improvements
in institutional rulemaking practices to create Internet and desktop
tools that help gather, process, analyze, and communicate information in
the rule writing process. He noted the output from the workshop would
directly inform the development process.

In addition, invited speakers made session-specific presentations
focusing on the specific needs of the stakeholder community. At the
first session, targeting representatives of larger businesses, Bruce
Lundegren, Director of Regulatory Affairs at the U.S. Chamber of
Commerce,[^23] stated that the success of the FDMS will be measured by
how well the new software ensures public participation in the regulatory
process and facilitates meaningful input from interested stakeholders.
From the U.S. Chamber\'s perspective, transparency and data quality are
the key goals that must underpin all aspects of the eRulemaking
Initiative. Mr. Lundegren discussed E-Government as a revolutionary
concept that will be hugely beneficial if properly implemented. He
argued that IT should help ensure that the voice of the business
community is heard, and that it will allow the business community to
respond to issues raised by other stakeholders. This open debate should
help improve the quality of agency rules.

Mr. Lundegren stated that all rules should be based on clear statutory
authority, sound science, and rigorous technical analysis. He also said
that regulatory dockets should include all background materials, public
comments, and legal/technical analyses outlining how the agency reached
its decision. Finally, Mr. Lundegren said it would be interesting to
make historic dockets available, to reassess what agencies did in the
past, but acknowledged that the huge costs associated with such an
effort probably required agencies to prioritize which historic dockets
would be made accessible first.

At the second session, Andrew Langer, of the National Federation of
Independent Business,[^24] spoke to a group assembled to consider
electronic rulemaking issues for smaller businesses. Mr. Langer
emphasized the unique challenges small business owners face in terms of
regulatory compliance, with a particular emphasis on paperwork burden
hours. He noted regulatory costs are higher per employee for smaller
businesses that often lack the specialized personnel and state of the
art tools for understanding and complying with federal regulations. Mr.
Langer provided an example of a first-time filer spending up to 60 hours
completing paperwork for the EPA's Toxic Release Inventory. He called
for innovations that ease the burden related to regulatory compliance
and which facilitate better partnerships between stakeholders and
regulators.

The invited speakers for the third session were Tom Mattzie, an
organizer with the AFL-CIO, and Devin Ceartas, a private sector web
programmer with Nacredata who has worked with many organizations in the
sustainable agriculture and forest protection movements.[^25] Tom
Mattzie spoke about the need to consider the different roles that NGOs
play as intermediaries relaying word of important decisions and the
opportunities to comment. His on-line advocacy resulted in 50 million
emails to members and 5 million comments or letters to decision makers
over the last two years. For Mr. Mattzie, the e-advocacy model is one
that relies on using an email list to grow and mobilize the union
membership around issues that are important to working people.

Devin Ceartas identified two factors that framed his comments. First, he
expressed concern about the digital divide. For Mr. Ceartas, while many
low-income and/or rural individuals and groups would welcome the arrival
of broadband as well as the skills and equipment to make the most of it,
access is far from universal and some communities are likely to remain
behind the curve for the foreseeable future. In the case of Mennonite
and Amish communities affected by federal farm policies, as well as
those choosing to live \"off the grid,\" the divide is intentional and
will never go away. In many cases, those with lower levels of Internet
access correspond directly to the groups of people most in need of the
government programs for which rules are being developed. These groups
have always had less access to government. Such people have the same
rights as any other group of citizens.

Second, Mr. Ceartas suggested his relationship with agency rule-makers
was adversarial rather than cooperative. In part, this stemmed from
dissatisfaction with agency decisions. In other cases, however, the
antagonism involved difficulties with comment submission in the past. He
noted some of the agency "Spam" filters encountered have been crude and
over-broad. Mr. Ceartas argued systems that block IP numbers or rely on
reverse DNS mapping run the risk of throwing away many legitimate emails
and create a particular disadvantage for smaller organizations, which
often use shared name-based servers. He concluded that organizations
will strongly resist sending visitors to their websites away to an
agency or government wide web portal to make comments, since the chance
to donate or volunteer via the NGO website is a primary reason for its
existence.

For the fourth session, Robert Carlitz and Rosemary Gunn, of Information
Renaissance, made a joint presentation. They were followed by Seth
Schoen, of the Electronic Frontier Foundation (EFF).[^26] The objective
of Information Renaissance, said Carlitz and Gunn, is an e-rulemaking
system that facilitates broader public involvement and deeper
discussions of the issues behind the rules. They presented an overview
of features of value to individuals and organizations. The basics
include a system that encourages participation by meeting high usability
standards, providing active notification and presenting background
information about rules and rulemaking; and promotes transparency,
accountability, use of public input---and in the end \"better\" rules.
More advanced features include structured input forms (to help assure
meaningful comments), rebuttal comments (and eventually online
dialogue), open modeling (to help in understanding technical data), a
continuous electronic record (to link rulemaking documents to what comes
before and after a notice and comment process) and Web services. Web
services are of special interest to organizations that work with rules.
They could allow, for example, analysis with local tools, submission of
complex sets of comments, bulk data transfers and development of
alternative front ends.

Seth Schoen spoke about the lack of public awareness of rulemaking.
Citizens often don't know it exists, how it works, that the public can
participate, what subjects it addresses, and that particular rulemakings
are taking place. He argued it is the role of advocacy organizations is
to increase public awareness of and participation in rulemaking
proceedings, since relatively few individuals will learn about them on
their own from agencies, as opposed to through advocacy organizations.

Mr. Schoen asserted agencies should never reject or ignore duplicative
comments. Agencies, he noted, are legally required to consider comments
and make them a part of the public record, although they are certainly
welcome to use software to help them organize comments and identify
which are duplicative. Preserving the full record is important to many
entities for their own reasons. The ability to send form letters helps
make people aware of issues, familiarize them with the comment process,
and causes them to think of themselves as people who can and do comment
on proposed regulations. As a result, he suggested they will be more
willing and able to participate in other ways in the future. In short,
the ability to send form letters has value to people outside of
agencies, for example, political advocacy value to advocacy
organizations and their constituents.

Mr. Schoen stated that what advocacy organizations would like from the
docket management system programmatic input and output (for example, by
means of a web services interface), for reasons including integration
with \"action centers\" that help people submit comments through an
advocacy organization\'s web site, which is and probably will continue
to be overwhelmingly the way that non-specialists learn about
proceedings and comment. He advocated for flexible searches and
notifications, whether via e-mail or Rich Site Summary (RSS)[^27] and
the ability to design custom notifications with high specificity, in
addition to bulk data retrieval from dockets.

At the fifth session, Christine Reed, of the University of Nebraska,
Omaha, representing the National Association of Schools of Public
Affairs and Administration (NASPAA), was followed by Jay Lagarde, of the
Virginia Department of Planning and Budget. [^28] Dr. Reed is the
Principal Investigator in Nebraska for the most recent round of the
Small Communities Outreach Project for Environmental Issues (SCOPe), a
project funded through a Cooperative Agreement between NASPAA and the
U.S. EPA Office of Policy Economics and Innovation. The purpose of SCOPe
is to engage small local government elected officials and administrators
more effectively in the rulemaking process. Since 1998, NASPAA member
schools and their Public Administration faculty have met with over 350
small communities in 20 states to discuss clean air, safe drinking water
and other EPA rules as they were being developed by agency rule-writers.

Dr. Reed discussed the initial results of the most recent round of
SCOPe, which focused on the eRulemaking Initiative, rather than a
specific rule. Although two other states were involved (Pennsylvania and
Indiana), Dr. Reed reported only on her own study in Nebraska. She
visited 16 different sites across the state. Local participants included
city administrators, municipal clerks and county clerks, and all had
access to high-speed internet. None of the participants were aware of
the eRulemaking Initiative, but once Dr. Reed demonstrated use of the
new portal all agreed it would be easy to access for purposes of
submitting comments. All stressed the continued need for membership
associations (e.g. League of Nebraska Municipalities and Nebraska
Association of County Organizations) to "filter" the large amount of
technical and legal information contained in each NPRM and to alert
members about particular rules and salient issues. They stressed the
institutional constraints affecting their participation as small local
governments: limited human and financial resources, as well as limited
expertise needed to analyze complex rules and associated impact
analyses. All stressed the fact that despite access to high-speed
internet, small local governments were still at a disadvantage compared
to larger entities in making an impact on the development of federal
rules.

Jay Lagarde noted that Virginia has been a leader in the e-rulemaking
revolution since 1999, when it first launched the Virginia Regulatory
Town Hall website. The Town Hall is a web-based system for managing,
accessing, and participating in state rulemaking. In addition to
managing rules review and other internal aspects of the rulemaking
process, the Town Hall also provides public access to current and
historical rulemaking documents; regulatory meetings, meeting agenda,
and minutes; guidance documents; legislative mandates requiring new
regulations; custom tracking reports; and other information useful for
public participation. Users of the Town Hall can receive free,
customized email notifications about regulatory meetings and events.
Users also can view and enter comments on regulations through the Town
Hall's public comment forums.

At the sixth session, Friday afternoon, it was decided that 10 focus
groups provided sufficient data to draw substantive conclusions. As a
result, the workshop concluded with a plenary session targeting lawyers
and lobbyists. Neil Eisner, an Assistant General Counsel at the U.S.
Department of Transportation (USDOT), followed Peter L. Strauss, of the
Columbia Law School. [^29]

Peter Strauss presented findings from a recent survey of members of the
American Bar Association's Section on Administrative Law and Regulatory
Practice.[^30] He framed his presentation with a set of questions about
the impact of electronic rulemaking: At what point in the process should
agency deliberations become part of the docket? How will the character
of public participation change? Will a centralized, electronic docket
further transform the role of the White House, creating new issues
related to information control and transparency?[^31] In sum, Professor
Strauss noted the move from expertise to a form of plebiscite,
machine-generated comments, and the possible shift from one-way to
dialogical commenting, were among the issues requiring greater
attention.

Neil Eisner spoke about six legal issues involved in internet-accessible
dockets. He stressed that some are not new issues, but are exacerbated
by placing the dockets on the Internet. The first issue is whether
electronic records can be a legal record or whether the agency would
also have to keep a hard-copy record. Mr. Eisner believes this issue
already has been resolved and that the electronic record can be the
legal record. Another issue is whether agencies can accept anonymous, or
\"Jane Doe,\" comments. The third issue is whether agencies must censor
documents they receive to delete obscene material, especially because
children would have easier access to the material on-line. There is a
similar issue concerning copyrighted material, since it can be copied
more easily via the Internet. Due to the ease with which individuals can
search electronic, Internet-accessible dockets, there is also the
question of whether privacy rights would be violated. Finally, Mr.
Eisner noted the issue of whether agencies must prevent information they
would have reason to believe has been obtained illegally from being
placed in the docket.

After a break in all but the final half-day session, the participants
split into two focus groups for the remainder of the time. The smaller
groups engaged in a facilitated and recorded discussion of a series of
open-ended questions about the practical and political impact of
information technology on the \"notice and comment\" process (see
Appendix A for the focus group protocol). The discussion questions
sought to elicit feedback that could help shape the development of
requirements for future eRulemaking research and practice. Participants
signed informed consent forms allowing them to choose anonymity or
disclosure regarding their identity.

**Report Preparation Methodology**

After the workshops, the audio recorded sessions were transcribed into
MS Word documents and loaded into Atlas.ti; a qualitative data analysis
software package. Based on a review of the full transcripts, a list of
15 themes was generated that were raised by the workshop participants.
These broad, often over-lapping themes became were the basis for the
following analysis. The transcripts were re-read in Atlas.ti and codes
were associated with passages in the transcript that touched on one or
more of the themes used for the analysis. Code reports were exported for
closer analysis of each theme and their intersections. A total of 10
major themes are reported below.

**Summary of the Major Themes and Issues**

[1. Tools for Electronic Rulemaking]{.underline}

Many of the discussions focused on the role of new tools and technology
that are, or soon will be, part of the digital landscape of rulemaking.
A wide range of opinions were expressed on the impact of the Internet,
web portals (e.g,, Regulations.Gov), electronic dockets (e.g., DOT's
Docket Management System or EPA's EDOCKET), government and
nongovernmental organization (NGO) web sites, e-advocacy techniques, and
human language technologies (e.g., Google or duplicate detection
algorithms). While a number of possible impacts and outcomes were
raised, many of the participants qualified their own remarks, or the
comments of others, by suggesting the true impact of these developments
was not yet known, or perhaps even knowable.

Thus, at the very least, there was a convergence of viewpoints around
the notion that questions about the impact of these new tools were rife
with uncertainty. Some suspected that the unintended consequences of
technological innovation might overshadow the hoped-for gains. One
participant pointed out that any improvements would be "illusory" if
they were not designed carefully and measured accurately. Readers of the
transcripts and this report will note the frequent juxtaposition of
technological impacts that constitute, in the words of one speaker, a
"double-edged sword" in the e-rulemaking environment.

As a result, more ubiquitous and accessible information in rulemaking
was seen as potentially a benefit and cost---both in and out of
government---that might either help or hurt the process and its
outcomes. For example, amid the general sense that more accessible
information generally is a positive development, lay a solid
undercurrent of skepticism that the information "flood" could be
managed. It was suggested this was true not only for individual citizens
and overburdened agency officials, but also for many organizations with
staff and resources spread thin. Unless there were user-friendly
technological and procedural innovations, electronic rulemaking would
result in *de facto* information overload.

A number of participants nonetheless mentioned the likelihood of time
and cost savings derived from better on-line navigation tools and
automated categorizing of more complete and well-indexed dockets. [^32]
In a nutshell, electronic rulemaking would be more efficient and
effective, overcoming many of the shortcomings associated with paper
dockets, while pushing the benefits out to a more geographically and
demographically diverse set of stakeholders.[^33] Other comments,
however, predicted an increase in the number of hours or dollars devoted
to developing and using information technology during both the
preparation of comments and the agencies' analytical process. Some of
the top quality e-advocacy services available in the current market, as
one person noted, are not cheap.

The paradox for many observers is that the same technologies that make
it easier to submit or read public comments and aggregate supporting
information also can make it both easier and more difficult to make
sense of the comments and the agency's underlying rationale for its
decisions. Several participants noted the presence and associated
implications of these new tools would mean unmanageable costs and time
burdens for NGOs with limited resources, or for citizens with limited
time to master the details of navigating through a sea of documents. One
asked that designers of these tools attend to the unique needs of small
business owners.[^34] Another thought that for some constituencies
already not used to conducting mass-mail campaigns that electronic
rulemaking was another opportunity simply beyond their ken.

Another set of participants focused on the idea that e-rulemaking tools
would result in better comments. Rulemaking submissions during "notice
and comment" could be better informed and expertly guided in a
hyperlink-intensive, interactive, or well-structured commenting
environment. The guidance could come from experts inside the agencies as
well as interest groups with specialized knowledge.[^35] Yet, as will be
noted in more detail below, the idea that information technology
necessarily will result in better comments was not universally accepted.

For example, to the extent IT is employed to mobilize one-click "me too"
comments, the fear for some is that far from improving the quality of
public comments, the current e-advocacy model will result in a dramatic
increase in comments that have little or no value to the administrative
process. The proliferation of these electronic postcards might give the
impression that the administrative process is actually a forum for
direct democracy via nose-counting plebiscites.[^36] Dissenters spoke of
the right for anyone to submit any quality of comment, with some arguing
effective engagement with the rulemaking process legitimately could be a
secondary concern. Some nongovernmental representatives describe the
notice and comment process as a legitimate forum to propel political and
legal strategies into the spotlight or the courts.

The gap in the ability to mobilize resources to perform these functions
was noted as an issue for participants both in and out of government.
Resource-poor groups and IT-dollar strapped agencies cannot necessarily
be counted on to provide the needed content and delivery mechanisms
required to produce better comments. One participant questioned whether
it is the proper role of government to concern itself with improving the
quality of the public comments.

A number of participants also raised the notion that new tools for
on-line collaboration might result in innovative deliberation
possibilities. The most commonly mentioned tools were listservs, online
chats, and moderated discussions. Here again, there was a recurrent
tendency to temper enthusiasm for new communications technologies with
considerations that were much more social, political, and legal in
nature then they were technical. A key debate concerned who, or what
entity, was able to make legitimate choices about the nature and scope
of new deliberative mechanisms. Many non-governmental stakeholders
questioned whether even a well-meaning federal government can get the
critical design choices right. Others wondered if a single design choice
for the FDMS would ill-serve the divergent business processes at the 180
or so rule writing entities in the federal government, or their many
constituencies.

Significant concerns were expressed by some attendees that the ongoing
development of the Federal Docket Management System (FDMS)---a
centralized system for the entire federal government---might constrain
innovation in the marketplace of ideas about electronic rulemaking. An
inflexible architecture for citizen government interaction might result
in a vast, monolithic system that functions poorly and draws in few
users. Instead of fostering, in the words of one participant, "a
thousand flowers blooming," with innovation in the manner of
deliberation at the agency or sub-agency level, the new centralized
system could forestall constituency-specific creativity. The effort to
build a FDMS that is "all things to all people," noted another
participant, likely would produce a lowest common denominator system
that rulemaking agencies and stakeholders alike would reject or ignore.
Similarly, if the FDMS was perceived as incomplete by agencies with
existing docket systems, over time duplicative systems might emerge and
compete. Furthermore, one person stated that if the FDMS was developed
in a secretive, top-down manner, the likelihood of failure would
increase.

Dissenting views on this point stressed that the rollout of the FDMS
would solve a significant digital divide that exists within the federal
government, between those agencies (about 30) that have electronic
docketing systems and the over 150 rule-writing entities that do not.
Another participant remarked that it actually required a process of
forcing rule writers to use a new technology to make them think
creatively and feed back critical information about how to improve it
over time. Another stated that no amount of planning could overcome the
complexity involved, and that trial and error was unavoidable. Agency
representatives noted they tend to worry that technical error in a
rulemaking procedure could endanger years of work on an important rule.

Perhaps most important, the corrective many stakeholders identified for
any present or future flaws in the design of the FDMS was an open
application programming interface, or API. An API is a set of
definitions that computer programmers can use to allow two computers to
perform a wide range of data exchanges. A clear consensus was expressed
that an open API would relieve the developers of the FDMS from the
burden of making their comment submission portal "all things to all
people." For nongovernmental actors and electronic advocacy specialists,
this single FDMS design choice---whether or not to publish and open
API---seemed to hold the promise that the traditional intermediaries
(the interest groups) would use their own electronic interfaces to
inform their members, dissect the docket, prepare meaningful comments,
and manage high-value client lists.

There were a number of remarks about the way that on-line tools might
transform the nature of advocacy, activism, and the analytical process
itself. Perhaps the most salient advocacy transformation was the idea
that the increasing trend toward mass campaigns of identical (or nearly
identical) public comments might be replaced over time with more
sophisticated methods for gathering and synthesizing data, or novel
ideas, that are applicable to the decision-making process under the
traditions, laws, and rules of administrative practice. The hope
expressed by some was for a steady move away from duplicative form
letters toward more interactive electronic forms and forums that cull
actual data and truly innovative ideas from stakeholders.[^37]

The dissenting view, generally held by NGO representatives, was that
these mass mailing campaigns have a meaningful role that is both
educational and political, even if 500,000 nearly identical
opinion-based, vote-like submissions count only once in the decision
process. Many participants had a hard time imagining that the campaigns
would go away or that form letter submitters would find the time for
more demanding forms of engagement. Others insisted the preservation of
this form of comment was critical to ensuring democratic legitimacy,
even if the comments ultimately had no bearing on the decision.

The perception of a tool-driven transformation of activism was two-fold.
First, some participants viewed the new technologies as enablers of more
litigation that would increase the vulnerability of agencies to legal
challenges from "losers" in the most vigorously contested rulemakings.
For those activists who thrive on playing the game of defeating,
delaying, or substantially altering undesirable rules, some said the new
IT would be a boon. The fear of electronic monkey-wrench activities was
palpable for the agency participants, while some nongovernmental
speakers saw it instead as the check and balance of last resort when
politics or some other factor precludes winning on the presumed merits.

The second activist transformation suggested was that electronic
rulemaking would either expose, or else proliferate, "flawed" models
that sometimes are the basis for a rulemaking decision. Better access to
industry, academic, or government data would increase the ability to
replicate and test models. Others noted the same tools might just as
well explain and demonstrate the legitimate models on which many expert
decisions rest. Many seemed to agree that user-friendly, non-expert
visualization and manipulation applications (akin to UrbanSim[^38]),
would make the operative assumptions and rationale more transparent.

In terms of the transformation of the analytical process, there were
concerns that new technologies invariably will bias the process. One
participant thought de-contextualizing comments using advanced human
language technologies would diminish their significance. Another asked
whether the tools themselves would be transparent. If agencies are
sorting and categorizing public comments using advanced technologies,
participants wondered about the extent to which that fact and the
technical details behind it need to be made public. Others questioned
whether the expertise in agencies might shift from knowledge of rules
and data to mastery of large-scale language analysis tasks. The dissent
here was to note that good tools would take an already bogged down
process and shift the analyst workload from sorting similar and
identical documents to actually doing analysis on all the unique and
valuable submissions.[^39]

[2. Usability of the New Tools and the FDMS]{.underline}

Workshop participants were in general agreement that the usability of
electronic rulemaking systems was a high priority. If future systems,
such as the FDMS, were to make usability studies a central part of an
iterative development process, the perception was widely shared that
electronic rulemaking would become a "great mobilizer" and a net benefit
to society. In the absence of such studies, many participants could
imagine the new FDMS being underutilized by agencies and the public. If
the system is slow, lacks adequate categorization, or is poorly indexed
(like the current EDOCKET at the EPA), it will inspire "work-arounds"
and off system solutions to conduct routine business for which the
system was supposed to be designed.

One participant noted that future regulatory reforms, such as those
promulgated by Congress or the Office and Management and Budget (OMB),
ought to target those aspects of rulemaking that have the effect of
confusing or overwhelming non-expert participants. Some of the specific
sources of confusion noted were themselves past regulatory reforms.
While it was widely thought that IT can help in this regard, if done
right, there also was a sentiment that no amount of technology can trump
the numbing effect of the Paperwork Reduction or Data Quality Acts on
the ability of average citizens to engage the process.

It was noted that e-commerce firms, such as eBay and Amazon.com, pay
particular attention to every mouse click, looking for signs in their
web logs that their systems are losing or confusing users at specific
points in a transaction. The hope was expressed that the federal
government would improve a poor record in this area by commissioning
regular and extensive user studies and borrowing the best practices from
e-commerce where applicable. The web-based interface allows unique
opportunities for anonymous tracking via session cookies; thus the
e-commerce model ought to be employed to compile baseline data on how
the system is used. Referral systems, for example, seamlessly could
facilitate easier and broader dissemination of the most commonly-viewed
documents and comments, making navigation inside a complex docket more
manageable for non-experts.[^40] One participant stated that a
user-friendly format will help bring about better comments.

Some concern was expressed about how future systems might be
technologically sophisticated and therefore difficult to use. For
example, one person noted a system that automatically chops up complete
comments and compartmentalizes them for analysis can be cumbersome
analytically in the absence of the full context of the original comment.
Similarly, the widely discussed hope that IT-enhanced systems will
increase transparency in the process, noted some participants, can be
realized only if you can search for and find what you need with the
efficacy and ease that many now attribute to Google searches.

On the issue of the usability of the application programming interface
(API), one participant made it clear that for advocacy organizations,
this simply means a lower cost to access and manage data during both the
data submission and export processes. If the government publishes a
standard set of API specifications that apply across all rulemakings and
agencies, most of the participants seemed to agree that the usability of
the system will go up significantly, while the pressure on the federal
government to structure its FDMS interface to suit all future users'
needs and abilities, in and out of government, would drop considerably.

[3. The Structure of the FDMS and the Comment Process
Itself]{.underline}

There were two predominant lines of discussion about the role of
structure in electronic rulemaking. At the micro level, participants
talked about how to structure the comment intake to improve the ability
of people submitting comments to provide usable information and the
analysts, in and out of government, to sort through the comments. At the
macro level, there was concern (and some hope) expressed about the
notion that all agencies and constituencies would be adapting to a
single, centralized structure for electronic rulemaking.

On the issue of comment intake, a central question was: Can it be
structured usefully, and, if so, by whom? If such a structured
commenting platform were to be developed, would anyone want to use it?
There was a fear expressed of forms that ask "stupid" questions. Another
person warned about the limited thinking about the process, and the
excessive focus on technology. It was stated repeatedly that the needs
of the rulemaking process, and not the availability of the technology,
should be the driver of change. Another remarked that the people in and
out of government who understand the business process and the nature of
the available IT are rare, but critical for transformational activities
to take place.

The development to date of a centralized FDMS was criticized by some
participants. The different agency cultures and constituencies were
pictured as a barrier to effective centralization that in the worst case
might thwart innovation at the agency level and foist an unwanted system
on thousands of rule analysts and millions of commenting citizens.[^41]
One speaker praised efforts in other E-Government initiatives to embrace
the design principles based on federal enterprise architecture, noting
that the developers of the FDMS had no such affinity for "modularity" or
architectural transparency.

[4. Legitimacy Gains and Losses]{.underline}

On the issue of increasing rulemaking legitimacy, participants remarked
that hopes were high, but the ultimate impact was uncertain. The
benefits of enhanced legitimacy, one participant noted, were difficult
to gauge due to the number of intangible factors and the lack of a
reliable metric. One speaker identified less controversial rulemaking
outcomes as a pathway to rules that are more likely to be implemented
effectively and observed. Another stated categorically that enforceable
rules are not by definition good, because the ability to overturn a bad
rule is fundamental to the mission of their organization.

Most of the participants seemed to agree that to the extent the
transparency of the process, data, models, and workflow in rulemaking
increased, so, too, would procedural and substantive legitimacy.
Hyperlinked navigation to the statutory authority for a rule, or OMB
guidance on rulemaking itself, also was mentioned as innovation likely
to enhance the legitimacy of the process. One participant noted
electronic rulemaking would result in more supportive comments and with
a visible rationale, thereby enhancing agency legitimacy. Better access
to a more complete set of documents might translate into fewer Freedom
of Information Act (FOIA) requests. Conversely, increased access to FOIA
procedures could produce a spike in requests for agency documentation.
Either way, the hoped-for effect is that rulemaking legitimacy may
increase as the public comes to better understand and appreciate the
regulatory process and their rights and role in it.

A dissenting view, however, argued that super-transparency, revealing
perhaps long-obscured aspects of agency discretion in an unflattering
light, inadvertently could harm the legitimacy of the process. Awareness
of a shift from black box to glass box in sensitive aspects of
rulemaking might compel rule writers and regulatory managers to move
substantive decision-making and deliberations into forums that in effect
fly below the information highway's Super-Doppler version of the
transparency radar. One participant noted legitimacy derives from
serving the public, not the government's own interests.

It was suggested that electronic rulemaking could enable more open peer
review of scientific studies. Participants recoiled at the suggestion
that an e-commerce-style rating system might allow the public to vote on
the validity of a particular study. Nonetheless, most participants
seemed to agree that getting models used in the process into the public
domain, available for public comment and subject to a rebuttal, would
help to "de-bug" error-prone assumptions. A particular problem mentioned
more than once was the reliance on proprietary models that are not
transparent. For some participants, increasing the transparency level
would make the agencies more vulnerable to technical and procedural
challenges.

Several participants remarked that legitimacy might be enhanced if new
tools rendered the assumptions involved in cost-benefit analysis more
amenable to inspection and analysis by experts and lay persons alike.
This innovation might also expose any failures to understand fully the
nature of stakeholder needs. Whether the stakeholder is a small business
owner or an organic farmers' association, the ability to see, submit,
and comment on the use of cost-benefit data is likely to improve the
opportunity to shape the outcome of rulemaking.

Some participants worried that human language technologies might be used
by agencies "automatically" to discount form letters. It was noted that
current accepted administrative procedure demands that duplicative
comments be treated in that manner anyway, and that technology simply
would render it a less burdensome and more accurate process. One person
remarked that recent regulatory reforms and Executive Orders put
rulemaking on a trajectory to diminish the significance of all
non-expert public commentary, regardless of the media used to deliver
it. For some, the implementation of the Data Quality Act and OMB peer
review guidelines represent an overt threat to all public comment; to
others, there are clear and fair options to eliminate non-substantive
input from consideration in the decision process.

Another participant claimed the ability to point to and view one million
public comments, duplicative or not, instills a sense of legitimacy
about the transparency of the process. Another noted some courts will
occasionally cite the agency's tally of the yeas and nays in their
decision. There was general agreement that transforming mass email
campaigns into distributed data collection enterprises had potential to
remake the model of e-advocacy and public participation in
rulemaking.[^42] Nonetheless, advocacy groups are likely to cling to the
mass email campaign as long as there is at potential political, legal,
or organizational utility in doing so.

A related discussion concerned uncertainty about how different comments,
submitted via different media, would be valued. Whereas some fear
sorting technologies will cause paper (especially handwritten comments)
to lose relevance, versus electronic comments, others predict a
diminished fate for the e-comment and the continued supremacy of the
paper comment, particularly on organizational letterhead. One person
noted it was not so important to be concerned about how your medium of
choice is viewed. Rather, what matters is the ability to see how your
input is used. The generation of an automatic acknowledgement of a
received comment and a link to it in the docket were considered easy
places to realize legitimacy gains. Another participant, however, spoke
of concerns that federal agencies will foster and avail themselves of a
citizen management industry; something more like an automated public
relations firm rather than an administrative law enterprise.

[5. Issues Related to the Law & Litigation]{.underline}

There was strong sentiment expressed by some that radical change indeed
is underway at the confluence of IT, presidential Executive Orders, and
the regulatory reform efforts in Congress over the past two decades.
There was consensus that the regulated community is better informed and
more strategic than ever before. The proliferation of litigation
opportunities, whether to delay or derail unpopular rules, has increased
the perception, rightly or wrongly, that there is a general up-tick in
interest group opportunism. The optimists see IT used as a mechanism for
enhanced early participation by stakeholders to stave off time- and
resource-consuming litigation. Especially in an era of declining agency
resources and expanding responsibilities, some participants thought the
use of IT in this manner might free up agency resources.

Whereas the agency personnel seem to view legal challenges as
roadblocks, the interest group representatives were more likely to
characterize them as needed checks on arbitrary administrative powers.
When asked if the rate of litigation in the era of electronic rulemaking
was a proxy for better or worse rulemaking, many agreed subject matter,
level of political involvement, and the supporting cost-benefit
analyses, not the availability of IT, determine whether there will be
litigation. One participant stated that some public comment is submitted
primarily to lay the groundwork for litigating against a rule. Whereas
IT can help members of a group shape, understand, and comply with a
rule, it equally well provides the tools to undermine an undesirable
outcome.

[6. Collaboration: Opportunities and Barriers]{.underline}

Participants generally were optimistic about the impact of IT on
collaborative elements of the process, although one dissenter noted
collaboration was an "overblown" goal. Another stated that interagency
meetings help to sort out unresolved issues and spread best practices by
talking about how different agencies experience the same process. The
possible need for a Federal Advisory Committee to formalize the
collaborative development of the FDMS was raised. A concern was
expressed that, although some interagency agreement on data collection
and other business process questions was possible, the tendency for
agencies to covet their own way of doing business was not necessarily
going to be reduced by collaboration on the development of the FDMS. The
complexity is manageable, noted one participant, but the minutiae
represented a considerable burden. At least one participant stated the
FDMS staff had spent too little time collaborating with its customers,
and therefore was too reliant on the agencies to look out for the needs
of their constituencies. The role of the EPA as managing partner for the
federal eRulemaking Initiative raised some issues for more than one
participant.

One person noted the way their NGO used IT to identify and coordinate
with allies in a rulemaking. Another thought the new technology might
"socialize" government agencies in a manner that made collaboration
within agencies and with the public more meaningful. It was suggested
that electronic rulemaking will ease, and thereby increase the level of,
state and local government interactions with federal rulemaking.
Following on the idea that mass mail campaigns might be converted to
more useful data gathering exercise, one participant noted it would take
substantial, possibly IT-enabled, collaboration between agencies and
stakeholders to make this happen.

It was stated that some form letters are the product of significant
collaboration, between, for example, the experts in the various
divisions of an NGO or advocacy coalition, and the supporting
membership. In such cases, some contextual knowledge about the source of
the form letter and the process by which it was derived would be needed
to realize the extent to which the "form letter" is in fact the
end-product of a lengthy, deliberative process, rather than an
insubstantial "rant" based on ideology or a summer project of an
isolated intern. No consensus emerged about the role of value-based
outrage expressed via public comments in rulemaking

[7. Innovation]{.underline}

There was wide agreement about the nature of IT as a driver of
innovation. As with other elements of the discussion, the participants
did not agree whether the innovation would help or hurt the process. One
concern was that the FDMS would be slow to respond to the needs of its
users. There were discussions about how best to ensure flexibility,
agility, privacy, security, and reliability. Participants noted that
rigid architectural choices could produce stagnation in the evolution of
electronic rulemaking. Another predicted a fight for control of the
structure of the FDMS, making it a site for battles over the hard-wired
rules for public participation in rulemaking. The solutions noted ranged
from open source code to the facilitation of web services via an open
API.

Discussions of procedural innovations facilitated by IT focused on the
ability to see and comment on other comments or documents in the docket,
as well as the possibility of making rebuttal comment periods (which
currently are rare) a standard part of the rulemaking process. While IT
could greatly facilitate such a change, it would take Congress or an
Executive Order to get all the agencies on board for such a major
innovation. The rebuttal comments on the comment process, most agreed,
would address the gaming of the system by those participants who file
comments at the last minute to avoid being subjected to a rebuttal
comment. One participant noted that IT speeds the process up, creating
more pressure, and leaving those who prepare public comments with the
sense of rising expectations about their work.

[8. Educational Impact]{.underline}

Many participants spoke positively about the educational potential for
electronic rulemaking. The hope was expressed that IT-enhanced
rulemaking would contribute to making better citizens and comments,
while increasing compliance with and understanding of federal rules.
Citizens would benefit not only from the ubiquity of accessible
information, but also the ability to personalize government and interest
group information services. While some predicted better education for
traditionally underrepresented groups, dissenters warned that no amount
of on-line information or tools will solve education issues for the many
digitally and otherwise illiterate citizens who disproportionately make
up underrepresented groups.

One thread of discussion focused on the proper role of government as an
educator of its citizens. One person remarked it was insufficient simply
to post information on the web, since the rulemaking process and its
imperatives were unfamiliar to many citizens. It was suggested that the
FDMS could have a prominent link to a "This is Rulemaking" page that
explained the notice and comment process in straightforward terms. This
page could explain what constitutes a good public comment, and that page
could link to agency-specific definitions of the type of information
that is credible to rule writers (e.g., scientific studies, reports,
hard data, and replicable models).

A recurrent theme expressed by agency personnel in these and other focus
groups stressed the need for those facilitating and participating in
mass mail campaigns to understand better the limited impact of such
efforts. Agency personnel have stated consistently that whether it is 50
or 500,000 identical or similar comments, the value added to the
rulemaking process is considerably less than one careful, specific,
substantive comment. One participant, however, stated that for many of
the one-click submitters participating in a rulemaking, the sending of a
form letter as part of an email campaign is often a first step into the
political process, one that can be a gateway to more significant
engagement and participation.

Others expressed interest in having the government use the web to get an
explanation of the agency mandate into the hands of the public. It was
noted that most citizens remain ill-informed not only about the
rulemaking process and its statutory authority, but also the rules
themselves, once they are part of the Code of Federal Regulations. One
participant was concerned that government use of IT to produce more
informed comments or better-informed citizens might exceed the proper
scope of its role and generate a whole new set of issues. There was
general agreement, however, that the government bore some responsibility
to try and reach the affected stakeholders in a rulemaking, regardless
of whether they spoke English. To that end, many agreed that a *Federal
Register* notice was insufficient and that IT presented a new toolbox
for educational outreach.

It was noted that there exists a possibility for traditional
intermediaries (interest groups) to educate their membership more
carefully, with the goal of generating more detailed, high-impact public
comments. One participant stated that thousands of associations count
educating their members among their top priorities. In return, many of
those groups rely on having their members go through their system for
preparing and submitting comments on rules. The ability to harvest
membership data via these systems in turn educates the groups about the
needs and composition of their constituency. For one participant, this
data became the basis for more carefully targeted action alerts and
other campaigns. Another person suggested that there ought to be annual
conferences on the "ABCs" of rulemaking, which might ensure that the
groups' leaders and mobilization specialists themselves better
understand the function of public comment in agency decision-making.

[9. Level and Nature of Public Participation: Quantity vs.
Quality]{.underline}

Many of the groups wrestled with the sense that increased public
participation might degrade the overall quality and efficacy of public
comment. The implicit assumption seemed to be that a quantity/quality
tradeoff exists. According to this view, the increasing number of
comments necessarily means more duplicative, non-substantive form
letters and possibly less attention to those comments that might
contribute meaningfully to a better rule. An "explosion" of anecdotal
comments also was thought, by some, to be a net loss for the overall
quality of the comment process. One person saw the move to
rhetoric-based comments as unlikely to solve any of the problems rule
writers face. Another remarked that some of the web applications
currently running are automated dumps without any practical meaning.

Some dissenters noted that this view of a clear trade-off misses the
fact that amongst the flood of form letters there also might be
significantly larger numbers of useful comments. Another participant
noted a clear trade-off for groups that might take on the task of
synthesizing all their members' input into one coherent, high-impact
comment, or could instead seek a newsworthy large count of forms and
what the US Forest Service Content Analysis Team calls "form-plus"
comments.[^43] One person noted any turn to technology that limits or
marginalizes participation would constitute a democratic loss.
Conversely, any technology that ensures every comment submitted gets
considered, even in a torrent produced by well-organized campaigns,
would increase the public legitimacy and legal defensibility of the
process.[^44]

There was some debate about what actually counts as participation. Most
seemed to agree that the submitter who never reads the proposed rule or
even the preamble before clicking on an email or web site link to
comment barely passes the lowest possible threshold. When that person
goes to an automatically populated web action form (the 2-click,
cost-free comment), the temptation for some is to dismiss this act for
failing to clear the threshold of meaningful participation. While some
participants identified this as a less than fully legitimate form of
engagement in rulemaking, others saw it as a proper compliment to more
thorough, costly comments prepared by organized interests. One person
stated the EDOCKET system was designed to limit participation until it
was too late to make a difference. Another suggested the focus ought to
be on the entire process, and not simply improving notice and comment.

Some were concerned that the logical outgrowth of these actions would be
the full and formal disenfranchisement of mass submissions.[^45] Another
spoke to the fear that the people are being written out of the process.
One person remarked that the goal is not to ensure everyone can
participate in every rule; rather, it is to get real stakeholders
interested in rules that impact them, and to do so early in the process.
Many agreed with the notion that letter counts were inferior by
definition to solid data and facts, yet one person recognized there is a
"legitimate tension" between anecdotes grounded in values and facts
derived via the scientific method.

Some participants were ambivalent about the benefits of a
point-counterpoint dialogue. A proponent stated that a visible and open
debate would sharpen both sides, and another saw potential for more
scientists pointing out flaws in assumptions. One person questioned
whether such a dialogue would feature statements such as "there's no
science" by people who have never read any scientific studies
themselves. There were several discussions of the use of rebuttal
periods as a way to get in-depth responses to specific assertions. One
person called the ability to see and respond to the comments of others a
source of checks and balances in the system.

[10. Overall Costs & Benefits]{.underline}

The groups highlighted the many unknowns with regard to the costs and
benefits of electronic rulemaking. Will the FDMS work? Will anyone use
it? Will it reduce or increase the costs and burden hours devoted to
sorting through information, or the comments on comments? Will it lead
to quicker promulgation of rules, and should it? How much will it
actually cost? Will it facilitate the proliferation of "me, too" votes,
or bring the wisdom of more diverse opinion to bear? As was the case for
so many of the issues raised in this report, there was no overarching
consensus, except to say, as one person did, that "we just don't know
yet."

It was suggested that cost savings ought to be less of a factor if the
money spent resulted in much better information services. The reduction
of manual labor and content analysis subcontracting was identified as an
avenue of real savings. One nongovernmental participant, however,
thought it would be a significant cost to monitor discussion threads and
rebuttals to be able to get the "last word" in. Another spoke of the
privacy cost if companies used the new tools to gather data on citizens.
A consistent concern was expressed about the misuse of tools designed to
enhance the process.

The reduction of paper and potential ease of access were noted
regularly. An ability to produce and share a better legal record was
considered a plus. One representative of a group with 8,000 scientist
members noted a steady increase in the willingness of scientists to
engage in the rulemaking process. Another stated that the focus on
E-Government spent too little time dealing with the digital divide.[^46]
There was a potential benefit noted in tracking visitors to the site and
how they use the materials and tools that were made available.
Electronic rulemaking, remarked one person, would allow the agencies to
communicate better with and understand the regulated community. To
fulfill this promise, however, the agencies would need to commit to
substantial outreach to stakeholders and the general public. It was
suggested that there is a need to go well beyond "notice and comment"
rulemaking, and that the Administrative Procedure Act of 1946 was "out
of date."

**Conclusion**

Much of what has been written and said about electronic rulemaking over
the last three years has been forward looking and highly speculative.
The dearth of empirical studies and agreed-upon metrics were
characteristic of rulemaking scholarship prior to the emergence of
electronic government.[^47] We continue to sort out exactly what to
measure and how best to measure it. Simultaneously, developers of the
FDMS are determining what to build and how to fund it. This report
represents another step in the maturation of both processes. While
questions remain unresolved, several likely scenarios are unfolding. At
first glance, some of these seem mutually exclusive, but in fact they
are more often complimentary and may be sustainable over time.

Electronic rulemaking may fundamentally transform the process or it may
simply digitize established paper-based processes. Evidence gathered at
this series of focus groups supports the latter conclusion. The pace of
procedural and technological change is slow due to considerable inertia
and territoriality among both governmental and nongovernmental actors.
Despite idealism among advocates of a highly engaged digital democracy,
people and institutions tend to do what they've always done. If there is
a stamp-free method to register a practically effortless preference on a
favorite issue, such as the roadless conservation areas or hours of
service for truckers, many citizens will take the chance to cast what
they many times perceive as a vote for a favorable decision, often as
defined and scripted by their interest group of choice.

Moreover, citizens are wise to the fact that that both the prevailing
architecture of first generation electronic government and practices in
administrative rulemaking impede truly deliberative opportunities. In
addition, many submitters of comments prefer not to have their comments
rebutted and therefore submit on the last day or, with the advent of the
electronic mode, even at the last minute. The anti-deliberative effect
of this practice might be mitigated through various technological or
procedural innovations, but how would this then affect citizens'
willingness to commit the extra time to provide input and consider
feedback? We know very little about how the behavior of affected groups
will change when (and if) duplicate detection algorithms or moderated
on-line dialogues emerge as regular features of electronic rulemaking.
It is safe to surmise that some individuals and groups will find the
value-added of advanced information and communications technologies to
submit more effective public comments. Others will look for new ways to
beat the system and a cat and mouse game may move from the paper-based
to the digital playing field.

Administrative law scholars worry about a perceived shift away from
agency discretion and expert decisions toward the politics and the
psychology of plebiscites. They are not alone. At a recent agency focus
group, one participant stressed: "Rulemaking is not a democracy." While
some NGOs seem committed to mass mailings, many agency personnel and
other observers believe these efforts aim to undermine the role of
expertise in public administration. Mass mailers may inadvertently
petition themselves into obscurity. The use of mass-mailing campaigns
during public comments periods, while occasionally effective, engenders
a degree of cynicism and resentment among those intended to be
influenced by such comments. As a result, the general level of respect
of agency personnel for citizen comments and values may decline over
time.

Despite the skepticism linked to mass submissions, electronic
rule-making will nonetheless result in at least some better quality
comments and perhaps more durable rules, alongside a proliferation of
quasi-spam in the form of automated junk-mail, that one-click cacophony
striving for a virtual direct democracy. Promising opportunities exist
for IT to facilitate, structure, and guide informed, meaningful public
input. Rule-writers agree that public comments can be more on point,
substantial, and manageable than they are currently. At stake is whether
the guiding or interactive structure will reside inside or outside the
federal government. Focus group participants made it clear that the best
thing developers of the FDMS can do to ensure innovation is to publish
an open application programming interface. The result will be a vibrant
marketplace for ideas, data, studies, and other comments relevant to the
rulemaking enterprise.

Two further outcomes are possible. One likely scenario is that IT will
increase the transparency and legitimacy associated with federal
rulemaking. Another is that less transparency will result as agencies
fail, for whatever reason, to document the interagency or *ex parte*
communications on which some critical decisions are based. Consider the
current debate over the role of OIRA review and early participation in
decisions about the scope of regulation. The Internet as potentially a
24-7 spotlight on agency activities may result in further blurring the
boundary between informal and formal interagency communications. With
formal communications subject to Freedom of Information Act (FOIA)
requests, government personnel may shift their communicative routine to
compensate for the enhanced transparency radar.

It should be obvious by now that the impact of information technology
and the Internet is limited. It cannot solve problems that are legal,
political, social, and economic in nature. However, it can provide more
choices and tools than ever before to those with the knowledge of
electronic opportunities to engage the rulemaking process. The playing
field is shifting and much is unsettled. Significant choices lie ahead
for the developers and users of the Federal Docket Management System,
Congress, and the courts. As Lawrence Lessig might say, how you build it
matters. Balancing the interests of regular users and other stakeholders
against the imperatives of governing a complex democratic society is no
easy task. To the extent we talk openly about the challenges and
opportunities, the democratic potential for innovative and equitable
solutions remains intact.

**Appendix A**

Focus Group Protocol: eRulemaking Workshop 2.0

June 2004

**{For the purpose of this discussion, we will use IT to denote the wide
range of information and communications technologies that are used in
eRulemaking}**

1.  What are some of the costs and benefits of applying IT to the
    rulemaking process?

2.  How can information technology improve the rulemaking process?

    a.  How do you define "improved" in the context of rulemaking?

3.  How do stakeholder groups currently use IT to realize the objectives
    of their organization?

    a.  How will this use likely change in light of the tools and
        capacities discussed during the presentations?

4.  What kinds of cultural changes do you think IT may facilitate or
    necessitate in stakeholder organizations?

    a.  Will IT result in a more collaborative rulemaking process?

5.  What specific things would you suggest to make electronic rulemaking
    more transparent, accessible, and effective?

6.  Will eRulemaking help \"level the playing field\" between
    large/small organizations and/or between urban/rural entities?

7.  Would government decision-making be improved if eRulemaking created
    the opportunity for a dialogue among stakeholders?

    a.  If so, how would you structure it?

8.  What role should IT play improving regulatory compliance?

9.  Is there anything else we should discuss that we have not addressed?

[^1]: See <http://www.diggov.org/search/projects/project.jsp?ID=140>

[^2]: The Kennedy School of Government workshop web site is:
    <http://www.ksg.harvard.edu/cbg/Conferences/rpp_rulemaking/home.htm>.
    Harvard University also hosts the most comprehensive clearinghouse
    of electronic rulemaking documents at:
    <http://www.ksg.harvard.edu/cbg/rpp/erulemaking/>.

[^3]: For more information on the project history, see
    [http://erulemaking.ucsur.pitt.edu](http://erulemaking.ucsur.pitt.edu/).

[^4]: The GWU Workshop page can be found at: <http://snipurl.com/7hh5>.

[^5]: Carnegie Mellon University hosts the rulemaking text data testbed,
    which serves as a clearinghouse for researchers, agency personnel,
    and the public. Large public comment data sets (e.g., USDA's organic
    or DOT's CAFE rulemakings) are available to download and search
    using test versions of new human language technologies. See
    <http://hartford.lti.cs.cmu.edu/callan/Projects/ERulemaking/Data.html>.

[^6]: See <http://erulemaking.ucsur.pitt.edu/group_report.htm>

[^7]: For survey access and results, see:
    <http://erulemaking.ucsur.pitt.edu/survey.htm>.

[^8]: See <http://erulemaking.ucsur.pitt.edu/grant_proposal.htm>.

[^9]: Stephen Johnson. 1998. "The Internet Changes Everything:
    Revolutionizing Public Participation and Access to Government
    Information through the Internet," *Administrative Law Review, 50*:
    277-337.

[^10]: Cary Coglianese. 2004. "E-Rulemaking: Information Technology and
    Regulatory Policy," Regulatory Policy Program Report No. RPP-05.
    Available at: <http://snipurl.com/82pi>.

[^11]: Beth Noveck. 2004. "The Electronic Revolution in Rulemaking,"
    which can be downloaded at <http://snipurl.com/88jf>. Noveck asserts
    that "this radical overhaul of the administrative process is
    conducted in a closed and almost secretive manner without public
    consultation."

[^12]: Mordecai Lee. 2004. "E-Reporting: Strengthening Democratic
    Accountability," *The Business of Government Sixth Anniversary
    Issue*. See
    [www.businessofgovernment.org](http://www.businessofgovernment.org/).

[^13]: See <http://www.whitehouse.gov/omb/egov/> for updated information
    on the Federal E-Government Initiatives. The Congress has been
    critical of limited progress while appropriating insubstantial sums
    for cross-agency efforts. On the progress to date, see the GAO
    report "Electronic Government: Initiatives Sponsored by the Office
    of Management and Budget Have Made Mixed Progress"
    [GAO-04-561T](http://www.gao.gov/new.items/d04561t.pdf).

[^14]: Stephen Zavestoski and Stuart W. Shulman. 2002. "The Internet and
    Environmental Decision-Making," *Organization and Environment*,
    15(3): 323-327; Robert D. Carlitz and Rosemary W. Gunn. 2002.
    "Online Rulemaking: A Step Toward E-Governance," *Government
    Information Quarterly, 19*: 389-405; Stuart W. Shulman. 2003. \"An
    Experiment in Digital Government at the United States National
    Organic Program,\" *Agriculture and Human Values, 20*(3): 253-265.

[^15]: See <http://www.regulations.gov/eRuleMaking.cfm>.

[^16]: All of the workshop presentations are collected at:
    <http://erulemaking.ucsur.pitt.edu/talks.htm>.

[^17]: See <http://erulemaking.ucsur.pitt.edu/workshop.htm> regarding
    the history of the workshops. On the irregular nature of rulemaking
    within and across agencies, see the GAO report "Regulatory Reform:
    Procedural and Analytical Requirements in Federal Rulemaking"
    [GAO/T-GGD/OGC-00-157](http://www.gao.gov/new.items/g800157t.pdf).

[^18]: See <http://www.whitehouse.gov/omb/egov/pres_state.htm> on the
    E-Government Act and
    <http://www.whitehouse.gov/omb/budget/fy2002/mgmt.pdf> for the
    President's Management Agenda.

[^19]: Ex., see <http://www.ctsg.com/>, <http://www.getactive.com/>, or
    <http://capitoladvantage.com/>.

[^20]: For example, recent Environmental Protection Agency (EPA) notices
    about possible rulemakings in the Federal Register concerning the
    definition of U.S. waters and mercury emissions each have resulted
    in hundreds of thousands of emailed public comments. At this
    writing, about 15 EPA EDOCKET staff members are sorting through over
    500,000 comments, manually identifying the unique comments for
    inclusion in the docket. So far, these amount to about 3,400
    distinctive comments.

[^21]: See <http://snipurl.com/7uz4> for an example of near duplicate
    detection.

[^22]: See [www.Regulations.Gov](http://www.Regulations.Gov/).

[^23]: See [http://www.uschamber.com](http://www.uschamber.com/) for the
    U.S. Chamber.

[^24]: See <http://www.nfib.com/page/home.html> for the NFIB

[^25]: See <http://www.aflcio.org/> for the AFL-CIO and
    <http://www.nacredata.com/> for Nacredata.

[^26]: See <http://www.info-ren.org/> for Information Renaissance and
    <http://www.eff.org/> for the EFF.

[^27]: "Short for RDF Site Summary or Rich Site Summary; an XML format
    for syndicating Web content. A Web site that wants to allow other
    sites to publish some of its content creates an RSS document and
    registers the document with an RSS publisher. A user that can read
    RSS-distributed content can use the content on a different site.
    Syndicated content includes such data as news feeds, events
    listings, news stories, headlines, project updates, excerpts from
    discussion forums or even corporate information. " Source:
    <http://snipurl.com/89bl>.

[^28]: See <http://www.naspaa.org/> for NASPAA and
    <http://www.dpb.state.va.us/> for the Virginia Department of
    Planning and Budget.

[^29]: See <http://www.dot.gov/> for the USDOT and
    <http://www.law.columbia.edu/> Columbia Law School.

[^30]: Peter Strauss. 2004. "The ABA's Ad Law Section's E Rulemaking
    Survey," *Administrative Law and Regulatory News* 29(*3*): 8-10.

[^31]: The role of the Office of Information and Regulatory Affairs
    (OIRA) within the Office and Management and Budget (OMB), described
    in a recent General Accounting Office report as "not well documented
    or well understood," has come under greater scrutiny in this regard.
    See the GAO report "Rulemaking: OMB's Role in Reviews of Agencies'
    Draft Rules and the Transparency of Those Reviews"
    [GAO-03-929](http://www.gao.gov/new.items/d03929.pdf).

[^32]: In separate focus groups with EPA officials in late July 2004,
    participants remarked that the EPA's current EDOCKET system lacked
    many of the rudimentary document categorization and indexing
    principles that were routine in the paper docket era. Several
    mentioned the irony that the move to a rigid electronic repository
    without categories or adequate navigational tools had been a step
    backward. Personal communications with the managers of the
    eRulemaking Initiative suggest they recognize the limits of EPA's
    current EDOCKET system.

[^33]: It is possible that IT will succeed where legislative reforms
    have fallen short, such as in the effort to reduce the paperwork
    burden. See the GAO report "Federal Rulemaking: Procedural and
    Analytical Requirements at OSHA and other Agencies"
    [GAO-01-852T](http://www.gao.gov/new.items/d01852t.pdf).

[^34]: The recent Final "Report of the Small Business Paperwork Relief
    Act Task Force," echoed this concern when it outlined a number of
    information access needs. Among these is the ability to search the
    full volume of Federal information by subject or task, rather than
    through older structures based on agency organization or structure.
    See <http://www.whitehouse.gov/omb/inforeg/sbpr2004.pdf>.

[^35]: For more on improving the quality of public comments, see Barbara
    H. Brandon and Robert D. Carlitz. 2002. "Online Rulemaking and Other
    Tools for Strengthening Civic Infrastructure," *Administrative Law
    Review* 54(*4*), 1421-1478.

[^36]: A leading rulemaking scholar asserted the "overall trend has been
    away from the expertise model and toward the politics model," as the
    comments submitted increase by orders of magnitude. See Jeffrey S.
    Lubbers (ed.) *Developments in Administrative Law and Regulatory
    Practice 2002-2003* (Chicago, IL: American Bar Association),
    149-150.

[^37]: U.S. DOT officials have explained in other interviews that an
    unmistakable benefit derives from public comments that call
    attention to overlooked data, studies, or novel concepts. In the
    case of the Corporate Average Fuel Economy (CAFE) standards, a well
    thought-out public comment resulted in the addition of a missing
    benefit in the critically important cost-benefit analysis. This is
    perhaps suggestive of a future direction for e-advocacy. Instead of
    60,000 form letters, a campaign might use IT to foster a massive
    brainstorming session that raises, debates, and distills the best
    ideas a constituency can offer collectively.

[^38]: See <http://www.urbansim.org/>.

[^39]: In truth, most agencies farm out to the private sector or special
    internal branches of government, such as the USFS Content Analysis
    Team, the task of sorting and doing content analysis on the large
    batches of public comments. New tools for accessing and analyzing
    the docket are already in use. An interesting question is whether
    the public is aware of the extent to which contractors are involved
    in the process.

[^40]: Respondents to the web survey also identified value added
    features that would cluster and rank frequently viewed comments or
    supporting documents.

[^41]: There has been evidence of agency resistance to cross-agency
    standardization for some time. See the GAO letter in response to
    Congressman Henry Waxman and Senator Joseph Lieberman on the subject
    "Federal Rulemaking: Agencies' Use of Information Technology to
    Facilitate Public Participation"
    [GAO/GGD-00-135R](http://www.gao.gov/archive/2000/gg00135r.pdf).
    More recent signs have emerged in the House appropriations process.
    Some individual agencies balk at the idea of joining a centralized
    docket management system. See Jason Miller. 2004. "House Plans E-Gov
    Cuts" *Government Computing News* (June 28).

[^42]: In the case of the Advanced Notice of Proposed Rulemaking (ANPR)
    EPA issued related to issues associated with the scope of waters
    that are subject to the Clean Water Act in light of the U.S. Supreme
    Court decision in Solid Waste Agency of Northern Cook County
    (SWANCC) *v.* U.S. Army Corps of Engineers, the total volume of
    public comment was discussed in a White House West Wing meeting on
    whether to go ahead with the rulemaking (Source: EPA focus group,
    July 26, 2004).

[^43]: Dr. Jamie Callan has developed a rudimentary near-duplicate
    detection tool that, if incorporated into the FDMS, will simplify
    and improve the accuracy of efforts to cull the distinctive
    additions to form letters that count as "form-plus" in the USFS
    lexicon. See the recent example at: <http://snipurl.com/858i>. Since
    the Administrative Procedure Act requires that the agency address
    every substantive comment, this increased ability to highlight and
    review near-duplicates may lessen the likelihood that meaningful
    additions to form letters are sorted out of consideration by
    accident.

[^44]: In other venues, agency personnel have expressed concern about
    losing an otherwise carefully prepared rule in the courts based on
    failure to address substantive comments that are lost amongst
    thousands of modified form letters.

[^45]: Depending on whom you ask, the USFS either floated or had a
    free-wheeling staff member let slip the idea that electronic mass
    submission could be eliminated. See: <http://snipurl.com/858y>.

[^46]: A recent report from the American Political Science Association
    Task Force on Inequality and American Democracy stated "the Internet
    may 'activate the active' and widen disparities between participants
    and the politically disengaged by making it easier for the already
    politically engaged to gain political information." See "American
    Democracy in an Age of Rising Inequality" at
    <http://snipurl.com/7egy>.

[^47]: For more on the nature of rulemaking and the trends in its
    scholarship, see Cornelius M. Kerwin. 2003. *Rulemaking: How
    Government Agencies Write Law and Make Policy* 3^rd^ Ed. (CQ Press:
    Washington, DC).