REPORT FOR THE
A DMINISTRATIVE C ONFERENCE OF THE U NITED S TATES

MASS, COMPUTER-GENERATED, AND FRAUDULENT
COMMENTS
Steve Balla
George Washington Regulatory Studies Center
Reeve Bull
Administrative Conference of the United States
Bridget Dooling
George Washington Regulatory Studies Center
Emily Hammond
George Washington Law School
Michael Herz
Cardozo School of Law
Michael Livermore
University of Virginia School of Law
&
Beth Simone Noveck
The Governance Lab

This report was prepared for the consideration of the Administrative Conference of the United States. It does not
necessarily reflect the views of the Conference (including its Council, committees, or members).
Recommended Citation
Steve Balla, Reeve Bull, Bridget Dooling, Emily Hammond, Michael Herz, Michael Livermore, & Beth Simone
Noveck, Mass, Computer-Generated, and Fraudulent Comments (June 1, 2021) (report to the Admin. Conf. of the
U.S.).

1

Executive Summary
This report explores three forms of commenting in federal rulemaking that have been
enabled by technological advances: mass, fraudulent, and computer-generated comments. Mass
comments arise when an agency receives a much larger number of comments in a rulemaking than
it typically would (e.g., thousands when the agency typically receives a few dozen). The report
focuses on a particular type of mass comment response, which it terms a “mass comment
campaign,” in which organizations orchestrate the submission of large numbers of identical or
nearly identical comments. Fraudulent comments, which we refer to as “malattributed comments”
as discussed below, refer to comments falsely attributed to persons by whom they were not, in fact,
submitted. Computer-generated comments are generated not by humans, but rather by software
algorithms. Although software is the product of human actions, algorithms obviate the need for
humans to generate the content of comments and submit comments to agencies.
This report examines the legal, practical, and technical issues associated with processing
and responding to mass, fraudulent, and computer-generated comments. There are cross-cutting
issues that apply to each of these three types of comments. First, the nature of such comments may
make it difficult for agencies to extract useful information. Second, there are a suite of risks related
to harming public perceptions about the legitimacy of particular rules and the rulemaking process
overall. Third, technology-enabled comments present agencies with resource challenges.
The report also considers issues that are unique to each type of comment. With respect to
mass comments, it addresses the challenges associated with receiving large numbers of comments
and, in particular, batches of comments that are identical or nearly identical. It looks at how
agencies can use technologies to help process comments received and at how agencies can most
effectively communicate with public commenters to ensure that they understand the purpose of the
notice-and-comment process and the particular considerations unique to processing mass comment
responses. Fraudulent, or malattributed, comments raise legal issues both in criminal and
Administrative Procedure Act (APA) domains. They also have the potential to mislead an agency
and pose harms to individuals. Computer-generated comments may raise legal issues in light of
the APA’s stipulation that “interested persons” are granted the opportunity to comment on
proposed rules. Practically, it can be difficult for agencies to distinguish computer-generated
comments from traditional comments (i.e., those submitted by humans without the use of software
algorithms).
While technology creates challenges, it also offers opportunities to help regulatory officials
gather public input and draw greater insights from that input. The report summarizes several
innovative forms of public participation that leverage technology to supplement the notice and
comment rulemaking process.
The report closes with a set of recommendations for agencies to address the challenges and
opportunities associated with new technologies that bear on the rulemaking process. These
recommendations cover steps that agencies can take with respect to technology, coordination, and
docket management.

2

Introduction
In 2015, the Environmental Protection Agency (EPA) issued a rule defining the “Waters
of the United States.” During the course of the rulemaking, the EPA received more than one million
comments.1 Over ninety percent of the comments were submitted as part of mass comment
campaigns. At about the same time, a large number of mass comment campaigns was also
submitted in response to the EPA’s proposed “Clean Power Plan.” These and other similar
occurrences signaled the regularity of mass comment campaigns, at least in the context of highly
salient, controversial rulemakings.
In February 2020, the Subcommittee on Oversight and Investigations of the U.S. House of
Representatives Committee on Financial Services investigated allegations that the Securities and
Exchange Commission (SEC) may have received fraudulent comments from “an Army veteran, a
Marine veteran, a single mom, and a couple of retirees.”2 During the committee hearing, witnesses
alleged that those public comments were sent by advocacy groups and signed with the names of
people who never saw the comments or who did not exist at all. Although these allegations have
been disputed, the possibility of fraudulent comments in the government’s rulemaking process has
prompted questions and concern.
In 2017, the Federal Communications Commission (FCC) “Restoring Internet Freedom”
(i.e., net neutrality) rulemaking attracted a record number of public comments: almost 22 million
by the official close of the comment period, with another three million arriving after the fact.3
Although about six percent of the comments were unique, the rest were submitted multiple times,
in some cases hundreds of thousands of times.4 On nine different occasions, more than 75,000
comments were dumped into the docket at the very same second.5 The comments “included
comments from stolen email addresses, defunct email accounts and people who unwittingly gave

1

See Steven J. Balla et al., Lost in the flood?: Agency Responsiveness to Mass Comment Campaigns in Administrative
Rulemaking, REGUL. & GOVERNANCE (May 20, 2020), https://onlinelibrary.wiley.com/doi/abs/10.1111/rego.12318;
Rachel Augustine Potter, More than Spam?: Lobbying the EPA through Public Comment Campaigns, BROOKINGS
INST. (Nov. 19, 2017), https://www.brookings.edu/research/more-than-spam-lobbying-the-epa-through-publiccomment-campaigns/.

2

Fake It Till They Make It: How Bad Actors Use Astroturfing to Manipulate Regulators, Disenfranchise Consumers,
and Subvert the Rulemaking Process: Hearing Before the H. Fin. Servs. Comm., 116th Congress (Feb. 6, 2020)
(hereinafter Fake It Till They Make It).

3

In 2015, the FCC issued its “net neutrality rule,” prohibiting broadband Internet providers from blocking, degrading,
or interfering with Internet traffic. Protecting and Promoting the Open Internet, 80 Fed. Reg. 19737 (June 12, 2015).
The rule was upheld in United States Telecomm. Ass’n v. FCC, 825 F.3d 674 (D.C. Cir. 2016). In 2017, the FCC
proposed repealing the 2015 rule. Restoring Internet Freedom, Notice of Proposed Rulemaking, 32 FCC Rcd 4434
(2017). The Notice of Proposed Rulemaking was released in May and published in the Federal Register in June. 82
Fed. Reg. 25,568 (June 2, 2017). The Final Rule was promulgated in early January 2018. See Restoring Internet
Freedom, 83 Fed. Reg. 7852 (Feb. 22, 2018). For information on the comments, see Paul Hitlin, Kenneth Olmstead,
& Skye Toor, Public Comments to the Federal Communications Commission About Net Neutrality Contain Many
Inaccuracies and Duplicates, PEW RES. CTR. (Nov. 29, 2017), https://www.pewresearch.org/internet/wp-content/
uploads/sites/9/2017/11/PI_2017.11.29_Net-Neutrality-Comments_FINAL.pdf.

4

Id.

5

Id.

3

permission for their comments to be posted.”6 A consulting firm later determined that about a third
of the comments were sent from temporary or disposable email domains, and about 10 million
were from senders of multiple comments.7 FCC Commissioner Jessica Rosenworcel has stressed
that 500,000 or so comments came from Russia.8 The New York Attorney General concluded that
9.3 million comments were submitted with false identities, including 7 million from a single
submitter.9 In sum, the nature of public participation—the mass occurrence of identical and near
duplicate comments, the malattribution of identities, and the apparent automation of comment
submission—called into question elements of the process behind the FCC’s regulation.
Partly in response to issues posed by such rulemakings, in 2019 the U.S. Senate Permanent
Subcommittee on Investigations issued a staff report entitled, “Abuses of the Federal Notice-andComment Rulemaking Process.”10 The report identified problems associated with mass,
fraudulent, and computer-generated comments, including a lack of agency processes and policies
aimed at identifying, managing, and addressing such comments.
This attention underscores the importance of comments for the rulemaking process, which
generates thousands of regulations every year that touch many aspects of economic and social
life.11
The Administrative Conference of the United States (ACUS) has also been tracking these
issues for several years. In 2018, ACUS held a forum addressing mass and fraudulent comments.
At the time, the general consensus of the panelists, which included prominent academics with
6

James V. Grimaldi & Paul Overburg, Millions of People Post Comments on Federal Regulations. Many Are Fake.,
WALL ST. J. (Dec. 13, 2017), https://www.wsj.com/articles/millions-of-people-post-comments-on-federalregulations-many-are-fake-1513099188. A video on the newspaper’s website summarizes: “[T]he Wall Street Journal
uncovered thousands of comments from fake email addresses, abandoned or defunct email accounts, posted on behalf
of unwitting participants. For example, 818,000 identical comments on the FCC site favor repealing the rules. In a
random sample of people whose emails were used for those posts, 72% said they had nothing to do with them. Jack
Hirsch was one of them. “I was horrified. Knowing that this is actually an issue that I cared enough to write my
representatives about, and knowing that my information had been falsified to support a completely opposing view, it
was really frustrating, and honestly, I felt like there was no recourse.” Thousands of Fake Comments on Net Neutrality:
A WSJ Investigation, WALL ST. J. (Dec. 12, 2017, 12:02 PM), https://www.wsj.com/video/thousands-of-fakecomments-on-netneutrality-a-wsj-investigation/8E52172E-821C-4D89-A2AA-2820F30B8648.html.

7

FCC Restoring Internet Freedom Docket 17-108: Comments Analysis, EMPRATA 2 (Aug. 30, 2017),
https://www.emprata.com/emp2017/wp-content/uploads/2017/08/FCC-Restoring-Internet-Freedom-CommentsAnalysis.pdf.

8

Nicholas Confessore and Jeremy Singer-Vine on Request for Inspection of Records, 33 FCC Rcd. 11808 (adopted
Nov. 7, 2018) (Rosenworcel, dissenting).

9

N.Y. STATE OFF. OF THE ATTORNEY GENERAL LETITIA JAMES, FAKE COMMENTS: HOW U.S. COMPANIES &
PARTISANS HACK DEMOCRACY TO UNDERMINE YOUR VOICE (2021). See also Letter from Eric Schneiderman, Att’y
Gen., N.Y., to Thomas M. Johnson, Jr., Gen. Counsel, FCC (Dec. 13, 2017), https://ag.ny.gov/sites/default/files/ltr_t
o_fcc_gen_counsel_re_records_request.pdf (noting 8 million comments filed under false identities).

10

PERMANENT SUBCOMMITTEE ON INVESTIGATIONS, U.S. SENATE COMMITTEE ON HOMELAND SECURITY AND
GOVERNMENT AFFAIRS, STAFF REPORT, ABUSES OF THE FEDERAL NOTICE-AND-COMMENT RULEMAKING PROCESS
(2019),
https://www.hsgac.senate.gov/imo/media/doc/2019-10-24%20PSI%20Staff%20Report%20%20Abuses%20of%20the%20Federal%20Notice-and-Comment%20Rulemaking%20Process.pdf
(hereafter
“Subcommittee Report”).

11

CONG. RSCH. SERV., R43056, COUNTING REGULATIONS: AN OVERVIEW OF RULEMAKING, TYPES OF FEDERAL
REGULATIONS, AND PAGES IN THE FEDERAL REGISTER (2019).

4

expertise in rulemaking and governmental officials working in the rulemaking space, was that
agencies were mostly well-equipped to handle mass comments and that fraudulent comments,
though unseemly, were relatively uncommon and didn’t merit any extensive efforts on the part of
agencies to combat them. Deputy Office of Information and Regulatory Affairs (OIRA)
Administrator Dominic Mancini emphasized that mass comment responses were rare.12 And
Deputy Assistant Attorney General Matthew Miner noted that a criminal prosecution for
submitting false information to the government could be difficult to sustain and potentially raise
First Amendment concerns.13
Under the 1946 APA, the public has a right to participate by commenting on draft
regulations in the rulemaking process, which is why it is often referred to as notice-and-comment
rulemaking.14 Such participation in rulemaking enhances both the legitimacy and the quality of
regulations by enabling agencies (and the executive offices and congressional committees that
oversee them) to obtain information from a wide audience of stakeholders, interest groups,
businesses, nongovernmental organizations (NGOs), academics, and interested individuals.
Participation also provides an accountability check on the rulemaking process by ensuring public
scrutiny prior to rules going into effect. Overall, the process facilitates the reason-giving
requirement that is necessary for a rule to survive judicial review.
The shift over the last two decades to a digital process, in which participants submit
comments via the Internet, has made commenting easier. Expanding the ability to participate in an
important governance process is an overwhelmingly positive change. Rulemaking has long been
criticized as an insiders’ game, invisible to the general public, with modest levels of participation,
almost entirely by organizations rather than individuals. But, although online participation has the
potential to increase the quantity and diversity of participation, it has also inadvertently opened
the floodgates to mass comment campaigns, malattributed comments, and computer-generated
comments, potentially making it harder for agencies to extract the information needed to inform
decision making and undermining the legitimacy of the rulemaking process.
As this discussion indicates, the technology of electronic commenting has enabled three
kinds of potentially problematic comments that are the subject of this report:15
1. Mass comments (sometimes orchestrated as a campaign by one or more entities);
2. Fake, fraudulent, or what this report below refers to as “malattributed” comments; and

12

Dominic Mancini, Deputy Adm’r, Off. of Info. & Reg. Affs., Keynote Address at the Administrative Conference
of the United States Symposium on Mass and Fake Comments in Agency Rulemaking 14 (Oct. 5, 2018) (transcript
available
at
https://www.acus.gov/sites/default/files/documents/10-518%20Mass%20and%20Fake%20Comments%20in%20Agency%20Rulemaking%20Transcript.pdf).

13

Matthew Minor, Deputy Assistant Att’y Gen., Dep’t of Justice, Remarks at the Administrative Conference of the
United States Symposium on Mass and Fake Comments in Agency Rulemaking 113-14 (Oct 5, 2018) (transcript
available
at
https://www.acus.gov/sites/default/files/documents/10-518%20Mass%20and%20Fake%20Comments%20in%20Agency%20Rulemaking%20Transcript.pdf).

14

CONG. RSCH. SERV., supra note 11.

15

Journalistic and popular attention has focused on comments that fall into all three categories simultaneously, i.e.
mass computer-generated malattributed comments. But these three distinct characteristics do not necessarily coincide.
Each presents distinct practical and normative issues. While this report examines all three types of comments, it is
careful to disaggregate them and take into account the important ways in which they differ. These three types of
comments are defined in further detail below. See section II.A.

5

3. Computer-generated comments.
All three of these types of comments can generate serious challenges to agencies, raising a
pressing set of questions concerning how best to respond while ensuring the functioning of the
informal rulemaking process. The task of this report, therefore, is to evaluate whether and to what
extent such submissions are problematic, and to make recommendations for how rulemaking
agencies should respond using legal, policy, and technological strategies.
Our overarching conclusion is that agencies should adopt both low- and high-tech measures
to limit the negative impact of these sorts of comments. Mass, malattributed, and computergenerated comments, however, do not represent a crisis for the regulatory state at this time. They
have not been found to violate federal law and do not generally undermine the integrity of noticeand-comment rulemaking, and we are not aware of evidence of widespread substantive harms in
particular rulemaking efforts or to the rulemaking system overall. However, appropriate responses,
especially those that take advantage of new technology, could reduce the cost and negative impacts
of technology-enabled comments.
Adopting such techniques could, for example, afford agency officials more time to improve
the opportunity for a diverse public to participate in the rulemaking process meaningfully and to
augment current practices with new forms of citizen engagement. Indeed, in addition to exploring
how new technologies–the very same technologies that enable mass, fraudulent and computergenerated comments–can also help with analyzing those comments, we also explore throughout
how technology can help regulatory officials make sense of public input and draw greater insights
from public comments of all kinds. Finally, other jurisdictions at the state and local level and
internationally are turning to the use of new technology to enable innovative forms of public
participation to improve the quality of rule- and policymaking. As detailed in Part VI, these
illustrate hopeful opportunities for future experimentation.
Written by seven professionals with expertise in administrative law, rulemaking practice,
and new technologies, this report is informed by a review of the Permanent Subcommittee’s
report,16 relevant law, related legal and social scientific scholarship, and a set of interviews with
agency personnel with background in the rulemaking process at agencies with substantial
rulemaking dockets during the summer and fall of 2020. The interviews, which were not meant to
capture the views of a representative or random sample, were with staff of the EPA, the Consumer
Finance Protection Bureau (CFPB), the Department of Transportation (DOT), and the FCC, as
well as with officials from the General Services Administration (GSA) responsible for developing
and maintaining the Federal Docket Management System (FDMS). A technical advisory group of
experts drawn from government, private industry, and academia also provided feedback to the
report authors, as did an additional online roundtable of agency officials with experience in the
notice-and-comment process.
The report is divided seven parts. Part I provides a general introduction to notice-andcomment rulemaking and the role of technology in that process. Part II discusses recent
technological development that have contributed to the growth of mass, malattributed, and
computer-generated comments, and describes some of the challenges associated with these types

16

The Subcommittee Report addressed a handful of additional topics, including obscenity and copyrighted materials
in public comments, that are not addressed in the current report. ACUS will provide a copy of the Subcommittee
Report to the Committee on Rulemaking as they begin deliberations on this project.

6

of comments. Parts III, IV, and V focus on each of these comment types in turn. Part VI discusses
technological opportunities, with a focus on current available tools that can be used to facilitate
the processing of information from the notice-and-comment process or enhance supplements to
the notice-and-comment process. Part VII offers draft recommendations based on the findings in
this report.
I.

Technology and Notice-and-Comment Rulemaking

During the latter half of the twentieth century, there was considerable growth in the use of
informal rulemaking by administrative agencies.17 The procedures for informal rulemaking are set
out in section 553 of the APA. A cornerstone of this process is the opportunity for members of the
public to submit comments on rulemaking proposals,18 which is why it is often referred to as
notice-and-comment rulemaking. For decades, organizations and individuals have availed
themselves of this opportunity to help inform the process of regulatory development. Public
comments on agency rulemakings take a wide variety of forms that include detailed submissions
by sophisticated repeat players, short expressions of support or opposition from members of the
public, signed form letters in response to solicitations from NGOs, and technical reports from
unaffiliated experts.19
There is considerable variation in the level of public participation from one rulemaking to
another. The vast majority of rulemakings are relatively unremarked upon by the public, with—at
most—participation by the stakeholders most affected by a rule.20 This level of participation is not
surprising given the often highly technical and specialized nature and low visibility of many
rulemakings. While federal agencies publish the opportunity to participate in the Federal Register
(effectively, the newspaper of the federal government), they generally do not advertise
rulemakings elsewhere and the public tends to have little knowledge of the right to engage unless
a third party promotes the opportunity. In a small percentage of well-publicized rulemakings with
particular public salience—such as those highlighted above—public participation can be orders of
magnitude above the norm, with the number of comments ranging from thousands to millions.

17

See Christopher DeMuth, Can the Administrative State Be Tamed?, 8 J. LEGAL ANALYSIS 121, 126–27 (2016).

18

Cary Coglianese, Citizen Participation in Rulemaking: Past, Present, and Future, 55 DUKE L.J. 943, 945 (2006)
(“participation in rulemaking is one of the most fundamental, important, and far-reaching of democratic rights”).

19

See Michael A. Livermore, Vladimir Eidelman & Brian Grom, Computationally Assisted Regulatory Participation,
93 NOTRE DAME L. REV. 977 (2018).

20

David M. Shafie, Participation in E-Rulemaking: Interest Groups and the Standard-Setting Process for Hazardous
Air Pollutants, 5 J. INFO. TECH. & POL. 399, 403–405 (2008). See also Stuart Shapiro, When Do Agencies Change
Their Proposed Rules (Nov. 10, 2007) (presented at APPAM: Association for Public Policy Analysis & Management
Conference) (unpublished manuscript), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1026066 (finding that
fewer than 10% of rulemakings received more than 100 comments); John M. de Figueiredo, E-Rulemaking: Bringing
Data to Theory at the Federal Communications Commission, 55 DUKE L.J. 969 (2006). Other agency actions that are
subject to public commenting follow as similar pattern. An analysis focusing on the fifty draft guidance documents
published for public comment by the Environmental Protection Agency (EPA) from 2011-2014 found that just eight
received more than 5,000 comments, with five exceeding 40,000 comments. Nicholas R. Parrillo, Should the Public
Get to Participate Before Federal Agencies Issue Guidance? An Empirical Study, 71 ADMIN. L. REV. 57, 97-98 (2019)
[hereinafter Parrillo, Should the Public Get to Participate]; Nicholas R. Parrillo, Federal Agency Guidance: An
Institutional Perspective (Oct. 12, 2017) (report to the Admin. Conf. of the U.S.).

7

The APA sets forth the key elements of notice-and-comment rulemaking. Subject to certain
exceptions, agencies first must publish a general notice of the proposed rulemaking in the Federal
Register.21 That notice “shall include—
(1) a statement of the time, place, and nature of public rule making proceedings;
(2) reference to the legal authority under which the rule is proposed; and
(3) either the terms or substance of the proposed rule or a description of the subjects and
issues involved.”22
The APA further provides that “[a]fter notice required by this section, the agency shall give
interested persons an opportunity to participate in the rule making through submission of written
data, views, or arguments with or without opportunity for oral presentation. After consideration of
the relevant matter presented, the agency shall incorporate in the rules adopted a concise general
statement of their basis and purpose.”23 “Person” is broadly defined to include “an individual,
partnership, corporation, association, or public or private organization other than an agency.”24
Consistent with the broad ideals underlying the commenting process, courts read this provision
expansively.25 Courts have also elaborated that the purpose of the notice requirement is to facilitate
meaningful comments.26 For example, agencies must disclose in their notice any scientific or
technical details on which they base their proposed rules in order to give the public a fair
opportunity to react and comment thereon.27
Agencies not only must provide notice and an opportunity for public comments but also
must then “consider[] . . . the relevant matter presented” in those comments.28 The courts have
interpreted this language to require that, in the notice of the final rule, agencies respond to
“significant” comments—those that, “if true, . . . would require a change in [the] proposed rule.”29
Failure to so respond is grounds for remand.30 Under § 706(2)(A) of the APA, agency rulemakings
will be set aside if they are “arbitrary, capricious, an abuse of discretion, or otherwise not in
accordance with law.”31 The failure to acknowledge and respond to substantive concerns raised in
21

5 U.S.C. § 553(b).

22

Id.

23

5 U.S.C. § 553(c).

24

5 U.S.C. § 551(2).

25

E.g., O’Rourke v. U.S. Dep’t of Justice, 684 F. Supp. 716, 718 (D.D.C. 1988) (holding “person” includes noncitizens and collecting cases); Neal-Cooper Grain Co. v. Kissinger, 385 F. Supp. 769, 776 (D.D.C. 1974) (foreign
government or instrumentality thereof is a “person”).

26

Portland Cement Ass’n v. Ruckelshaus, 486 F. 2d 375, 393 (D.C. Cir. 1976).

27

Id.

28

5 U.S.C. § 553(c).

29

Am. Mining Cong. v. EPA, 907 F.2d 1179, 1188 (D. C. Cir. 1990); Home Box Office, Inc. v. FCC, 567 F.2d 9, 3536 (D.C. Cir. 1977) (per curiam). Without such an obligations, courts have said, the opportunity to comment would
be “meaningless.” Id; see Carlson v. Postal Regulatory Comm’n, 938 F.3d 337, 351 (D.C. Cir. 2019); ACLU v. FCC,
823 F.3d 1554, 1581 (D.C. Cir. 1987); St. James Hosp. v. Heckler, 760 F.2d 1460, 1470 (7th Cir. 1985).

30

E.g., La. Fed. Land Bank Ass’n v. Farm Credit Admin., 336 F.3d 1975 (D.C. Cir. 2003).

31

5 U.S.C. § 706(2)(A).

8

the rulemaking process is one of the grounds for a court to find that an agency’s rulemaking fails
under the arbitrary or capricious standard.32
Although the requirement to respond to comments is serious, it is not absolute. The “APA
requirement of agency responsiveness to comments is subject to the common-sense rule that a
response [is not always] necessary.”33 Comments that “are purely speculative and do not disclose
the factual or policy basis on which they rest require no response.”34
In recognition of the potential for information and communication technologies to facilitate
broader participation in the regulatory process, Congress enacted the E-Government Act in
October 2002. Among other things, the George W. Bush administration established the eRulemaking Program to spearhead the creation of an online system for conducting the notice-andcomment process at agencies throughout the federal government.35 To the extent deemed
practicable, each agency must post information required to be published in the Federal Register
online, maintain online rulemaking dockets, and allow for electronic submission of comments
accepted under § 553(c). To better facilitate this public online access, regulations.gov was created
in January 2003.36 All executive agencies were required to join the e-Rulemaking initiative.37 As
of this writing, many but not all independent agencies also use regulations.gov for their
rulemakings; those that do not prominently include the FCC and the SEC.38
Filing comments, electronically or by mail, is not the only way that individuals and
organizations may participate in agency rulemaking. Agencies also occasionally hold public
hearings, consult with experts in advisory committees, and work with interest group stakeholders
in negotiated rulemakings. The Negotiated Rulemaking Act encouraged agencies to use a dispute
resolution process for soliciting stakeholder comment to enhance the informal rulemaking
process.39 Public participation also occurs after-the-fact through intervention in agency
adjudications, citizens and groups informal monitoring activities, and litigation. But notice-andcomment rulemaking remains a central mechanism for public participation in agency policy
making.

32

See, e.g., Bus. Roundtable v. SEC, 647 F.3d 1144, 1152 (D.C. Cir. 2011) (discussing agency failure to address cost
issues raised in comments).

33

NRDC v. EPA, 859 F.2d 156, 188–89 (D.C. Cir. 1988).

34

Home Box Office, Inc. v. FCC, 567 F.2d 9, 35, n.58 (D.C. Cir. 1977) (per curiam); see also Pub. Citizen, Inc. v.
FAA, 988 F.2d 186, 197 (D.C. Cir. 1993).

35

See E-Government Act § 206, 44 U.S.C. § 3501 note.

36

Regulations.gov was initially managed by the Environmental Protection Agency. The General Services
Administration (GSA) assumed the role of managing partner of the e-Rulemaking Program at the beginning of October
2019; see About the eRulemaking Initiative, REGULATIONS.GOV, https://www.regulations.gov/about (last visited Mar.
31, 2021).

37

See Cynthia R. Farina, Recent Development: Achieving the Potential: The Future of Federal E-Rulemaking, Report
of the Committee on the Status and Future, 62 ADMIN. L. REV. 279, 282 (2009).

38

A complete list of participating and nonparticipating agencies appears at https://www.regulations.gov/agencies.

39

Negotiated Rulemaking Act, 5 U.S.C. § 561 et seq. (1994) (the purpose of the Act is to encourage agencies to use
this innovative dispute resolution process for soliciting stakeholder comment to enhance the informal rulemaking
process). See also Admin. Conf. of the U.S., Recommendation 2017-2, Negotiated Rulemaking and Other Options for
Public Engagement, 82 Fed. Reg. 31039 (June 16, 2017).

9

The adoption of e-rulemaking by agencies, along with broader associated technological
developments, has led to fundamental changes in notice-and-comment rulemaking. What was once
a paper process that was difficult to access and generally dominated by a small number of repeat
players has become more visible and therefore more accessible. The elimination of barriers to
participation has brought with it meaningful change. Much of this is for the better. The move online
has increased participation, made for better-informed agencies, made rulemaking more
transparent, and provided commenters, stakeholders, and rulewriters within the agencies with
easier access to materials in the docket. It has also enabled certain forms of commenting that lack
obvious benefits and that may do affirmative harm. The remainder of this report examines these
opportunities and challenges.
II.

The New World of Technology-Enabled Comments

For much of its history, the APA’s notice-and-comment process involved a particular form of
commenting: an individual or entity with “data, views, or arguments” relevant to the draft rule
produced a bespoke comment that reflected that individual’s or entity’s expertise or concerns. One
person, one comment, and every comment made a unique contribution because it was from a
unique submitter.
The three sorts of technology-enabled comments addressed in this report do not fit this model.
Many commentators believe that this departure from the past is a problem. One example: the U.S.
Senate Permanent Subcommittee on Investigations’ 2019 report pointed to a variety of issues
associated with mass, malattributed, and computer-generated comments (in addition to
highlighting other problems associated with comments such as the inclusion of obscenity and
copyrighted materials).40 It saw these types of comments as contributing to “abuses” that “reduce[]
the effectiveness of the notice-and-comment process; cost[] taxpayer funds to mitigate; allow[]
identity theft-related crimes to go unaddressed; and leave[] the rulemaking process vulnerable to
disruptive activity.”41 This report offers, among other things, a response to some of the concerns
raised by the Staff Report and many other observers. The widespread outrage caused by FCC’s net
neutrality rulemaking—with its massive total number of comments as well as fake or
“malattributed” (a comment filed under the name of someone who did not in fact prepare, approve,
or submit it) and computer-generated comments—suggests that technology-enabled comments
may be a major problem but, as we shall explore, the challenges are surmountable.
We begin this section with a taxonomy, identifying the particular sort of commenting
activity that is our focus. We then turn to a discussion of the overall issues that these sorts of
comments raise. In the following sections, we consider each of the three types of comments in
particular.

40

Subcommittee Report, supra note 10.

41

Id. at 1.

10

A. Three Types of Technology-Enabled Comments
1. Mass Comments
In a relatively small number of rulemakings, agencies receive an unusually large number
of comments (e.g., hundreds or thousands, as opposed to the few dozen, or fewer, that are
typical).42 We designate such situations as a “mass comment response.” We do not define a specific
threshold for the number of comments to qualify as a mass comment response, as the threshold
will tend to vary from agency to agency and rule to rule. As a general matter, a “mass comment
response” will feature at least an order of magnitude increase (e.g., 10x) in the number of
comments received vis-à-vis a typical rulemaking for that agency.43 We use the term “mass
comment campaign” to refer to the special case of a mass comment response in which one or more
organizations has successfully urged a large number of individuals or groups to submit comments
to the agency or allow those organization(s) to submit comments in their names.44 Mass comment
responses and mass comment campaigns have grown in frequency and scope as information and
communication technologies including email and the internet have reduced the cost of
participating in the notice-and-comment process.
2. Malattributed Comments
Much of the outraged reaction to the net neutrality rulemaking focused not on the sheer
number or duplicativeness of the comments but on the fact that millions were submitted under
false names, purporting to be from someone who either did not exist or had no awareness of the
comment. We term comments falsely attributed to persons by whom they were not in fact
submitted “malattributed comments.”45
Malattributed comments are similarly facilitated by new technologies. Easy access to very
large data sets of personal information makes the task of malattributing comments much easier
than in the past. In addition, it is possible to automate the malattribution of comments, using simple
software applications coupled with publicly available information such as DMV listings or voter
registration data.

42

See supra note 18; Hearing of the Permanent Subcommittee on Investigations and the Subcommittee on Regulatory
Affairs and Federal Management, S. Homeland Sec. & Governmental Affs. Comm., October 24, 2019 (statement of
Dominic J. Mancini, Deputy Administrator, Off. of Info. & Regul. Affs.) (noting that internal analysis “suggested that
about 80 percent of proposed rules receive 10 or fewer comments”).

43

Admittedly, this definition is somewhat arbitrary. There is no accepted definition of a “mass comment response,”
and we adopt this working definition for purposes of this report based on our interviews with agency officials.

44

See Balla et al., supra note 1.

45

The more common term is “fraudulent comment.” For a more complete discussion of possible labels, including
“fraudulent,” “fake,” “pseudonymous,” “fabricated,” “inauthentic,” and “misattributed,” see Michael Herz,
Fraudulent Malattributed Comments in Agency Rulemaking, 42 CARDOZO L. REV. 1, 10-12 (2020). A false name can
be seen as just one instance of false statements in comments generally. We address that larger problem briefly below,
but our focus is on the false identity issue alone. The question of how agencies should respond to false information
included in the body of a comment is also important, but it is only tangentially related to challenges created by new
technologies, which are the focus of this report. The malattribution problem, by contrast, has been greatly accentuated
by the development of new technologies.

11

3.

Computer-Generated Comments

The notice-and-comment process invites interested persons to submit their views on
proposed rulemakings. A tacit assumption of this invitation is that the text contained within a
public comment will be written by a human commenter. This assumption can be violated when a
software program is used to generate the text. We define “computer-generated comments” as those
that are generated by a software algorithm, thus replacing both human content generation and
human interaction with the agency. Although a human must create the software to accomplish this
task, once automated, that individual need not further engage in the commenting process and,
instead, the software can submit comments via regulations.gov, possibly even repeatedly.46
Computer-generated comments are enabled by advances in automated text creation. To date,
computer-generated comments have been fairly crude cut-and-pastes that are easy to detect.
However, researchers in the field of natural language processing (NLP) continue to make striking
progress. Although the first computer program attempting to mimic human conversation was
introduced in the mid-1960s,47 in recent years more sophisticated software is making it possible to
produce comments that seem to be unique and written by humans, when they are actually produced
by machines. Contemporary algorithms have achieved results that are difficult to distinguish from
human writers.48 If similar (or more advanced) systems were used to generate public comments,
they could overwhelm an agency with an arbitrarily large number of human-quality comments.
The creators of a recently developed artificial intelligence technique for generating human-like
text known as GPT-3 recognized this risk, including “abuse of legal and governmental processes”
among the potential misuses of text generation tools.49 A recent experiment involving computergenerated comments submitted in response to a proposed Medicaid waiver, which we describe
below, gives a glimpse of the potential of this technology to produce realistically human comments
in the rulemaking process.
Such advances need not necessarily be problematic. One can envision computer-generated
comments that add value to the rulemaking process. For example, a program could review an
agency’s published proposed rule for spelling errors, broken links, and other simple errors and
then file a comment in the relevant docket to provide a report of these errors. In this example the
program is not providing input on policy views; it is merely providing technical assistance.
In addition, there may be more sophisticated ways to bring the benefits of big data and artificial
intelligence to the comment process. In the same way that credit card companies can offer fraud alerts
to individual cardholders when purchasing activity is unusual, perhaps there are ways for people to

46

Many malattributed comments are also computer-generated comments, but neither is a full subset of the other. Some
concerns—for example, that they make it look as if more people are submitting comments or taking a particular
position than is in fact the case—may apply to both. Our discussion of malattributed comments, however, focuses
specifically on the malattribution, not the fact that it is often done by a computer. And our discussion of computergenerated comments focuses on the source of the comment, not the fact that it may well attach a false identity to the
submission.

47

Joseph Weizenbaum, ELIA—A Computer Program for the Study of Natural Language Communication between
Man and Machine, 9 COMMUNC’NS OF THE ACM 37 (1966).

48

See e.g., TOM B. BROWN, ET AL., LANGUAGE MODELS ARE FEW SHOT LEARNERS 5 (July 2020),
https://arxiv.org/pdf/2005.14165.pdf (discussing the GPT-3 autoregressive language model).

49

Id. at 35.

12

leverage technology to assist them in producing draft comments based on the interests they have
expressed through their purchasing habits or other administrative data. There are plenty of operational,
privacy, and other issues to consider, but we raise it here as a possibility that may caution against overly
restrictive approaches to computer-generated comments. To the extent that these types of tools could
help overcome the collective action problems that inhibit public participation in the rulemaking
process, they could offer a useful path forward.

B. Challenges Posed by Technology-Enabled Comments
All three types of comments we discuss in this report affect the rulemaking process. We
explore in more detail below the specific challenges posed by mass, malattributed, and computergenerated comments. But, first, it is important to note that these technology-enabled comments are
not an entirely new phenomenon.
With respect to mass comments, there were high-salience rulemakings that generated
substantial numbers of public comments prior to the advent of online commenting. In the 1995-96
rulemaking in which it first asserted regulatory authority over tobacco cigarettes, the Food and
Drug Administration received over 700,000 paper comments, many of which were identical, socalled “postcard comments.”50
Yet orchestrating a campaign to mail in masses of comments was much more burdensome
and expensive than it is now. Once it became possible to comment with the click of a button, mass
participation, both spontaneous or orchestrated, became much more straightforward as did the
potential for duplicative comments.51
Similarly, it has always been the case that a commenter could sign a phony name to a
comment or include other falsehoods in the comment. But the digital availability of personal
information, automation tools, and online submission makes it much easier to submit such
comments at scale. And the difficulty of identifying misleading or mislabeled comments is
heightened in a deluge of comments.
Even computer-generated comments could, in theory, have been submitted on paper.
Automated text-generation software has existed since the 1960s, and it would be a trivial task to
print out computer-generated text and place it in the mail. Again, however, the shift to electronic
submissions—alongside tremendous advances in artificial intelligence and NLP in recent years—
facilitates and reduces the costs of submitting comments written by computers.
All three of these types of comments can generate serious challenges to agencies, raising a
pressing set of questions concerning how best to respond while preserving a functional rulemaking
50

Food and Drug Admin., Regulations Restricting the Sale and Distribution of Cigarettes and Smokeless Tobacco to
Protect Children and Adolescents, 61 Fed. Reg. 44396, 44418 (1996). As the FDA described it:
Altogether, the agency received more than 700,000 pieces of mail, representing the views of nearly
1 million individuals. Most of the submissions were form letters or post cards. The agency identified
more than 500 different types of form letters. Others were petitions with sometimes hundreds of
signatures. More than 95,000 submissions expressed individual comments on the 1995 proposed
rule, including more than 35,000 from children who were overwhelmingly supportive.
Id.

51

Beth Simone Noveck, The Electronic Revolution in Rulemaking, 53 EMORY L.J. 433 (2004) (expressing concern
that new technology might open the floodgates to “notice-and-spam” rulemaking).

13

process. We now turn to exploring what is at stake with mass, malattributed, and computergenerated comments.
1. Information
Agency rulemakings often touch on important areas of social and economic life, and can
have complex and difficult-to-anticipate effects. Agencies bring considerable internal expertise to
the task of crafting rules, and officials can also often draw on published research. However, there
is often useful information that an agency might not have readily at hand during its deliberations,
and an important part of the value of notice-and-comment rulemaking is the public’s opportunity
to bring such information to the agency’s attention.
Often, the most useful information for an agency will be technical or operational. This type
of information includes scientific or engineering studies, relevant data, analysis about how well
the proposed regulatory change will address the problem being solved, what kind of changes
compliance will require, or legal or policy analysis. Technical or operational information
empowers agency decision makers to anticipate the consequences of the choices they face in the
design of regulations. Information along these lines facilitates higher quality rulemakings, where
quality is understood in terms of technical or operational proficiency.
A second type of information concerns conclusions drawn by stakeholders or members of
the public concerning the desirability of a rulemaking. Under the APA, agencies are bound to
consider relevant substantive arguments offered by commenters in support of their conclusions.
But the status of the ultimate evaluation offered by a commenter is less clear. There is a debate
among administrative law scholars concerning the extent to which agencies should consider
commenter preferences. Cynthia Farina, for example, has argued that the deliberative and technical
nature of the rulemaking process makes consideration of pure expressions of preference
inappropriate, especially if they are not informed or representative.52 Nina Mendelson, by contrast,
has argued that expressions of preference can and should be considered by agencies, at least in
some contexts, because agencies are often called on to “decide values and policy questions left
unresolved by their authorizing statutes.” 53 For Mendelson, concerns about representativeness, for

52

Cynthia R. Farina et al., Rulemaking vs. Democracy: Judging and Nudging Public Participation That Counts, 2
MICH. J. ENVTL. & ADMIN. L. 123, 137–39 (2012).

53

Nina A. Mendelson, Rulemaking, Democracy, and Torrents of E-Mail, 79 GEO. WASH. L. REV. 1343, 1350–51
(2011) [hereinafter Mendelson, Rulemaking, Democracy]; see also Nina A. Mendelson, Should Mass Comments
Count?, 2 MICH. J. ENVTL. & ADMIN. L. 173, 1818 (2012) (“[A]gency officials might pay attention to large volumes
of comments, for example, to help gauge public resistance or anticipate significant opposing views.”). This debate
tracks, in certain respects, the scholarly conversation on whether courts should accept “political reasons” as
justifications for agency decisions. Jodi L. Short, The Political Turn in American Administrative Law: Power,
Rationality, and Reasons, 61 DUKE L.J. 1811 (2012); Nina A. Mendelson, Disclosing “Political” Oversight of Agency
Decision Making, 108 MICH. L. REV. 1127 (2010); Kathryn A. Watts, Proposing a Place for Politics in Arbitrary and
Capricious Review, 119 YALE L.J. 2 (2009). More broadly, the debate about the appropriate role for consideration of
expressions of preference in public comments tracks alternative views about the legitimate foundations of
administrative decision making. See generally Vanessa Duguay, Views or Votes: The Challenge of Mass Comments
in Rulemaking, 26 GEO. MASON L. REV. 625 (2018); Steven P. Croley, Public Interest Regulation, 28 FLA. ST. U. L.
REV. 7, 7 (2000); Mark Seidenfeld, A Civic Republican Justification for the Bureaucratic State, 105 HARV. L. REV.
1511 (1992).

14

example, must be balanced against the drawbacks of ignoring the sentiments of those who have
taken the time to comment.
Without settling this debate, it is worth noting that there are often substantive limits on the
kinds of information that agencies may consider in the course of rulemaking.54 Under the State
Farm formulation, “[n]ormally, an agency rule would be arbitrary and capricious if the agency has
relied on factors which Congress has not intended it to consider.”55 There are many examples
where courts have found limits on agencies’ ability to consider certain factors. For example, in
Whitman v. American Trucking, the Court found that the EPA may not consider costs when setting
the National Ambient Quality Standards under the Clean Air Act.56 Accordingly, public
preferences concerning whether the benefits of more stringent air quality standards outweigh the
costs are not relevant under American Trucking.
Mass, malattributed, and computer-generated comments can make it difficult to extract
both technical/operational and preference information from the notice-and-comment process, but
the potential challenges for preference information are greater. With respect to
technical/operational information, the identity of the commenter (even whether the commenter is
a human being) and how frequently that information appears in the record will often not be
relevant. The primary challenge raised by mass, malattributed, and computer-generated comments
is that useful technical/operational information may be difficult to find within a large flood of
comments. Preference information, by contrast, would only be relevant inasmuch as it relates to
the views of a genuine person, making it necessary to separate out bot and malattributed comments
from those that are genuinely submitted by a person. Further, for rules that result in a mass
comment response, agencies face a range of difficult questions (discussed in more detail below)
concerning the representativeness of the pool of commenters and the role of intermediary groups
that conduct mass comment campaigns.
Finally, information is valuable only if it is accurate. When a comment contains false or
erroneous statements, intentional or otherwise, it is at least a delay and distraction. If the falsehood
is important and undiscovered, it could negatively affect the substance of the final rule. From the
outset, one concern about e-rulemaking has been that it would lead to agencies being deluged with
misinformation.57 The agency personnel we interviewed reported that misinformation in
comments has not been a major problem to date. Presumably this is for at least two reasons. First,
agencies are repositories of significant expertise. That means they will often recognize substantive
errors in comments or at least know enough to realize that further investigation is required. Second,
broad participation is a prophylactic against misinformation; the false submission might be
countered by a true one, often multiple true ones. This is not to dismiss all concerns about false

54

See Stuart M. Benjamin, Evaluating E-Rulemaking: Public Participation and Political Institutions, 55 DUKE L. J.
893, 906–907 (2006) (“Many statutes leave no room for an agency to consider public sentiment.”).

55

Motor Vehicle Mfrs. Assn. of United States v. State Farm Mut. Automobile Ins. Co., 463 U.S. 29, 43 (1983)
(emphasis added).

56

Whitman v. Am. Trucking Ass’ns, 531 U.S. 457 (2001).

57

See, e.g., Daniel C. Esty, Environmental Protection in the Information Age, 79 N.Y.U. L. REV. 115, 172 (2004)
(“The promise of cyberdemocracy with a fully informed and engaged populace could give way to ‘spam,’
misinformation, and dialogue among the uninformed that diminishes thoughtful deliberation.”).

15

submissions, which may be seen as a growing concern, but it is a reminder that a falsehood in a
comment is only a small first step toward a substantive error in a final rule.
2. Legitimacy
The concept of legitimacy is complex and its full exegesis is beyond the scope of this
report. We focus on the potential effects of mass, malattributed, and computer-generated
comments on positive (or sociological) legitimacy, which concerns empirical questions related to
public acceptance of the exercise of government power,58 as distinct from normative or moral
legitimacy.59
There is a considerable body of behavioral and social scientific research on the causes of
positive legitimacy.60 One important thread of that literature concerns perceptions of “procedural
fairness” and the components of decision-making processes that tend to enhance or undermine
those perceptions. In a recent OECD report by E. Allan Lind and Christiane Arndt summarizing
some of this literature, the authors identify “[t]hree general elements of process . . . [that] stand out
in terms of their impact on whether a citizen will feel fairly treated in his or her interactions with
government [. . .]: voice, polite and respectful treatment, and explanations.”61 The authors define
voice as “a chance [for affected people] to present their views” along with “some indication that
the input was actually given consideration.”62 According to this review of the literature, voice
“remains the most extensively researched and arguably the most powerful antecedent of perceived
procedural fairness.”63
The relationship between voice and the notice-and-comment process is obvious—the APA
requirement that agencies solicit and consider the views of interested persons maps exactly onto
the definition of voice offered by Lind and Arndt. Based on current research in the field, there is
reason to believe that the notice-and-comment process enhances the positive legitimacy of agency
rulemaking, particularly when compared to an imagined counterfactual in which there is no

58

Questions concerning the sociological legitimacy of the state reach back to the foundations of contemporary social
sciences. See Max Weber, Die drei reinen Typen der legitimen Herrschaf, 187 PREUSSISCHE JAHRBÜCHER 1 (1922)
(appearing later as Max Weber, The Three Types of Legitimate Rule, 4 BERKELEY PUBLICATIONS IN SOC’Y &
INSTITUTIONS 1 (1958) (Han Gerth trans.)). These questions arise for all governmental bodies, but are particularly
pressing for those with more tenuous relationships with the electoral process. Cf. Jeffery J. Mondak, Policy Legitimacy
and the Supreme Court: The Sources and Contexts of Legitimation, 47 POL. RES. Q. 675, 690 (1994); James L. Gibson,
Understandings of Justice: Institutional Legitimacy, Procedural Justice, and Political Tolerance, 23 LAW & SOC.
REV. 469, 471 (1989).

59

Cf. Richard H. Fallon, Legitimacy and the Constitution, 118 HARV. L. REV. 1787, 1790 (2005); Paul Weithman,
Legitimacy and the Project of Rawls’s Political Liberalism, in RAWLS’S POLITICAL LIBERALISM 73 (T. Brooks and
M. Nussbaum eds. 2012).

60

See, e.g., James L. Gibson & Michael J. Nelson, The Legitimacy of the US Supreme Court: Conventional Wisdoms
and Recent Challenges Thereto, 10 ANN. REV. L. SOC. SCI. 201 (2014).

61

E. Allan Lind & Christiane Arndt, Perceived Fairness and Regulatory Policy: A Behavioural Science Perspective
on Government-Citizen Interactions 20 (OECD Regulatory Policy Working Paper No. 6, 2016).

62

Id.

63

Id.

16

consistent opportunity for the public to comment or such comments are not considered by agency
decision makers.
Malattributed and computer-generated comments may undermine the confidence of
members of the public in their ability to have their voices heard. Observers reasonably worry that
computer-generated comments submitted at scale could drown out comments from real persons,
create confusion on relevant issues, or prompt an agency to ignore even legitimate comments,64
and malattributed comments could be perceived as hijacking or expropriating a person’s voice.
These issues will be discussed in more detail below.
Mass comments present other questions concerning their interaction with voice and
perceptions of procedural fairness. Lower costs of submitting comments and broader public
participation creates more widespread opportunities for voice. But if comments contain
information that agencies may not or do not consider—including expressions of preference—it is
not clear that the process will ultimately enhance perceptions of procedural fairness. As noted by
Lind and Arndt “research on voice makes it clear that it is not enough just to allow for more raw
input or comment: There must also be some indication that the input was actually given
consideration.”65 A mismatch between commenter expectations and agency treatment of
comments raises serious concerns, which are discussed in more detail below.
It is worth emphasizing that for purposes of positive legitimacy, perceptions of procedural
fairness matter, irrespective of how well those perceptions map onto reality. For example, even if
agencies are able to easily sort through bot or malattributed comments, these phenomena could
lead to misimpressions about the integrity of the system that undermine public confidence in the
process.66 Experience with the FCC’s net neutrality rulemaking demonstrates that the public
comment process can become publicly salient without warning, with the associated risks of
sensational commentary and the potential for people to draw inferences about the entire process
based on an exceptional example.

64

See, e.g., Fake It Till They Make It, supra note 2, at 11 (statement of Paulina Gonzalez-Brito, Executive Director,
California Reinvestment Coalition) (stressing the need “to ensure that community voices are not drowned out by
fabricated comments fraudulently submitted in favor of industry”); Bob Barr, Massive Fraud in Net Neutrality Process
Is a Crime Deserving of Justice Department Attention, MARIETTA DAILY J. (Dec. 25, 2017),
https://www.mdjonline.com/opinion/bob-barr-massive-fraud-in-net-neutrality-process-is-a-crime-deserving-ofjustice-department/article_87a01d86-e9c5-11e7-af34-3bc55501c7a0.html (“[B]efore too long, the voices of real
people, expressing genuine opinions on regulations, will be drowned out and ignored all together by those in power.”);
Eric
T.
Schneiderman,
An
Open
Letter
to
the
FCC,
MEDIUM
(Nov.
21,
2017),
https://medium.com/@NewYorkStateAG/an-open-letter-to-the-fcc-b867a763850a (objecting that “the perpetrator or
perpetrators attacked what is supposed to be an open public process by attempting to drown out and negate the views
of the real people, businesses, and others who honestly commented on this important issue”).

65

Lind & Arndt, supra note 61, at 20.

66

“The last thing we need is a common view that essentially the entire rulemaking process is being gamed by a variety
of machines and shadowy players.” Nicole Ogrysko, GSA Launches Public Campaign to Battle Bots, Fake Comments
from Online Rulemaking Forums, FED. NEWS NETWORK (Jan. 31, 2020) (quoting Michael Fitzpatrick, head of global
regulatory affairs for Google), https://federalnewsnetwork.com/management/2020/01/gsa-launches-public-campaignto-battle-bots-fake-comments-from-online-rulemaking-forums/.

17

In addition to perceptions of procedural fairness, scholars have identified an alternative
source of positive legitimacy: it flows from the outcomes of government decisions themselves.67
The basic idea is that high quality, effective government decision making leads to public
acceptance. If mass, malattributed, or computer-generated comments reduce regulatory quality by,
for example, making it more difficult for agencies to extract useful information from the noticeand-comment process, then, over time, they could erode confidence in agency decision making.
3. Processing Costs
It takes time and resources to review, analyze, and respond to comments and use the
insights to recraft the rule. When there is a small number of comments, those costs are relatively
low. As the number of comments grows, processing costs can be expected to increase. For
example, agencies sometimes hire outside contractors to help process comments, which helps
alleviate the processing burden, but adds to the overall expense. When they do this work in-house,
the review process, albeit important, consumes significant staff power. Similarly, if agencies must
spend resources to identify malattributed or computer-generated comments, that only further
increases processing costs. Time spent sorting out mass, malattributed, and computer-generated
comments also delays the process and takes time away from other productive policymaking
activities.
When the informational and legitimacy-conferring benefits of comments are high, then the
time invested may be well worth their processing costs. Nevertheless, the direct financial and delay
costs of spurious or low-quality comments are nontrivial and worth keeping in mind.
III.

Mass Comments

Large volumes of public comments present both opportunities and challenges for agencies.
On the one hand, current participation in the notice-and-comment process demonstrates substantial
interest in agency rulemaking, which creates occasions for meaningful engagement between
agencies and the public. Public comments can also contain helpful information that agencies can
use to improve their rulemakings.
On the other hand, large volumes of comments are burdensome to process and digest,
increase the risk of missing important arguments or information, and may make it more difficult
to extract overall patterns in the content of comments.
As noted above, for purposes of this report, we distinguish between a mass comment
response and a mass comment campaign. The latter is a special case of a mass comment response
in which an individual or organization successfully urges a large number of individuals to file
comments that express a similar set of views or positions. Often, comments made in response to a
mass solicitation contain identical or nearly identical language. The soliciting organization may
post a sample comment and encourage the submitter to file the comment verbatim but include a
sentence or two at the end explaining how the rule personally affects the submitter. It is also
possible that soliciting organizations may encourage submissions of unique—if substantively

67

See, e.g., FRITZ SCHARPF, GOVERNING IN EUROPE: EFFECTIVE AND DEMOCRATIC? 11 (1999) (distinguishing “inputoriented” and “output-oriented” forms of democratic legitimacy).

18

similar—comments. In such cases, it may be difficult to disentangle a mass comment campaign
from a more spontaneous mass response.
A. Legal Issues Raised by Mass Comments
Courts have had many opportunities to visit the question of how agencies must consider
information generated during the notice-and-comment process. In one early and important
formulation, the D.C. Circuit’s Home Box Office v. FCC opinion provided the following standard:
In determining what points are significant, the “arbitrary and capricious” standard
of review must be kept in mind. Thus only comments which, if true, raise points
relevant to the agency’s decision and which, if adopted, would require a change in
an agency’s proposed rule cast doubt on the reasonableness of a position taken by
the agency. Moreover, comments which themselves are purely speculative and do
not disclose the factual or policy basis on which they rest require no response. There
must be some basis for thinking a position taken in opposition to the agency is
true.68
A few years previously, in the canonical Portland Cement decision, Judge Leventhal
expressed a similar sentiment, writing that “comments must be significant enough to step over a
threshold requirement of materiality before any lack of agency response or consideration becomes
of concern.”69 More recently, the D.C. Circuit stated the point this way: “An agency is not obliged
to respond to every comment, only those that can be thought to challenge a fundamental
premise.”70
Courts have repeatedly noted that agencies are required to consider the substance of
comments: “An agency need not respond to every comment, but it must ‘respond in a reasoned
manner to the comments received, to explain how the agency resolved any significant problems
raised by the comments, and to show how that resolution led the agency to the ultimate rule.’”71
There is no obligation to respond to comments per se.72 Rather, “[t]he failure to respond to
68

Home Box Office, Inc. v. FCC, 567 F.2d 9, 35 n.58 (D.C. Cir. 1977); see Pub. Citizen, Inc. v. FAA, 988 F.2d 186,
197 (D.C. Cir. 1993) (citing HBO for standard); Nat’l Shooting Sports Found., Inc. v. Jones, 716 F.3d 200, 215 (D.C.
Cir. 2013) (same).

69

Portland Cement Ass’n v. Ruckelshaus, 486 F.2d 375, 394 (D.C. Cir. 1973). This language continues to be cited by
courts to express the relevant standard. See, e.g., Am. Great Lakes Ports Ass’n v. Zukunft, 296 F. Supp. 3d 27, 53
(D.D.C. 2017).

70

MCI WorldCom, Inc. v. FCC, 209 F.3d 760, 765 (D.C. Cir. 2000); see also Am. Mining Cong. v. U.S. EPA, 907
F.2d 1179, 1187-88 (D.C. Cir. 1990) (“[I]n assessing the reasoned quality of the agency’s decisions, we are mindful
that the notice-and-comment provision of the APA . . . ‘has never been interpreted to require [an] agency to respond
to every comment, or to analyse [sic] every issue or alternative raised by comments, no matter how insubstantial.”)
(quoting Thompson v. Clark, 741 F.2d 401, 408 (D.C. Cir. 1984)).

71

Action on Smoking & Health v. CAB, 699 F.2d 1209, 1216 (D.C. Cir. 1983) (quoting Rodway v. U.S. Dep’t of
Agric., 514 F.2d 809, 817 (D.C. Cir.1975)); see South Carolina ex rel. Tindal v. Block, 717 F.2d 874, 885 (4th Cir.
1983) (“The purpose of allowing comments is to permit an exchange of views, information, and criticism between
interested persons and the agency.”).

72

Sherley v. Sebelius, 689 F.3d 776, 784 (D.C. Cir. 2012) (“[A]n agency’s failure to address a particular comment or
category of comments is not an APA violation per se.”); United States v. Nova Scotia Food Prods. Corp., 568 F.2d

19

comments is significant only insofar as it demonstrates that the agency’s decision was not ‘based
on a consideration of the relevant factors.’”73
A corollary to this focus on the substance of comments has been a tendency to deemphasize
the importance of the number of comments received. The D.C. Circuit has stated directly that
agencies are under “no obligation to take the approach advocated by the largest number of
commenters.”74 There is a broad consensus that the public comment process is “not a vote.”75
Courts have sometimes explicitly considered the number of comments received by an agency,76
but these are outside the rulemaking context and not to evaluate the adequacy of the agency’s
explanation or the support in the record for its conclusion..
Although courts have emphasized the importance of substance over volume in evaluating
agency responses to public comments, there is no general bar against agencies relying on
information contained in form comments. For example, the plaintiffs in Resident Councils v.
Leavitt argued that a regulation was invalid because the vast majority of the supportive comments
on the proposed rule were form letters and the agency’s reliance on them was therefore
unwarranted. The court disagreed, stating that “there is no reason the Secretary was not entitled to
rely on such letters in promulgating the regulations.”77 The court followed up by stating that just

240, 252 (2d Cir. 1977) (“‘We do not expect the agency to discuss every item of fact or opinion included in the
submissions made to it in informal rulemaking.’”) (quoting Auto. Parts & Accessories Ass’n v. Boyd, 407 F.2d 330,
338 (D.C. Cir. 1968)).
73

Thompson, 741 F.2d at 409 (quoting Citizens to Pres. Overton Park v. Volpe, 401 U.S. 402, 416 (1971)); see also
Covad Commc’ns Co. v. FCC, 450 F.3d 528, 550 (D.C. Cir. 2006).

74

U.S. Cellular Corp. v. FCC, 254 F.3d 78, 87 (D.C. Cir. 2001). See also Nat. Res. Def. Council, Inc. v. EPA, 822
F.2d 104, 122 n.17 (D.C. Cir. 1987) (noting that rulemaking is not a process where “the majority of commenters
prevail by sheer weight of numbers”).

75

Bridget C.E. Dooling, Legal Issues in E-Rulemaking, 63 ADMIN. L. REV. 893, 901 n.33 (2011); Michael Herz,
“Data, Views, or Arguments”: A Rumination, 22 WM. & MARY BILL RTS. J. 351, 369-74 (2013); Tips for Submitting
Effective Comments, REGULATIONS.GOV (“The comment process is not a vote.”) (on file with authors)

76

For example, in North Carolina Growers’ Ass’n v. United Farm Workers, the court found that a truncated comment
period with substantial content restrictions was inadequate, in part relying on the small number of comments received
compared to a prior related rulemaking. N.C. Growers’ Ass’n v. United Farm Workers, 702 F.3d 755, 770-71 (4th
Cir. 2012); see also California ex rel. Becerra v. U.S. Dep’t of the Interior, 381 F. Supp. 3d 1153, 1176-1178 (N.D.
Cal. 2019). Some courts have also looked to the number of comments received during the National Environmental
Policy Act review process to determine whether a project is “controversial” and therefore requires a full environmental
impact statement, although there is disagreement over the relevance of the scale of public reaction to that inquiry.
Sierra Club v. Bosworth, 510 F.3d 1016, 1032 (9th Cir. 2007) (“Given the large number of comments, close to 39,000,
and the strong criticism from several affected Western state agencies, we cannot summarily conclude that the effects
of the Fuels CE are not controversial.”); Greenpeace Action v. Franklin, 14 F.3d 1324, 1333-34 (9th Cir. 1992) (noting
that an “outpouring of public protest” along with a “substantial dispute . . . as to size, nature, or effect” of a proposed
action can demonstrate that the action is controversial and therefore requires an EIS (citations omitted)); Emily M.
Slaten, Note, “We Don’t Fish in Their Oil Wells, and They Shouldn’t Drill in Our Rivers”: Considering Public
Opposition Under NEPA and the Highly Controversial Regulatory Factor, 43 IND. L. REV. 1319 (2010).

77

Resident Councils of Wash. v. Leavitt, 500 F.3d 1025, 1030 n.5 (9th Cir. 2007).

20

because numerous people “share the same opinion and pooled their efforts does not undermine
their intended show of support.”78
B. Policy Issues Raised by Mass Comments
There are a number of policy issues raised by mass comment responses (whether or not
part of a mass comment campaign). Many researchers have found that a large percentage of the
comments received in mass comment responses are not highly substantive, but rather contain
general statements of support or opposition.79 As mentioned above, there is some debate
concerning whether and to what the extent to which agencies should consider comments that
contain only statements of preference.
The Permanent Subcommittee on Investigations of the U.S. Senate’s Homeland Security
and Governmental Affairs Committee has recommended that Congress consider amending the
APA to provide guidance to agencies on the extent to which they should consider the volume of
comments in favor of or in opposition to a proposed rule.80 Guidance from Congress could be
helpful to agencies in deciding when, if ever, they should take the number of comments and the
sentiment expressed in the comments into account when finalizing a rule.
Inasmuch as public opinion is relevant for a rulemaking, comments generally do not
provide a reliable metric of the views of the broader public. Commenters are an entirely selfselected group, and there is no reason to believe that they are in any way representative of the
larger public. Relatedly, the group of commenters may represent a relatively privileged group, with
less advantaged members of the public less likely to engage in this form of political participation.
There are also questions related to the actual influence of mass comment responses on
agency decision making. There is a considerable social science literature that examines the public
comment process and how it affects regulatory outcomes.81 An important early paper by Marissa
78

Id. Morales v. Lyng, 702 F. Supp. 161 (N.D. Ill. 1988) also discusses the role of form comments. In this case, the
Department of Agriculture was found to have acted in an arbitrary and capricious manner by choosing to generally
ignore certain comments. The Secretary of Agriculture argued that the ignored comments were “endless clones of
conclusory statements.” Id. at 163. However, the court found that by choosing to ignore these comments that offered
a differing view than that chosen by the agency, the agency had impermissibly “failed to consider important aspects
of the administrative record and hence the issue itself.” Id.

79

Thomas A. Bryer, Public Participation in Regulatory Decision-Making: Cases From Regulations.gov, 37 PUB.
PERFORMANCE & MGMT. REV. 263, 263 (2014) (analysis of EPA and HHS rulemakings finding that many comments
were “emotional, illogical and lacking in credibility”); Kimberly D. Krawiec, Don’t “Screw Joe the Plummer”: The
Sausage-Making of Financial Reform, 55 ARIZ. L. REV. 53, 58 (2013) (contrasting industry comments on the
Securities and Exchange Commission’s Volcker Rule, which were “meticulously drafted, argued, and researched”
with “citizen letters [which were] short and provide little evidence that citizen commenters even understand, or care,
what proprietary or fund investment is, much less the ways in which agency interpretation of the Volcker Rule’s
complex and ambiguous provisions might govern such activities”); Stuart W. Shulman, The Case Against Mass Emails: Perverse Incentives and Low Quality Public Participation in U.S. Federal Rulemaking, 1 POL’Y & INTERNET
23 (2009) (arguing the many comments lack substantive merit).

80

Subcommittee Report, supra note 10, at 3.

81

For an overview of this literature, see Susan Webb Yackee, The Politics of Rulemaking in the United States, 22
ANN. REV. POL. SCI. 37 (2019). An important general point is that other mechanisms for interested parties to affect
agency decision making, such as ex-parte communications during the pre-proposal stage, may be more influential than

21

Golden found that business interests tended to dominate the rulemaking process, but that the
overall influence of comments was low.82 Subsequent work has found that, at least under certain
conditions, agencies sometimes do make changes in response to comments.83 Among the factors
that have been found to affect commenter influence is the degree of sophistication in the comments
and the source of the comment.84 Studies of mass commenting in particular have found that
agencies tend to be less responsive to mass comment campaigns,85 and to refer to the number of
comments received in favor of or in opposition to a rule in opportunistic ways.86 Some have argued
that the reality that agencies are unlikely to alter rules in response to less substantive comments
provides a reason to discourage, or at least “not actively facilitate public participation” of this
sort.87
Direct influence may not be the only motivation behind comments, and advocacy groups
may solicit mass comments for many different reasons. Research on mass comment campaigns
suggests that that different groups carry out such campaigns to promote a range of goals, including
the public comment process. See Jeffrey J. Cook, Crossing the Influence Gap: Clarifying the Benefits of Earlier
Interest Group Involvement in Shaping Regulatory Policy, 42 PUB. ADMIN. Q. 466 (2018); Susan Webb Yackee, The
Politics of Ex Parte Lobbying: Pre-Proposal Agenda Building and Blocking during Agency Rulemaking, 22 J. PUB.
ADMIN. RES. & THEORY 373 (2012).
82

Marissa Martino Golden, Interest Groups in the Rule-Making Process: Who Participates? Whose Voices Get
Heard?, 8 J. PUB. ADMIN. RES. & THEORY 245 (1998). That study involved analysis of comments received by three
agencies (EPA, HUD, and NHTSA) to a set of eleven rulemakings. Generally, Golden finds that “business
commenters” dominated the public comment process, as “[b]etween 66.7 percent and 100 percent of the comments
received were submitted by corporations, public utilities, or trade associations.” Id. at 252–53. However, Golden did
not find a large substantive impact from the business community’s participation; she attributed this lack of influence
in part due to the fact that “business did not present a united front[;] . . . [t]here were frequently divisions within the
business community.” Id. at 262.

83

Amy McKay & Susan Webb Yackee, Interest Group Competition on Federal Agency Rules, 35 AM. POL. RES. 336
(2007); Susan Webb Yackee, Sweet-Talking the Fourth Branch: The Influence of Interest Group Comments on
Federal Agency Rulemaking, 16 J. PUB. ADMIN. RES. & THEORY 103 (2006).

84

Mariano-Florentino Cuéllar, Rethinking Regulatory Democracy, 57 ADMIN. L. REV. 411 (2005) (identifying three
instances in which agencies modified proposals in light of submissions from non-business commenters); Jason Webb
Yackee & Susan Webb Yackee, A Bias Towards Business? Assessing Interest Group Influence on the U.S.
Bureaucracy, 68 J. POL. 128, 133–35 (2006) (finding that agencies consistently alter proposals to reflect comments
from business interests but not others). One article observes: “The relatively high value placed on hard data in
comments is best summed up by the interviewee who stated, ‘We look at every comment; we consider every comment.
But unless there is data supporting the position, it’s just not that useful in the rulemaking process.’” Keith Naughton
et al., Understanding Commenter Influence During Agency Rule Development, 28 J. POL’Y ANALYSIS & MGMT. 258,
270 (2009).

85

Balla et al., supra note 1 (finding that agencies give mass comments limited attention in the preambles to final rules
and that “regulations are generally not consistent with changes requested in comments, a lack of association that holds
especially for mass comment campaigns.”). Some observers have identified cases where mass comments (at least
arguably) influenced regulatory outcomes. See Lauren Moxley, E-Rulemaking and Democracy, 68 ADMIN. L. REV.
661, 692–95 (2016) (attributing change in FCC’s 2015 final net neutrality rule to large number of and consensus
among commenters).

86

See Herz, supra note 75, at 372–73 (“When [the agencies’] conclusion has strong support in the [mass] comments
they tend to note that fact, and when it does not they tend to glide over it.”); Parrillo, Should the Public Get to
Participate, supra note 20, at 71.

87

Farina et al., supra note 52, at 150 (“A democratic government should not actively facilitate public participation
that it does not value.”).

22

calling public attention to a rulemaking.88 Others have pointed to internal organizational goals,
such as increasing membership and financial contributions and moving members up the “ladder of
engagement” towards greater involvement, as a motivation for efforts to mobilize actions like
petition-signing and sending public comments.89
Scholars have identified several potential problems with mass comment campaigns. Some
have argued that they may be used to distort regulators’ perception of public opinion, and may
lead to agency cynicism about the public comment process.90 Mass comment campaigns often
involve many duplicate, or near duplicate comments.91 Such duplicate comments impose various
real costs on agencies, without adding new substantive content to the rulemaking record.92 It is
worth noting that, the expert consensus that the public comment process is “not a vote,” appears
to conflict with “widely held views among participating individuals, advocacy groups, and
journalists that the public expression of preferences should and does carry some weight, entirely
apart from whatever substantive justification for those preferences is offered.”93 Cynthia Farina

88

Steven J. Balla et al., Where’s the Spam? Interest Groups and Mass Comment Campaigns in Agency Rulemaking,
11 POL’Y & INTERNET 460 (2019). Balla et al. find that that campaigns organized by regulated entities are more
substantive than campaigns organized by regulatory beneficiaries. Regulatory beneficiaries sponsored 73% of mass
comment campaigns analyzed (87% of campaigns with 1000+ comments), whereas regulated entities sponsored 27%
of mass comment campaigns (13% of those over 1000 comments). Campaigns sponsored by regulatory beneficiaries
were larger, averaging 15,783 comments, whereas those by regulated entities received an average of 4,345 comments.
Regulatory beneficiaries stated in interviews that during the Obama administration mass comment campaigns were
used to help the EPA justify proposed actions, whereas during the Trump-era the campaigns were used to cause the
administration to “feel pain” in the media and public opinion. Regulated entities, on the other hand, stated in interviews
that they use mass comment campaigns to try and counterbalance the mobilization by regulatory beneficiaries.

89

Farina, supra note 52, at 141; David Karpf, Online Political Mobilization from the Advocacy Group’s Perspective:
Looking Beyond Clicktivism, 2 POL’Y & INTERNET 1, 35 (2010) (“[Organizations use] email to mobilize member
interest around their top campaign priority, as a first step in a ladder-of-engagement.”); Shulman, supra note 79, at
27-30.

90

Sara R. Jordan & K. Lee Watson, Reexamining Rulemaking in an Era of Internet-Enabled Participation, 42 PUB.
PERFORMANCE & MGMT. REV. 836, 856 (2019) (“At the level of regulatory politics, manufactured salience is the
generation by politically or economically motivated actors of a large number of comments . . . in order to alter the
perceptions of regulators’ ascribed level of salience of a position on a rule.”); David Schlosberg et al., Deliberation in
E-Rulemaking? The Problem of Mass Participation, in ONLINE DELIBERATION: DESIGN, RESEARCH, AND PRACTICE
133, 143 (Todd Davies & Seeta Peña Gangadharan eds., 2009) (“Interviews with agency rule writers show that
agencies do not value and often openly resent form letters. The EPA, in fact, simply prints and stores an inaccessible
hard copy of all but one example of each identical or similar mass email.”); Stuart W. Shulman, The Internet Still
Might (But Probably Won’t) Change Everything, 1 I/S 111, 111–12 (2005) (raising concern that agency personnel
would become cynical about mass comment campaign).

91

Stuart Shulman, Whither Deliberation? Mass E-Mail Campaigns and U.S. Regulatory Rulemaking, 3 J. E-GOV’T
41, 58 (2006) (finding that very small percentage of mass campaign-generated comments include unique substantive
information); see also Cary Coglianese, Citizen Participation in Rulemaking: Past, Present, and Future, 55 DUKE L.J.
943, 952–59 (2006) (raising concern that e-rulemaking will increase number of comments but not range of
viewpoints).

92

Benjamin, supra note 54, at 904–05 (discussing costs associated with duplicative comments); see also Jeffrey S.
Lubbers, A Survey of Federal Agency Rulemakers’ Attitudes About E-Rulemaking, 62 ADMIN. L. REV. 451 (2010)
(describing a survey of officials involved in rulemaking that found a widespread view that e-rulemaking increased
total amount of participation but that there was rarely useful information or new arguments in the additional
comments).

93

Livermore, Eidelman & Grom, supra note 19, at 992.

23

argues that “powerful cultural patterns,” including “the popular equation in the United States of
democratic voice with casting a vote,” reinforce this “plebiscite assumption.”94 The conflict
between public and expert perception could lead to some commenters operating under a false
understanding of the weight that will be given to their views.
The interviews provided a variety of perspectives concerning how agencies respond to
mass comment campaigns and to expressions of opinion contained in public comments more
generally. The agency officials that we interviewed were uniform in their position that the noticeand-comment process is not a vote (i.e., agencies don’t tabulate comments “pro” and “anti” and
then choose the more popular position), but they have a wide array of approaches to addressing
opinions expressed in comments. Specifically, there seem to be three approaches taken by different
agencies: (a) opinions expressed in comments are irrelevant—only the factual content matters; (b)
opinions expressed in comments are relevant to the political perception of the rule and may affect
agencies’ activities on the Hill, color how agency leadership thinks about the viability of a
proposed rule, or affect how agencies roll out a rule if there is significant opposition; and (c)
opinions expressed in comments are relevant insofar as they express popular sentiment, and agency
decision makers (especially agency leadership) may consider that in deciding how to proceed,
though it should never be the sole factor in deciding whether or not to pursue a particular policy.
Some agencies appear not to track the overall number of comments received or the number
of times a particular comment was received. Agencies are well aware that organizations orchestrate
mass comment campaigns, and it is obvious that these campaigns will affect the comments the
agency receives. Some of the agency personnel we interviewed viewed the opinions expressed in
mass comment campaigns as mostly amounting to statements of preference.
C. Technological Responses to Mass Comments
Technologies have emerged that help agencies grapple with the large quantities of
duplicative comments that can result from mass or computer-generated comment campaigns.
Submissions in response to mass comment campaigns often include many duplicate comments,
which existing software can easily identify. The most important relevant software is known as “deduplication” (or “de-duping”) software. A de-duping program is software that scans each comment
and then compares it to every other comment that the agency has received.95 The program will
then identify the degree of overlap between each of the comments and group those comments that
appear to derive from a common source. For instance, if the content of a comment is 90% identical
to another comment or more and the comments are more than a few words long, it is safe to assume
that the submitters either coordinated with each other when preparing their comments or that they
both used a common source document that one or both of them slightly modified (or, as we shall
discuss below, that the comment was computer-generated). The vast majority of the time, this
pattern arises when an organization has supplied text to its members and urged them to either

94

Cynthia R. Farina et al., Rulemaking 2.0, 65 U. MIAMI L. REV. 395, 431 (2011).

95

A de-duping program can, of course, only process comments that are in electronic form. With respect to paper
comment submissions, an agency may, in theory, scan the comments and then use an optical character recognition
(OCR) program to convert the file into an electronic form. The electronic version can then be run through a de-duping
program. The integrity of the OCR process depends upon the quality of the underlying physical document.

24

submit the comment verbatim or modify it slightly so that each comment is largely, if not entirely,
identical.
De-duping programs allow the agency to set the threshold at which a comment is flagged
as likely being part of a mass comment campaign. For instance, if the agency sets the threshold at
90% identity, any comment that has 90% or more of overlapping content will be grouped with
other such comments; any comment that is less than 90% identical will not. De-duping software
only focuses on the actual words in each document and the order in which they appear. For
example, if a submitter took a form comment and changed most of the words to synonyms (e.g.,
“happy” to “glad”), the deduping program would not recognize the comment as being duplicative.
By batching identical and nearly identical comments in this way, de-duping software
greatly reduces an agency’s burden in processing comments in rulemakings involving mass
comment campaigns. For example, consider the following hypothetical scenario. An agency issues
a proposed rulemaking regulating Issue XYZ. Organization ABC supports the rule. ABC sends an
email to its members with a request to submit a public comment in support of the rule to
Regulations.gov. The email includes a four-paragraph sample comment and also asks that the
commenters include a sentence or two that explains how Issue XYZ relates to them. 60,000
members file a comment, and about half of them add the extra sentence or two. Organization LMN
opposes the rule and also sends out an email to its members, providing them with draft text to
submit in opposition to the rule and encouraging them to explain precisely how the regulation
would harm them. 200,000 members file a comment. Of these, 100 submit LMN’s text verbatim,
80 reproduce that text and add a sentence or two on how the regulation will hurt them, and 20
reproduce the comment and provide an extensive analysis on exactly how the regulation would
cause a specific set of harms.
When the agency processes the comments received, the de-duping software will
immediately identify these two separate campaigns and batch the comments. For the ABC
campaign, it can simply ignore the 30,000 comments that are 100% identical. Rereading the same
text 30,000 times would be an extravagant waste of time and taxpayer dollars and contribute no
new information to enhance the rule-writing process. And the software can make short work of the
other 30,000 nearly identical comments, having an agency official take a quick look at the added
language in each comment and decide if it adds substantive, new information.
The process for the LMN campaign is slightly more complicated. The 100 identical
comments can be ignored, and the 80 nearly identical comments can be quickly processed,
assuming that the added sentences contain little by way of empirical data. For the 20 comments
that contain extensive additional information, the agency will need to spend more time with each
one. Indeed, they may not even be flagged by the de-duping software, depending on the extent of
the changes (e.g., if the submitter pastes in a 500-word comment and then adds 500 additional
words, the de-duping program will not flag the comment unless it is set at 50% overlap or lower).
This simple example illustrates two key points. First, de-duping software can massively
decrease the processing burden for agencies. Second, unique or partially unique comments are
much more challenging for agencies to process than identical or very nearly identical comments.
For the latter, de-duping software can reduce the marginal processing time for each iteration of a
mass comment to zero (if the comments are identical) or close to zero (if the comments are almost
identical).

25

Such de-duplication software has been in use for well over a decade. Stuart Shulman
created some of the earliest deduplication software for the rulemaking context as a result of
research funded by the National Science Foundation.96 For example, Shulman de-duplicated the
public comment dataset from a 2013 school nutrition rulemaking in order to be able to quickly
reveal the substance of the comments and analyze them. Shulman de-duplicated the polar bear
rulemaking in 2007 (660,000 comments) and the national monuments rulemaking in 2017 (3.3
million comments).
Our interviews suggested that today most agencies use some sort of de-duplication tool,
though there is significant variation in how they do so. First, some agencies use a tool built into
FDMS, the federal toolkit for searching, viewing, downloading, and reviewing comments on
proposed federal rules. Others have their own program, others use contractors that have deduplication programs, and still others allow individual comment processors to use de-duplication
tools but don’t have any agency-wide prescribed tool.
While software makes it easy to spot identical or near-identical comments, agencies still
need to set the policies by which they decide how much overlap must exist between comments
before they qualify as being part of a mass comment campaign (and therefore do not review them).
De-duplication programs let an agency set the level of overlap (e.g., 90%), and different agencies
use different thresholds, though none appears to require 100% overlap for something to qualify as
being part of a mass comment campaign.
To the extent that a participant in a mass comment campaign adds unique information (e.g.,
submits the organizer’s form comment but then adds a sentence saying “I am personally supportive
of this rule because . . . .”), agencies generally review the unique information even if the identical
content is batched and treated as a group. Some agencies do not post all iterations of comments
received on Regulations.gov if they qualify as being part of a mass comments campaign. For
instance, certain agencies just post a representative example.
IV.

Malattributed Comments

Malattributed comments differ from mass and computer-generated comments in two
critical ways that raise particular legal and policy issues. First, they involve a direct falsehood; the
submitter makes an assertion about its identity that is untrue. Second, they may cause harm not
just to the notice-and-comment process but to some individuals outside that process, namely, those
whose names have been used. Some have gone so far as to characterize malattributed comments
as a form of identity theft. In this section we examine these and other claims.
A. Legal Issues Raised by Malattributed Comments
Two sorts of legal issues arise with regard to malattributed comments. First, many people
might assume, and some have asserted, that submitting a malattributed comment is illegal, indeed
criminal.97 Several members of Congress have requested the Department of Justice to undertake

96

Shulman, supra note 79.

97

See, e.g., In the Matters of Nicholas Confessore, 33 FCC Rcd. 11808 (2018); id. at 11821 (Rosenworcel, Comm’r,
dissenting) (noting that in the net neutrality rulemaking millions of “people had their identities stolen and used to file

26

criminal investigations and prosecutions (though it has not done so); at least two state Attorneys
General launched criminal investigations into the FCC net neutrality rulemaking; and the common
phrase “fraudulent comment” is itself an assertion that the submitter has violated the law.
However, some of this rhetoric may have gotten out ahead of legal realities.
Whether this activity constitutes a crime is central to the appropriate agency response for
three reasons. First, if it does, then one important federal response would be prosecution, and it
would be incumbent on the agency to refer significant examples to the Department of Justice.98
Second, if the activity is criminal, then the agency has a stronger obligation to discourage or
prevent it than would be the case if it is problematic but unregulated. And third, if this is criminal
activity, that signals a societal judgment that the problem here is serious indeed.
The second set of legal issues arise under the Administrative Procedure Act. Here the
question is whether the APA requires an agency to rely on malattributed comments, forbids it to
do so, or just has nothing to say on the matter.
Our role is not to reach definitive legal conclusions, particularly as the directly relevant
caselaw is non-existent. Rather, we flag the critical questions that agencies and prosecutors must
confront.
1. Criminal Prohibitions
A number of possible criminal prohibitions might conceivably apply to malattributed
comments—fraud, making false statements, computer crime, identity theft.99 The following
discussion touches on the two theories that seem most frequently mentioned.
(a) Fraud
The standard definition of fraud has five elements: (1) a false statement of a material fact,
(2) knowledge on the part of the defendant that the statement is untrue, (3) intent on the part of the
defendant to deceive the alleged victim, (4) justifiable reliance by the alleged victim on the
statement, and (5) injury to the alleged victim as a result.

fake comments, which is a crime under both federal and state laws.”); Catherine Sandoval, Reply Comments, In the
Matter of Restoring Internet Freedom, FCC 17-60, at 8 (Aug. 30, 2017) (“False filings based on stolen identities are
neither anonymous speech, nor protected speech; they constitute federal and state crimes.”).
98

Indeed, the Permanent Subcommittee on Investigations report lamented that “[o]nly one agency contacted by the
Subcommittee—the CFTC—said that it had referred suspicious activity to the Federal Bureau of Investigation (‘FBI’).
Other agencies, including the CFPB, the Department of Labor, and the FCC, all were aware of comments submitted
under false identities regarding their rules, but took little action to address them.” Subcommittee Report, supra note
10, at 17. In addition, it may be unlikely that the Department of Justice will undertake criminal prosecutions in this
space. See Matthew Minor, Remarks at the Administrative Conference of the United States Symposium on Mass and
Fake Comments in Agency Rulemaking, supra note 13 at 131-133.

99

Possibly relevant federal statutes include 18 U.S.C. § 1341 (2018) (mail fraud); 18 U.S.C. § 1343 (2018) (wire
fraud); 18 U.S.C. § 1037 (2018) (email fraud); 18 U.S.C. § 1030 (2018) (Computer Fraud and Abuse Act); 18 U.S.C.
§ 371 (2018) (conspiracy to defraud the United States); 18 U.S.C. § 1505 (2018) (obstructing agency proceedings);
18 U.S.C. § 1001 (2018) (making false statements “in any matter within the jurisdiction of the executive . . . branch”);
and 18 U.S.C. §§ 1028, 1028A (2018) (identity theft). For a full, and skeptical, discussion of the possible applicability
of these provisions, see Herz, supra note 45, at 33–55.

27

Malattributed comments clearly meet some elements of this definition—for example, they
involve a false statement of fact. But several elements raise real issues. First, the alleged defrauded
party is the agency. But it will be rare that an agency will “rely on” the false identity set out in a
comment. For run of the mill comments, agencies do not “rely on” the identity of the commenter;
they consider it irrelevant and so ignore it. In general, there will be nothing to rely on; reading a
comment that purports to be from “John Smith,”100 how could the agency “rely on” it really being
from John Smith. Where the false identity is recognizable (Barack Obama, Elvis Presley, a wellknown NGO), then it is possible that the agency could conceivably rely on the putative identity of
the submitter, taking it more or less seriously in light of its source. But it will not do so because it
will recognize the falsehood. Finally, if the submitter claims to be someone with relevant personal
experience, then (a) the relevant falsehood is not the name but the content of the submission, and
(b) it would not be reasonable for the agency to meaningfully rely on such assertions without
further investigation or confirmation.
For similar reasons, it could be difficult to satisfy the fifth prong, actual injury to the
defrauded party. A loss of public confidence in the rulemaking process is arguably an injury to the
agency, but not the sort of tangible harm to person or property that fraud generally requires.
Finally, fraud requires that the false statement of fact be material. Again, if the actual identity of
the commenter does not matter, which is often the case, then a malattribution is not material.
The analysis is similar under the various federal fraud statutes, of which there are many.
Submission of a comment will almost always involve the use of a wire or of the mails, thus coming
within the ambit of the wire fraud101 and mail fraud statutes.102 But both these statutes require that
the perpetrator be attempting to obtain “money or property” from the defrauded party. Even if
there is an ultimate financial interest, someone attempting to influence agency policymaking is not
trying to obtain money from the agency.103
There is one specific fraud provision that in some circumstances might reach malattributed
comments. For a century and a half, federal law has made it a crime to conspire to defraud the
United States. The current version, 18 U.S.C. § 371, makes it a crime to “conspire either to commit
any offense against the United States, or to defraud the United States, or any agency thereof in any
manner or for any purpose.” The courts have read the italicized language broadly, going beyond
common-law fraud and extending to “any conspiracy for the purpose of impairing, obstructing, or
defeating the lawful function of any department of Government.”104 There still needs to be some
sort of trick or deceit, but using false identities could be such. If the submitter is attempting to
influence the final decision by making the agency think, for example, that a raft of non-existent
individuals support a particular outcome, the submission can be seen as an attempt to impair the

100

As 3,997 submissions in the Net Neutrality rulemaking were. Hitlin, Olmstead, & Toor, supra note 4, at 4.

101

18 U.S.C. § 1343.

102

18 U.S.C. § 1341.

103

See Cleveland v. United States, 531 U.S. 12, 26–27 (2000) (concluding “that § 1341 requires the object of the fraud
to be ‘property’ in the victim’s hands” and that a state license does not qualify). If the effort to get the government to
give the supposed fraudster a license is not an attempt to obtain money or property, an effort to get the government to
adopt a particular regulation is not a fortiori.

104

Haas v. Henkel, 216 U.S. 462, 479–80 (1910) (upholding convictions under this provision where the defendants
had submitted false information to the Department of Agriculture, thereby skewing its published statistics).

28

lawful functions of the agency. Even if it is unlikely to succeed in doing so, that failure is irrelevant
to the existence of a conspiracy to impair, obstruct, or defeat lawful functions. And if the agency
is influenced by false information in a submission, its functions have been impeded or impaired in
the sense that it failed to reach the “right” result. Furthermore, submission of a huge number of
comments—whether malattributed or not—with the purpose of slowing down the agency could
perhaps be understood as an effort to “obstruct.”
On the other hand, this setting is quite different than those in which prosecutions under
§ 371 are generally brought. These prosecutions—so-called “Klein conspiracies”—are most
common in tax cases; they also arise where there is a legal obligation to disclose information that
a private party has withheld. Such conspiracies directly obstruct agency activities in a way that
phony names on comments do not. Additionally, the difficulty of showing substantial harm to the
federal government or an agency will disincline any prosecutor to pursue such a case. Finally, the
charge here is conspiracy, not fraud, so the usual elements of conspiracy would have to be shown.
(b) Making False Statements
Moving away from fraud-based crimes, the obvious basis for a possible prosecution is the
prohibition on making “false, fictitious, or fraudulent” statements to a federal agency found at 18
U.S.C. § 1001. This is a sweeping prohibition; unlike fraud, it does not require a showing of
financial or property loss to the government or reliance by the government.105 Most elements of
the crime seem satisfied here.
1. Though not “fraudulent,” a malattributed comment does make a “false” and “fictitious”
“statement or representation” in asserting that it is submitted by someone other than the
actual submitter.
2. The falsehood is knowing or willful. It is true that whoever is programming the computer
or authorizing the submissions does not know of each specific misidentification, but that
person does know that the misidentifications are being made.
3. A notice-and-comment rulemaking would seem to be “a matter within the jurisdiction” of
the agency conducting the rulemaking.
One issue remains: the false statement has to be material, meaning it “has a natural tendency
to influence, or [is] capable of influencing, the decision of the decisionmaking body to which it
was addressed.”106 Use of a false name is not “material” unless the effect or influence of a comment
hinges on who submitted it. If the agency is taking account only of the content of the comment,
not the identity of its author, then the malattribution seems immaterial. Using random names from
the phone book to misidentify the source of a comment would in that case not be a material
misstatement.107

105

See United States v. Richmond, 700 F.2d 1183, 1188 (8th Cir. 1983); United States v. Lichenstein, 610 F.2d 1272,
1278 (5th Cir. 1980).

106

Neder v. United States, 527 U.S. 1, 16 (1999).

107

See also Matthew Minor, Remarks at the Administrative Conference of the United States Symposium on Mass and
Fake Comments in Agency Rulemaking, supra note 13 at 125 (stating that “proving fraud here is hard to do” under
the materiality requirement in 18 U.S.C. § 1001).

29

A subcategory of malattributed comments could potentially violate § 1001, however.
Suppose a comment falsely claims to be from someone with extensive relevant expertise and
experience—a Ph.D. research chemist, a twenty-year line employee in the relevant industry, a user
of a product the agency proposes to ban, the owner or renter of property in the neighborhood of a
regulated facility. Because the person’s supposed unique, relevant experience would give the
comment more weight, that misstatement would be material. And if simply by using a particular
person’s name that information about background could be communicated, then just the false name
could be material.
It seems likely that such comments have been filed in federal agency rulemakings, though
instances would be hard to identify. A recent SEC rulemaking does provide an example, though it
arose prior to, rather than in comments on, the issuance of a proposed rule. In 2018, the SEC held
a roundtable regarding proxy rules and invited follow-up submissions, of which it received about
five hundred. In announcing the proposed rule, the Commission Chair, Jay Clayton, invoked
several of these:
Some of the letters that struck me the most came from long-term Main Street
investors, including an Army veteran and a Marine veteran, a police officer, a
retired teacher, a public servant, a single Mom, a couple of retirees who saved for
retirement, all of whom expressed concerns about the current proxy process.108
Later reporting put those letters in a different light. All had been assembled, organized, and
written by an industry group funded by supporters of the SEC proposal.109 The retired teacher did
sign her letter but had not written it; the veterans were the brother and cousin of the chair of the
industry group; the single mom did not write her letter; the retired couple were the in-laws of the
head of the industry group and when contacted had no recollection of ever writing any such letter;
the public servant reported that she had been contacted by a public affairs firm, that she did not
know what a proxy adviser is, and “[t]hey wrote [the letter], and I allowed them to use my name
after I read it. I didn’t go digging into all of this.”110
Whether the reporting is accurate, and whether any of this violated § 1001, the incident
does flag the possibility of bespoke comments that make material misstatements. Those may or
may not include the name of the submitter. The sort of malattributed comments that have generated
attention and concern to date are quite different. They are duplicative and generic, make no
representations as to background or expertise, and do not use recognizable names of experts.
Finally, one of the ways in which malattributed comments can be misleading, as we discuss
below, is that they make it appear that more commenters hold a particular view than is the case.

108

Statement of Jay Clayton, SEC Chairman, at Open Meeting on Proposals to Enhance the Accuracy, Transparency
and Effectiveness of Our Proxy Voting System (Nov. 5, 2019), https://www.sec.gov/news/public-statement/statementclayton-2019-11-05-open-meeting.

109

Zachary Mider & Ben Elgin, SEC Chairman Cites Fishy Letters in Support of Policy Change, BLOOMBERG (Nov.
19, 2019, 10:03 AM), https://www.bloomberg.com/news/articles/2019-11-19/sec-chairman-cites-fishy-letters-insupport-of-policy-change [https://perma.cc/ER4B-NZ7P].

110

Id. All in all, the interest group got about two-dozen people with connections to the organization to submit letters.
The group’s president insisted, by the way, that his mother- and father-in-law had known about the letter they
supposedly submitted: “They are 80-some-years-old. This happened months ago. I’m sure it’s not top of their minds.”
Id.

30

(This is true of large-scale, computer-generated comments generally; most malattributed
comments are just an example of that phenomenon in which the computer has attached a random
name to the comment.) A malattribution could perhaps be a false statement that is material not
because the agency cares who submitted the comment but because it cares that someone did. This
theory turns on the complicated question, discussed above, of whether and how an agency should
give weight to the number of comments taking a particular position.
2. The Administrative Procedure Act
Two separate issues arise under the APA. First, some have argued that it violates the APA
for agencies to accept malattributed comments and/or fail to remove them from the docket. Put
most strongly, the argument is that if there are a significant number of malattributed comments in
the docket the rulemaking is fatally tainted and must be abandoned. Presumably, on this reading,
the APA imposes an affirmative duty on agencies to prohibit submission of malattributed
comments and to police that prohibition conscientiously. The second argument is the opposite; it
holds that agencies have an obligation to accept malattributed comments and to give them
whatever weight they deserve. On this reading, it would violate the APA to do exactly what the
first reading says the APA requires.
Of course, it could be that both of these readings are mistaken and the APA is silent on this
matter. On this understanding, an agency could prohibit their submission and refuse to consider
them; but it could also instead choose to consider them along with all the other comments they
receive, giving each whatever weight it is due.
(a) An Obligation to Accept and Consider Malattributed Comments?
The APA requires agencies not just to accept but also to consider comments.111 Courts
have broadened this obligation by requiring that when issuing a final rule agencies respond to all
significant comments.112 Arguably, these obligations extend to malattributed comments just like
any other comment. To be sure, an agency will frequently conclude that the comment does not
require a response,113 and its malattribution, if detected, could be one factor supporting that
conclusion.114 But under one reading, the APA would require that an agency accept, review, and,
if there is something important and substantive in the comment, consider and respond to a
malattributed comments just like any other comment. And if that is true, agencies could not
prohibit submission of malattributed comments or remove malattributed comments from the
docket.

111

5 U.S.C. § 553 (2018) (providing that agency shall issue a final rule “[a]fter consideration of the relevant matter
presented”).

112

See, e.g., Del. Dep’t of Nat. Res. & Envtl. Control v. EPA, 785 F.3d 1, 15 (D.C. Cir. 2015); Cement Kiln Recycling
Coal. v. EPA, 493 F.3d 207, 225 (D.C. Cir. 2007); Grand Canyon Air Tour Coal. v. FAA, 154 F.3d 455, 468 (D.C.
Cir. 1998).

113

See Thompson v. Clark, 741 F.2d 401, 408 (D.C. Cir. 1984) (explaining that APA § 553(c) “has never been
interpreted to require the agency to respond to every comment . . . no matter how insubstantial”).

114

Cf. Mendelson, Rulemaking, Democracy, supra note 53, at 1378 (noting that an agency could “announc[e] that
anonymous comments will receive less weight, particularly when such comments purport to be informed by an
individual’s own experience”).

31

On the other hand, agencies unquestionably are free to impose reasonable requirements on
the form and content of public comments.115 The most obvious example is that notice-andcomment rulemaking always includes a comment deadline. An agency might consider late-filed
comments—different agencies have different practices. But it is universally accepted that an
agency can ignore a late-filed comment simply because it is a late-filed comment.116 To take a
more directly relevant example, agencies can, and many do, prohibit anonymous comments.117
Prior ACUS recommendations have not taken a position on whether agencies should or should not
accept anonymous comments, but one recommendation urges each agency to set a clear public
policy.118 The premise of this recommendation, of course, is that it is up to the agency whether it
will accept or reject anonymous comments. And if that is the case with regard to anonymous
comments, then it would seem to be the case for malattributed comments.
The foregoing assumes that an agency has a clearly stated policy regarding the
permissibility of malattributed comments. The argument that the agency must consider such
submissions is more plausible if the agency has never indicated that it will not do so.
(b) An Obligation to “Cleanse” the Docket of Malattributed Comments?
The opposite argument would be that the APA prohibits agencies from considering
malattributed comments, or even that it imposes an affirmative obligation to weed them from the
docket. Such a claim was made by many observers regarding the net neutrality rulemaking.119

115

See Dooling, supra note 75, at 905–15 (discussing such requirements, including civility and not revealing
confidential information).

116

See, e.g., Mont. Sulphur & Chem. Co. v. U.S. EPA, 666 F.3d 1174, 1195 n.12 (9th Cir. 2012) (“EPA was not
required to consider these untimely comments . . . .”); Reytblatt v. U.S. Nuclear Regulatory Comm’n, 105 F.3d 715,
723 (D.C. Cir. 1997) (concluding that agency need not respond to late comments even if it had indicated that it would
consider them); Bd. of Regents of the Univ. of Wash. v. EPA, 86 F.3d 1214, 1222 (D.C. Cir. 1996); JEFFREY S.
LUBBERS, A GUIDE TO FEDERAL AGENCY RULEMAKING 279 (4th ed. 2006).

117

In a 2011 ACUS report on agency practices regarding public comments, Professor Balla found that of the 25
agencies he studied, 15 (including EPA and several components of the Department of Transportation) allowed
anonymous comments and 10 (including the FCC, SEC, and Federal Reserve) required commenters to identify
themselves. See Steven J. Balla, Public Commenting on Federal Agency Regulations 23 (Mar. 15, 2011) (draft report
to the Admin. Conf. of the U.S.), https://www.acus.gov/sites/default/files/COR-Balla-Report-Circulated.pdf.

118

Admin. Conf. of the U.S., Recommendation 2011-2, Rulemaking Comments, ¶ 4, 76 Fed. Reg. 48789, 48791 (Aug.
9, 2011) (“The eRulemaking Project Management Office and individual agencies should establish and publish policies
regarding the submission of anonymous comments.”).

119

See, e.g., Klint Finley, FCC’s Broken Comments System Could Help Doom Net Neutrality, WIRED (Sep. 2, 2017),
https://www.wired.com/story/fccs-broken-comments-system-could-help-doom-net-neutrality (quoting Gigi Sohn as
stating that the agency might have an obligation under the APA to remove fake comments from the docket and that
“[a]t a bare minimum, they should investigate these comments and if they can’t actually remove the comments, they
can and should disregard them as part of their consideration of record”); Letter from Ellen F. Rosenblum, Or. Att’y
Gen. et al. to FCC 1 (Dec. 13, 2017), https://www.doj.state.or.us/wp-content/uploads/2017/12/ag_letter_12-132017.pdf (letter from eighteen state Attorneys General urging the FCC to delay action because “the well of public
comment has been poisoned by falsified submissions,” which makes it impossible to “listen to the public” as the APA
requires); Letter from Allen S. Hammond, IV & Catherine J.K. Sandoval to Sen. Roger Wicker et al. 2 (Dec. 13, 2017)
(“The FCC’s apparent tolerance of allegedly criminal behavior in its comment process [in the form of acceptance of
malattributed comments] falls far below the required standard of reasoned decision-making under the Administrative

32

In general, if a comment makes no contribution, the agency does not reject it, it just ignores
it. This makes sense. Removing useless comments would require effort, would be inconsistent
with the public’s opportunity to comment in the first place, and would make it impossible for a
reviewing court to review the full record and determine, among other things, whether the agency
had in fact considered and responded to all significant comments.120 If an agency must purge the
docket of malattributed comments, that must be because such comments are “worse than useless,:
that by their mere presence in the docket they cause affirmative harm that irrelevant or pointless
comments do not. It is hard to pin down precisely what this harm would be. The problematic aspect
of the comment—the false name—is easily ignored by the agency.121 If there is a harm under the
APA, it would not flow from the malattribution per se, but from legitimate comments being
drowned out or misled by the level of actual public support for a particular position. It is not clear
that either of those things actually occurs. More particularly, it is not clear that they would occur
if the agency is sufficiently aware of the dubious provenance of certain comments to be in a
position to purge them from the docket in the first place.
Perhaps the most potentially problematic malattributed comment would be a bespoke,
sophisticated comment rather than a mass comment that happens to have a phony name. Consider
a comment that purports to come from the General Counsel of a leading industry player or the head
of a prominent civil rights organization and reads as if it could be legitimate. Such a “deep fake”
comment could sow confusion at the agency and among other commenters and prompt a perceived
need among other stakeholders to respond. Because of the impact on other commenters, this is the
situation where the argument that the APA requires the agency to remove the comment—assuming
it is aware of the malattribution—seems strongest.
No court has considered any issue regarding malattributed comments and the APA, even
when such a challenge could have been raised. When the FCC issued its final net neutrality order,
opponents promised that they would challenge the order in court on grounds related to the
malattributed comments.122 Several petitions for review asserted that one of the Order’s legal
Procedures [sic] Act.”); see also Katherine Krems, Note, Crowdsourcing, Kind of, 71 FED. COMM. L.J. 63, 76 (2018)
(“When there is false information on the record, this information overshadows real public comments that reflect public
sentiment and contravenes the APA’s procedures meant to properly inform agencies of public opinion in decisionmaking processes.”).
120

See Dooling, supra note 75, at 917–20. Of course, an agency can set reasonable requirements—for example of
civility or not revealing confidential information—for comments and police those requirements. Id. at 905–15. But
comments in violation of those requirements are not merely unhelpful, they do affirmative harm. The affirmative harm
from malattributed comments is much less clear, as discussed earlier.

121

Some of the objections to the presence of malattributed comments in the rulemaking docket rest not on the fact that
the comments bear an incorrect name but that they are computer-generated or fake mass comments—what looks like
submissions from thousands of people is in fact from only one—and therefore misleading with regard to public
sentiment.

122

See, e.g., Brian Fung, FCC net neutrality process ‘corrupted’ by fake comments and vanishing consumer
complaints, officials say, WASH. POST: THE SWITCH (Nov. 24, 2017), https://www.washingtonpost.com/news/theswitch/wp/2017/11/24/fcc-net-neutrality-process-corrupted-by-fake-comments-and-vanishing-consumer-complaintsofficials-say/ (quoting Evan Greer of Fight for the Future as stating that “this will absolutely show up in court if we
get there”); see also Karl Bode, The FCC Is Blocking a Law Enforcement Investigation Into Net Neutrality Comment
Fraud, VICE (Dec. 12, 2017), https://www.vice.com/en_us/article/wjzjv9/net-neutrality-fraud-ny-attorney-generalinvestigation (“Expect the agency’s failure to police comment fraud to play a starring role in these legal arguments to
come.”).

33

defects was that it “conflicts with the notice-and-comment requirements of 5 U.S.C. § 553”123 but
no party actually raised the issue of malattributed comment to the court during briefing.124
(c) Agency Reliance on Malattributed Comments
The final question is whether it would violate the APA for an agency to read or to rely on
a comment submitted under a false name. The rulemaking provision itself, § 553, in no way
restricts what the agency can consider or who it can listen to. Rather, any such restriction would
rest on the requirement of “reasoned decision-making” embedded in the prohibition on arbitrary
and capricious agency action.125 Such determinations are case-specific. It would not be reasoned
decisionmaking to rely on malattributed comments as a measure of public sentiment or to rely on
a comment that purported to be from an authority in a relevant field when it was not. But if a
comment is relevant, factually accurate, and communicates something of value, there is nothing
arbitrary and capricious in an agency making use of what it has to offer, regardless of whether the
sender put someone else’s name on it. Thus, there would seem to be no per se rule allowing or
prohibiting agencies to rely on malattributed comments.
A. Policy Issues Raised By Malattributed Comments
In this section, we elaborate on the concerns raised by malattributed comments in particular
and also discuss how agencies can discourage submission of malattributed comments and handle
malattributed comments once they are discovered. At the same time, we acknowledge that a
malattributed comment may nevertheless contain useful content.
1. Misleading the Agency
Because malattributed comments, by definition, contain a falsehood, an obvious concern
is that the agency may be misled. The misleading might take either of two forms: the agency could
be misled with regard to the identity of the commenter, and the agency might be misled as to public
opinion, mistakenly viewing the phony comments as indicators of broader public support for a
particular position than actually exists.
(a) Commenter Identity
With regard to the first, in general, the agency simply will not notice the name of the
commenter. If the agency receives 10,000 very similar, computer-generated comments, no one is
paying attention to the names under which they are submitted, whether they are false or real. One
cannot be misled by something of which one is unaware. If the submission is not computergenerated—a unique comment filed under a false name—the falsehood is irrelevant for purposes

123

Pet. for Review at 2, Ctr. for Democracy and Tech. v. FCC, No. 18-1068 (D.C. Cir. Mar. 5, 2018); Pet. for Review
at 2, New York v. FCC, No. 18-1055 (D.C. Cir. Feb. 22, 2018).

124

The D.C. Circuit’s opinion, which was largely but not entirely in the agency’s favor, does not mention the
“fraudulent comments” issue. See Mozilla Corp. v. FCC, 940 F.3d 1 (D.C. Cir. 2019).

125

See 5 U.S.C. § 706(2)(a).

34

of the agency’s deliberation. The agency will take the comment for what it is worth; the name adds
nothing to its weight and will not affect how it is treated.126
Now suppose the name is one that someone in the agency recognizes. This is the “deep
fake” scenario described above. For example, it may be an important researcher or advocate in a
relevant field, or the general counsel of a prominent regulated entity. In this situation, the identity
of the commenter may matter for the agency’s deliberations. The agency could give particular
weight to such a comment. But it would be highly unlikely for the falsehood to go unnoticed. The
very background knowledge that makes the name recognizable will make it hard for someone to
pull the deception off. This is especially the case in a situation where the purported commenter’s
interests are well known to the agency, perhaps because of repeat interaction. However, for an
agency that does not regulate often, or that regulates only in certain domains infrequently, this
might be harder to ensure.
The malattributions that often grab observers’ attention involve using the name of a famous
(sometimes dead) person. But these are not misleading because it is apparent that the name is false.
For example, in the net neutrality rulemaking, there were multiple submissions from “Barack
Obama” and from “Ajit Pai.”127 This does not result in any actual deception; no rulemaking official
would think that the former President or the FCC Chair had submitted the comment. Same for
submissions from “Elvis Presley.”
Our discussions with agency officials are consistent with the foregoing. Their own sense
is that consequential instances of pseudonymous submissions are extremely rare, if not
nonexistent. Of course, we have not done a thorough study and by definition the victim of a
successful deception is unaware of having been deceived. Nonetheless, we credit these statements
because they reflect actual experience, they are consistent with what one would expect, and we are
unaware of a single demonstrated instance to the contrary.
There is one possible setting, however, in which the concerns about the agency being
misled may be more serious. Suppose a comment is not from a recognizable name but asserts that
the submitter has particular experience that appropriately goes to the weight of the comment. For
example, the commenter claims to have done research in the area, or to possess “situated
knowledge,”128 or to have had direct personal experience, or to be a person who will be directly
regulated or benefitted by the proposed rule. All of those people possess information that members
126

See, e.g., Letter from Ajit Pai, Chairman, FCC, to Rep. Michael E. Capuano (Apr. 12, 2018) (“Despite any
suggestion that the public comment process was somehow ‘flawed’ or ‘tampered with’ by the alleged submission of
comments under false names, any such activity did not affect the Commission’s actual decision-making . . . .”); Letter
from Thomas M. Johnson, Jr., Gen. Counsel, FCC, to Eric Schneiderman Att’y Gen., New York (Dec. 7, 2017) (“[T]he
Commission does not make policy decisions merely by tallying the comments on either side of a proposal to determine
what position has greater support, nor does it attribute greater weight to comments based on the submitter’s identity.”).

127

See, e.g., “Barack Obama,” ID 1051157755251, Restoring Internet Freedom, WC Docket 17-108 (May 11, 2017),
https://ww.fcc.gov/ecfs/filing/1051157755251 (submission from “Barack Obama” of “1600 Pennsylvania Avenue,
Washington, DC,” objecting to the “unprecedented regulatory power the Obama Administration imposed on the
internet” and “Obama’s . . . power grab”).

128

See Cynthia R. Farina, Dmitry Epstein, Josiah Heidt, & Mary J. Newhart, Knowledge in the People: Rethinking
“Value” in Public Rulemaking Participation, 47 WAKE FOREST L. REV. 1185, 1187–88, 1197 (2012) (describing
“situated knowledge” as “information about impacts, ambiguities and gaps, enforceability, contributory causes,
unintended consequences, etc. that is known by participants because of their lived experience in the complex reality
into which the proposed regulation would be introduced”).

35

of the general public do not and that the agency may find valuable. They may also have a stake
that should counsel caution in taking their assertions at face value. For both reasons, the agency
would want to know who the source of the comment is. An anonymous comment that claimed to
be from a person in such categories would be somewhat suspect; a signed comment may carry
more weight. If the name is a malattribution, and the actual submitter does not have the
qualifications claimed, there is a real risk of inappropriate reliance on the comment. Moreover,
suppose a rulewriter found that comment helpful but wanted to double-check its provenance. An
internet search might reveal the falsehood, but it might reveal nothing, or might appear to confirm
the biographical claims made in the comment.
This risk seems real but slim. We are not aware of real-world examples of such
submissions. That does not mean they have not occurred. In the real world, the malattributed
comments that have gotten attention were duplicative rather than bespoke; they do not make
individualized claims about the submitter. In addition, the real problem here is not the
malattribution so much as the biographical misrepresentation. The malattribution may make it
harder to uncover the relevant falsehood but is not itself misleading. Thus, the problem here is
actually the distinct one of accuracy in the assertions within comments. It is entirely possible for
commenters submitting under their own name to misrepresent their experiences, expertise, or even
views. The SEC proxy rule proceeding is an example.129
(b) The Weight of Public Support or Opposition to the Proposed Rule
The second concern is that the agency will be misled as to public sentiment. Malattributed
comments are often, though not necessarily, a form of computer-generated comments. Such was
the case in the Net Neutrality rulemaking, for example. Millions of individuals did not sit down
and prepare comments that they submitted under someone else’s name. A handful may have done
so, but presumably almost all the malattributed submissions involved a computer taking a prepared
text, or writing a text, and then randomly attaching actual names and email addresses to the
comment. As with computer-generated comments, part of what observers object to here is that
what looks like a set of mass comments submitted by millions of concerned individuals is in fact
just the effort of a single submitter. To the extent this is the concern, the malattribution is largely
irrelevant. Perhaps, however, an agency might think the 100,000 identical comments with different
names are more likely to be from different individuals than are 100,000 identical anonymous
comments, in which case malattributed computer-generated comments are more misleading. This
is especially problematic to the extent that public comments are understood by agencies as
providing insight into public sentiment.
2. Harms to Individuals
Unlike mass and computer-generated comments, malattributed comments can have
impacts outside the agency and the rulemaking process, imposing harms on the people whose
names and email addresses are used without permission. Many or most will never be aware that
they have supposedly submitted a comment in a federal rulemaking, and many or most may not

129

See supra text accompanying notes 108-110.

36

care. Even if using someone’s name and address on a comment does not constitute identity theft
under federal law,130 it still may be harmful to the person whose name is used in this manner.
Two sorts of harms can be imagined. The first is psychological. It would be understandable
that a person who learned that their name was used to submit a comment would be annoyed or
angry, especially if they disagreed with the content of the comment. The harm is somewhat
abstract; unlike standard identity theft, the victim’s bank account is intact. But for some people,
the distress or anger will be quite real.131 The second possible harm is reputational. For a
malattributed comment in a regulatory docket to cause reputational harm, it would have to be
noticed by someone who changes their opinion of the purported commenter for the worse. The
obscurity of the rulemaking process may make this unlikely, and we are not aware of any instance
in which it has occurred. Still, all it takes is one viral tweet by someone with a large following
about a comment considered benighted or outrageous to do serious harm to the ostensible author
of the comment. Quantifying these harms may well be impossible. Individual views on how
seriously to take them will vary. While some people will not care or perceive themselves to be
harmed at all, others may see themselves as victims of identity theft.
3. Discouraging Malattributed Comments
The e-rulemaking program has taken several recent steps to discourage the submission of
malattributed comments. For example, the user notice on regulations.gov now includes the
following under the heading “Terms of Participation”:
Public comments help agencies develop regulations; we encourage comments from
all viewpoints. Comments submitted to Regulations.gov should be the submitter’s
own comments or be submitted with the commenter’s permission. The development
of federal regulations is within the jurisdiction of the U.S. Government’s executive
branch agencies. It is a violation of federal law to knowingly and willfully make a
materially false, fictitious, or fraudulent statement or representation including false
statements about your identity or your authority to submit a comment on someone
else’s behalf, in relation to the development of such federal regulations, including
through comments submitted on Regulations.gov. See 18 U.S.C. § 1001.
Subject to 18 U.S.C. § 1028(c), it is also a violation of federal law to knowingly
use, without lawful authority, a means of identification of another person in

130

See Herz, supra note 45, at 51–55.

131

For examples, see Letter from Brittany Ainsworth et al., to Ajit Pai, Chairman, FCC (May 25, 2017) (letter from
27 individuals whose names and email addresses were used to submit comments without their involvement or
permission complaining that “someone stole our names and addresses, publicly exposed our private information
without our permission, and used our identities to file a political statement we did not sign onto” and calling on the
agency to remove these “fraudulent comments” from the docket and notify “all proper authorities”); Bode, supra note
122 (complaining that “the agency told me there was nothing it could do after someone hijacked my identity to claim
I falsely supported killing net neutrality protections”), https://www.vice.com/en/article/wjzjv9/net-neutrality-fraudny-attorney-general-investigation; Press Release, A.G. Schneiderman Releases New Details On Investigation Into
Fake Net Neutrality Comments (Dec. 13, 2017), https://ag.ny.gov/press-release/2017/ag-schneiderman-releases-newdetails-investigation-fake-net-neutrality-comments (quoting, among others, unidentified individual as saying “I’m
sick to my stomach knowing that somebody stole my identity and used it to push a viewpoint that I do not hold”).

37

connection with the violation of any federal law or the commission of a felony
under state or local law. See 18 U.S.C. § 1028(a)(7).
By clicking the submit button, you are verifying that you are not making any
materially false, fictitious, or fraudulent statement or representation regarding
your identity or your authority to submit on someone else’s behalf with regard to
the comment you are submitting on Regulations.gov, and that you are not using,
without lawful authority, a means of identification of another person, real or
fictitious, in connection with any comment you are submitting on
Regulations.gov.132
This notice implies that a malattributed comment could be a federal crime. This may be
part of a deterrence strategy on the part of the government to discourage anyone from sending
malattributed comments. Whether users are likely to see this, in a user notice that contains several
other paragraphs of policies, is uncertain. Moreover, although it may discourage some individual
submitters from using a false name, it is unlikely to have any impact on large-scale operations.
Agencies have to make decisions about how to treat malattributed comments, once
suspected or discovered. Because of the novelty of this issue, many agencies do not have protocols
for how to resolve whether a comment is malattributed and, if so, policies on how to handle that
comment in the docket. Questions that such policies would address include whether an agency
should strive to resolve a question about a comment’s provenance, or merely flag the potential
issue. Also, and in line with the user notice described above, to the extent that criminal action is
under consideration for particular malattributed comments, agencies may need to make staff
available to assist with any investigations or prosecutions.
B. Technological Responses to Malattributed Comments
The technology readily exists to authenticate users and is in widespread use in many
contexts. Common techniques include secure login, two-factor authentication, biometric
authentication using facial recognition or fingerprint, answering security questions or verifying
names against a database (as is the case in voter registration), or clicking an additional “I agree”
button to acknowledge and agree to terms of service. However, agencies currently do not have the
technology in place to authenticate those filing comments in the way a government department
authenticates someone applying for a driver’s license or a commercial website authenticates
someone buying a product to prevent credit card fraud.
While tools are not in place to authenticate someone’s identity, agencies do use tools to
ensure that a commenter is a human instead of a bot. These tools are primarily a response to
computer-generated comments but, by imposing a “speed bump” on the commenting process, they
may also help to reduce malattributed comments.

132

User Notice, REGULATIONS.GOV, https://www.regulations.gov/user-notice (emphases added).

38

The addition of reCAPTCHA to regulations.gov is intended to help to “improve[] the
integrity of the commenting process.”133 CAPTCHA is an example of a “Turing Test”—a thought
experiement developed by Alan Turing to evaluate artificial intelligence—and stands for
“Completely Automated Public Turing Test to tell Computers and Humans Apart.”134 With
CAPTCHA, users are presented an image of a set of visually distorted letters and numbers and
asked to enter the same characters into a textbox. When CAPTCHA was invented nearly two
decades ago, it was believed that machines would not be able to complete this task since only
humans would be able to interpret what the distorted characters were. With advances in computing
power this is no longer true and techniques to defeat CAPTCHA have been created. CAPTCHA,
however, has also been reinvented to protect against these attacks. In 2018, Google announced
“reCAPTCHA v.3” which eliminates the need for any human interaction with CAPTCHA at all.
By using risk analysis algorithms that assign a “risk score” to every person browsing a website
using the tool, the software alerts administrators if fraudulent activity is detected.135
Also, regulations.gov now includes a comment application programming interface (API)
to allow authorized entities to post mass comment campaigns to Regulations.gov if they have been
verified by GSA using a commercial identity validation service. In the press release announcing
these changes, GSA indicated that this was “to assure such entities ‘are who they say they are.’”136
The service does not aim to verify the identities of individual commenters, however.
The public prominence of malattributed comments prompts a fresh look at whether
agencies should verify commenter identity, either on the front-end or after either an internal or
external review flags a comment as potentially malattributed. While authentication is a common
practice and technically straightforward in many circumstances, the practice would be in tension
with agency policies to permit anonymous comments.
V.

Computer-Generated Comments

A. Legal Issues Raised by Computer-Generated Comments
The APA requires agencies to provide an opportunity to comment to “interested
persons.”137 The term “interested” is undefined and is generally understood not to limit the scope
of potential commenters.138 The term “persons” is defined as follows: “person includes an
individual, partnership, corporation, association, or public or private organization other than an
133

Press Release, Gen. Servs. Admin., GSA Launches Updated Regulations.gov to Improve the Integrity (Feb. 17,
2021),
https://www.gsa.gov/about-us/newsroom/news-releases/gsa-launches-updated-regulationsgov-to-improvethe-integrity-of-public-commenting-02172021#:~:text=WASHINGTON%20%E2%80%94%20The%20U.S.%20
General%20Services,gov%20launching%20February%2018%2C%202021.&text=%E2%80%9CThe%20new%20R
egulations.gov%20re,and%20mobile%2Dfriendly%20interface.%E2%80%9D.

134

The “imitation game” experiment proposed by Turing was invented as a way of approaching the hard question of
“Can machines think?”. See A.M. Turing, Computing Machinery and Intelligence, 59 MIND 433 (1950).

135

reCAPTCHA: Easy on Humans, Hard on Bots, https://www.google.com/recaptcha/intro/v3.html?ref=techmoon
(last visited Apr. 1, 2021).

136

See Gen. Servs. Admin., supra note 130.

137

5 U.S.C. § 553(b).

138

See Herz, supra note 75, at 357–58.

39

agency.”139 When Congress passed the APA, it would not have contemplated that a computer
might send a comment. But the definition is instructively broad; it is not limited to natural persons,
and courts have read the word capaciously.140 Moreover, because a person must set a computergenerated comment in motion, the section 551 definition is arguably met in any event.
As described above, agencies are required to respond to significant issues raised in
comments. As of this writing, no courts appear to have interpreted this requirement in light of
computer-generated comments. During the interviews, agency staff expressed skepticism that a
computer-generated comment would bring content or issues to the rulemaking docket that were
not otherwise raised by other comments. But these staff also expressed their commitment to
reviewing all comments, regardless of origin, to ensure compliance with their obligations to
consider and respond to comments.
It is theoretically possible—if highly unlikely at this time—that a person would challenge
an agency action on the basis of its failure to adequately account for the substance of a computergenerated comment that was not otherwise presented in other comments. Should such
circumstances arise, courts may determine whether the suit should move forward based on factors
such as whether the petitioner can demonstrate the reliability or authenticity of the computergenerated comment.141 It is also possible that authentication technology might exclude either
computer-generated comments or ordinary comments that raise unique significant issues. If the
agency’s obligation to consider and respond to significant comments does not change in such
circumstances, technological means of identifying computer-generated comments would have to
account for this overarching obligation.
B. Policy and Technical Issues Raised by Computer-Generated Comments
The policy issues raised by computer-generated comments overlap significantly with those
already identified for mass and malattributed comments. For example, the presence of computergenerated comments may undermine public confidence in the rulemaking process or draw down
agency resources. Many of the issues presented by computer-generated comments, however are
technical. First, one issue is the ability of agencies to identify computer-generated comments. In
2019, an experiment demonstrated the ease with which bots mimic human speech, therefore
making it difficult to distinguish computer-generated comments from comments directly submitted
by persons.142 The focus of this experiment was a comment period on a waiver from federal
requirements requested by the Idaho Medicaid program. A text generation model was utilized to
139

5 U.S.C. § 551(1).

140

See e.g., O’Rourke v. U.S. Dep’t of Justice, 684 F. Supp. 716, 718 (D.D.C. 1988) (holding that “person” includes
non-citizens and collecting cases); Neal-Cooper Grain Co. v. Kissinger, 385 F. Supp. 769, 776 (D.D.C. 1974) (foreign
government or instrumentality thereof is “person”).

141

In the evidentiary context, courts have managed to assess the admissibility of electronically stored information
(which may include computer-generated information) on the basis of the Federal Rules of Evidence; for example,
proponents must demonstrate the information’s relevance, reliability, authenticity, and so on. See, e.g., Lorraine v.
Markel Am. Ins. Co., 241 F.R.D. 534, 538 (D. Md. 2007). This is not to suggest that the Federal Rules apply to
administrative records; rather, this example is offered to demonstrate that courts may find useful analogies that may
be applied consistently with their equitable powers and authority under the APA.

142

Max Weiss, Deepfake Bot Submissions to Federal Public Comment Websites Cannot Be Distinguished from Human
Submissions, TECH. SCI. (Dec. 17, 2019), https://techscience.org/a/2019121801/.

40

submit one-thousand comments on the proposed waiver. The inputs for this model were thousands
of comments submitted in response to Medicaid waivers previously requested by a number of other
states. These inputs were used to train the model to employ search-and-replace techniques as a
means of generating comments, which were submitted automatically to the Centers for Medicare
and Medicaid Services at random intervals.
Following the submission of the computer-generated comments, subjects were recruited to
judge whether particular comments in the docket were submitted by a bot or human. On average,
the respondents—all of whom had previously demonstrated competency in identifying
conspicuous bot texts—correctly classified less than half of the comments. Performance was
particularly poor in the context of computer-generated comments, in that less than one-third were
correctly recognized. These results indicate that the computer-generated comments were as a
general matter plausibly human, therefore making consistent sorting of such submissions a nontrivial exercise for the agency. At the conclusion of the experiment, the researcher revealed the
computer-generated comments and requested that CMS withdraw the bot submissions from
consideration.
These results are consistent with assessments of computer-generated comments that have
occurred outside of the context of experimentation. A variety of analyses have emphasized that
search-and-replace algorithms, and the resulting comment-to-comment variation in content,
enhance the difficulty of identifying computer-generated comments. As a result, “analysts have
struggled to pinpoint” the precise frequency with which computer-generated comments occur.143
Are there approaches for identifying computer-generated comments in a systematic
manner? The FCC’s net neutrality policy is a good place to turn in this regard, as researchers have
expended considerable energy identifying computer-generated comments that were submitted in
this particular rulemaking. Note that these approaches entail identifying computer-generated
comments in hindsight, as opposed to screening for such comments during the intake process.
One analysis focused on the text of net neutrality comments, searching for expressions
regularly contained in submissions.144 The analysis discovered combinations of phrases consistent
with the automated deployment of search-and-replace algorithms. Take, for example, the
following comment excerpt: “Americans, as opposed to Washington bureaucrats, deserve to enjoy
the services they desire.” This sentence repeatedly appeared in comments in numerous other
permutations, with “Americans” replaced by terms such as “people like me” and “individual
citizens.” Similarly, “the FCC,” “so-called experts,” and other analogous phrases substituted for
“Washington bureaucrats.” One result of this automation was the submission of large numbers of
comments that, while not identical, conveyed essentially equivalent sentiments. Another
characteristic of this process was the brevity of the resulting computer-generated comments.145
Increases in comment length multiply opportunities “for the appearance of ‘tells’ (e.g., repeated

143

Katherine Krems, supra note 119, at 71.

144

Jeff Kao, More Than a Million Pro-Repeal Net Neutrality Comments Were Likely Faked, HACKERNOON (Nov. 22,
2017),
https://hackernoon.com/more-than-a-million-pro-repeal-net-neutrality-comments-were-likely-fakede9f0e3ed36a6.

145

See Krems, supra note 119, at 75.

41

words, incorrect grammar, nonsensical sentiment) that the comment was not created by a
human.”146
Other analysis has examined over-time patterns of the submission of net neutrality
comments with identical and near-duplicate content, an approach that is useful for identifying mass
comment campaigns (regardless of human or computer submission). One such pattern is the receipt
of large numbers of comments at precisely the same moment.147 Researchers discovered, for
example, that on “nine different occasions, more than 75,000 comments were submitted at the very
same second—often including identical or highly similar comments.”148 Another pattern is
embodied by the submission of the following comment excerpt: “The unprecedented regulatory
power the Obama Administration imposed on the internet is smothering innovation, damaging the
American economy and obstructing job creation.” This text occurred in approximately a halfmillion comments.149 These comments were submitted at near-constant rates for given periods,
which were punctuated by interludes during which no such comments were received.150 This cycle
suggests that bots were turning on and off at specified intervals.
Another indication of the submission of computer-generated comments was repetition in
email addresses, in particular domains and locations exhibiting behavior inconsistent with human
messaging activity. The FCC, for example, determined that millions of comments were the product
of websites that produce one-off emails and are unable to receive messages. The agency also
discovered that hundreds of thousands of emails originated “from the same address in Russia.”151
The regular submission of computer-generated comments was also suggested by the nature
of the information submitted along with the comments themselves. When humans fill out
information, the resulting inputs are typically inconsistent. For example, name, address, and email
fields are often left blank, and individuals utilize varying formats. In the context of the
“unprecedented regulatory power” comments referenced earlier, however, fewer than ten
submissions failed to contain complete information. Furthermore, these names, addresses, and
emails exhibited unusual similarity in presentation. Finally, exceedingly few comments requested
that the FCC provide email confirmation of receipt. These attributes suggest that algorithms, rather
than humans, were the immediate sources of the submitted information.152
1. Current Agency Practices
In the interviews, agency staff expressed their awareness of computer-generated comments
having been submitted in a few rulemaking proceedings. Despite this awareness, the staff we
146

Weiss, supra note 142.

147

DAVID FREEMAN ENGSTROM ET AL., GOVERNMENT BY ALGORITHM: ARTIFICIAL INTELLIGENCE IN FEDERAL
ADMINISTRATIVE AGENCIES 60 (2020), https://www-cdn.law.stanford.edu/wp-content/uploads/2020/02/ACUS-AIReport.pdf.

148

Hitlin, Olmstead, & Toor, supra note 4, at 3.

149

See Chris Sinchok, An Analysis of the Anti-Title II Bots, MEDIUM, https://medium.com/@csinchok/an-analysis-ofthe-anti-title-ii-bots-463f184829bc.

150

See id.

151

Grimaldi & Overberg, supra note 6.

152

See Sinchok, supra note 149.

42

interviewed did not report systematic approaches to identify computer-generated comments. One
agency discovered computer-generated comments through a Wall Street Journal report on the
rulemaking, as well as the rulemaking team’s identification of a number of unusual comments.
These comments consisted of strings of nonsensical words, which made the agency suspicious that
the submissions were not generated by humans.
Despite the availability of tools discussed earlier under mass and malattributed comments,
implementing approaches to systematically identify computer-generated comments was not a high
priority for the agencies we interviewed. Agency staff characterized the discovery of computergenerated comments as requiring substantial effort, a resource-intensive undertaking that is not
worth the dedication of agency bandwidth. In general, the interviews revealed that agencies are
not focused on the issue of computer-generated submissions in and of themselves. Rather, they
indicated greater concern about mass and malattributed comments whose detrimental attributes
may be deepened by computer generation. Despite this concern, the agency staff we interviewed
reported as their primary concern the need to identify and respond to significant issues that
comments raise, regardless of a given comment’s source.
One reason for the lack of attention to computer-generated comments in and of themselves
may be that agencies are already using de-duplication tools to address mass comments. Computergenerated comments, in other words, are not seen as creating problems in rulemaking proceedings
other than increasing the volume of comments received by agencies—thereby turning the matter
into one of mass comment management. As an agency official put it during an interview,
computer-generated comments essentially present agencies with a de-duplication task. With the
utilization of de-duplication software, the unique content in computer-generated comments can be
readily identified.
As this perception indicates, agency staff generally expressed that the generation of
comments by computers is not, in and of itself, an important attribute of submissions. The point
was repeatedly made during the interviews that it is the substance of comments that matters, as
opposed to the identity of submitters or the volume of comments. Agencies were not overly
concerned that computer-generated comments convey insights that have not separately been
communicated through human comments. That said, agencies emphasized that they would not
exclude a computer-generated comment on the basis of the source of the submission, but would
consider whether it raised significant issues requiring agency consideration.
2. Threat Versus Practice
In sum, there is currently a disjuncture between conceptions of computer-generated
comments from the vantage points of technologists and the agency staff we interviewed.
Technologists warn of a present—and especially future—in which computer-generated comments
effectively mimic human content, thereby making prevention and detection an impossibility. The
agency staff we interviewed, by contrast, were not overly concerned with such scenarios at this
time. They saw de-duplication tools as being adequate for the task and did not seem anxious to
experiment with additional technologies to streamline the comment review process.
Notwithstanding current perceptions, in the years ahead, it will be important to monitor whether
the technologies that enable mass, malattributed, and computer-generated comments threaten to
undermine the perceived legitimacy of the notice-and-comment process and the ability of agency
officials to make sense of and consider comments thoughtfully.

43

VI.

Innovations to Enhance Participation and Commenting

The foregoing discussion has identified some of the risks associated with mass,
malattributed, and computer-generated comments. These risks are real, and agencies must
undertake appropriate measures to ensure that they protect the integrity and value of the noticeand-comment process. But technology can present opportunities as well as challenges. As we have
already seen, agencies extensively use de-duplication software to help them process mass comment
campaigns. And this is only a preview of what agencies can accomplish with newly emerging
technologies. This section explores technologies that are not yet in widespread use but that might
enhance and supplement the notice-and-comment process.153
A. Summarization Technologies and Enhancing the Value of Public Commenting
Although our focus has been on mass, malattributed, and computer-generated comments,
an additional salient and urgent opportunity for regulators is using new technologies to enhance
the process of reviewing public comments.154
While our interviews found that agencies are not currently using such tools, affordable
technology is on the horizon to help agencies more easily make sense of public comments, helping
those reading the rules, not simply to save time, but to better analyze, spot patterns in, and
understand public comments.
The GSA’s innovative technology unit 18F comments: “It takes enormous amounts of staff
time, resources, and taxpayer dollars to manually analyze written public comments submitted to
agencies through various committees and many other channels.” Their experts go on to
recommend the need to “explore if Natural Language Processing and Artificial Intelligence can
semi-automate, streamline, and expedite the public comment process, and whether any additional
policy or guidance might be required to create a standard approach.”155
NLP techniques can help comment reviewers both summarize and sort comments, helping
them to extract the most important substantive information from the comments.156 For example,
these techniques can be used to identify those parts of comments that bear on questions that are of
particular interest to rule-writers, or that contain relevant legal, technical, or operational
information.157

153

See generally Beth Simone Noveck, The Innovative State, DAEDALUS (forthcoming 2021) (special issue on the
administrative state in the United States in the twenty-first century).

154

See Fake It Till They Make It, supra note 2 (testimony of Beth Simone
https://financialservices.house.gov/uploadedfiles/hhrg-116-ba09-wstate-noveckb-20200206.pdf.

Noveck),

155

Aditi Rao, Ben Peterson, & Andrew Suprenant, Synthesizing Public Comments: Phase 2 Report (on file with
authors).

156

See Livermore, Eidelman, & Grom, supra note 19, at 980 (discussing “needle-in-the-haystack” and “forest-forthe-trees” challenges of mass rulemakings).

157

The field of NLP encompasses a wide range of technologies that use computational tools to convert natural
language artifacts into a format that can be processed and analyzed using computational and statistical tools. NLP
techniques include deduping software as well as:

44

While still a challenging task, researchers and entrepreneurs have developed tools for
summarization, including shortening and extracting the most relevant portions of documents. To
sort information, one technique that can be used is topic modeling. In brief, a topic model is a
computational text analysis technique that extracts patterns in the semantic content in a corpus of
documents, generating a list of topics (which are distributions over the vocabulary in a corpus) and
characterizing every document as a distribution over those topics.158 Topic modeling makes it
possible to automatically and quickly sort textual information into semantic categories.
Both Google and Microsoft announced in 2019 that they had built systems capable of
summarizing an enormous range of texts, including news, fictional stories, instructions, emails,
patents, and legislative bills.159 The MIT Center for Constructive Communication conducted
research on large-scale Twitter data sets.160 Its Electome project, for example, extracts semantic

•
•
•
•
•
•
•

Flesch-Kincaid Readability: a measure of the difficulty or clarity of written English. The readability
score of a text is based on the average number of words per sentence and the average number of
syllables per word. Other readability metrics include the Gunning Fog Index and the Spache Index.
Linguistic Inquiry and Word Count (LIWC): a software application that counts “words in psychologyrelevant categories,” such as whether words are associated with honesty or deception or track
individual thinking styles.
Plagiarism Detection: a technique for detecting similarity in written texts, with the goals of identifying
plagiarism or copyright infringement.
Automated Document Summarization: an application that processes larger texts, or multiple texts, as
inputs with the goal of generating summary texts that convey a condensed version of the original input
texts.
Sentiment Analysis: a measure of words based on positive or negative valence, as a way to estimate the
opinions or attitudes expressed in a written text.
Topic modeling: a family of computational tools used to discover the latent thematic structure within a
collection of documents.
Word Embeddings: a technique for mapping words or phrases into a vector space that compactly
represents semantic content. One technique for generating word embeddings involved “skip-gram”
where a model is trained to use a word to predict surrounding words in a document.

158

For key works in the relevant NLP literature, see, e.g., Kincaid et al., Derivation of New Readability Formulas
(Automated Readability Index, Fog Count, and Flesch Reading Ease Formula) for Navy Enlisted Personnel, Research
Branch Report 8-75 (1975); Adam Feldman, Opinion Clarity in State and Federal Trial Courts, in LAW AS DATA:
TEXT, COMPUTATION, AND THE FUTURE OF LEGAL ANALYSIS 407, 415–18 (Michael A. Livermore & Daniel N.
Rockmore eds., 2019); Yla R. Tausczik & James W. Pennebaker, The Psychological Meaning of Words: LIWC and
Computerized Text Analysis Methods, 29J. LANGUAGE & SOC. PSYCH. 24 (2010); BING LIU, SENTIMENT ANALYSIS:
MINING OPINIONS, SENTIMENTS, AND EMOTIONS (2015). David M. Blei, Andrew Y. Ng, & Michael I. Jordon, LATENT
DIRICHLET ALLOCATION, 3 J. MACH. LEARNING RES. 993 (2003); Mikolov et al., Distributed Representations of
Words and Phrases and Their Compositionality, in ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 26,
3136 (2013).

159

Patrick Fernandes, Miltiadis Allamanis, & Marc Brockschmidt, Structured Neural Summarization, published at
Int’l Conf. on Learning Representations (Feb. 2019), https://arxiv.org/pdf/1811.01824.pdf; see also Peter Liu & Xin
Pan,
Text
summarization
with
TensorFlow,
GOOGLE
AI
BLOG
(Aug.
24,
2016),
https://ai.googleblog.com/2016/08/text-summarization-with-tensorflow.html.

160

The Laboratory for Social Machines, which carried out some of this research, was incorporated into the MIT Center
for Constructive Communication in 2021. See LABORATORY FOR SOCIAL MACHINES, MIT MEDIA LAB,
https://www.media.mit.edu/groups/social-machines/overview/.

45

content from the entire corpus of Twitter data—billions of tweets—in order to summarize the core
political messages of the day and help drive election coverage.161
Such summarization and sorting processes sometimes combine automation with human
intelligence to make quick work of large data stores and overcome the biases that arise from using
automation alone. Journalists took advantage of such tools, for example, when they needed to
rapidly sift through the 13.4 million documents that comprised the so-called “Paradise Papers.”162
Public institutions have also used natural language data analytical techniques to make sense of
social media data. To help UNICEF and other actors craft more effective pro-immunization
messaging programs, researchers set out to monitor social media networks, including blogging
platforms, forums, Facebook, Twitter, Tumblr, and YouTube. They sought to analyze prevalent
conversation themes according to volume, types of engagement, and demographics; to identify
influencers across languages and platforms; and to develop specific recommendations for
improving messaging strategies across languages, platforms, and conversation themes. The
research methodology involved scraping text from conversations on social media platforms in
English, Russian, Polish, and Romanian, in order to be able to summarize them and identify key
patterns.163
A recent State Department project offers a simple illustration of how agencies could make
sense of rulemaking comments using a combination of artificial intelligence and human oversight.
In 2016, the State Department sought to improve its passport application and renewal process in
anticipation of an increase in the number of passport application and renewal forms. It ran an
online public engagement process to ask people what improvements they wanted, receiving almost
1,000 suggestions. In order to make rapid sense of those submissions, it used a third-party software
company, which applied a text-mining algorithm that scanned the highlighted text for responses
containing similar keywords in order to create summaries. The public was invited to proof and
make suggestions for how to improve those highlights, adding accountability but in a way that is
efficient. The combination of human and machine intelligence made it faster and easier to
summarize content than using an algorithm alone.
To date, application of NLP to public comments in administrative rulemaking has been
largely limited to de-duplication. While still under development, more advanced NLP techniques
could eventually assist agency personnel in identifying relevant substantive content within
comments and summarizing the information presented across a broad spectrum of comments. One
of the challenges for deploying summarizing technology in the context of rulemaking is that there
is often domain specific language that requires retraining the relevant models. However, for
important rulemakings likely to receive a large number of comments, this investment may well be
worth it. Furthermore, the addition of human oversight can provide a check on the performance of
machine learning applications, making it possible to evaluate and confirm the reliability of new
tools for summarization. While NLP tools can be used to augment, rather than replace, human
161

See THE ELECTOME, http://www.electome.org/.

162

Fabiola Torres López, How They Did It: Methods and Tools Used to Investigate the Paradise Papers, GLOB.
INVESTIGATIVE JOURNALISM NETWORK (Dec. 4, 2017), https://gijn.org/2017/12/04/paradise-papers/.

163

STEFAAN G. VERHULST & ANDREW YOUNG, THE POTENTIAL OF SOCIAL MEDIA INTELLIGENCE TO IMPROVE
PEOPLE’S LIVES (Sept. 24, 2017) (report for The Governance Lab), http://www.thegovlab.org/static/
files/publications/social-media-data.

46

review, the agency staff we interviewed expressed concerns related to how the use of new
technologies might interact with their legal obligation to review and respond to comments. This
legal uncertainty creates the risk that agencies may innovate slowly. Depending on their risk
tolerance, it may prevent them from adopting these technologies at all.
B. CrowdLaw: Innovations in Equitable Participation
In addition to improving the commenting process ex-post using new technologies, agencies
could also explore using complementary platforms and processes—ones already well-honed and
tested by other governments—to create new opportunities for public engagement, especially to
solicit information and expertise from more diverse and varied audiences as a complement to
notice-and-comment. Building on ACUS’ earlier work, we conclude our discussion of public
participation in rulemaking by looking at several contemporary examples of how governments are
enhancing citizen participation using new technology.164
Over the last decade, federal agencies have expanded citizen engagement through the use
of prize-backed challenges or what is sometimes called open innovation via the Challenge.gov
website. Since 2011 with the reauthorization of the America Competes Act,165 a hundred federal
agencies have run online challenges to tap the intelligence and expertise of the public.166 NASA
has regularly used prize-backed challenges to spur crowdsourcing of innovative solutions from the
public. The Asteroid Grand Challenge, for example, was focused on finding all asteroid threats to
human populations.167 Prize-backed challenges require agencies to articulate and define exactly
what information they need from the public and provide very transparent and specific criteria for
evaluating public submissions. With ten years of experience with prized backed challenges, there
may be useful insights for federal agencies to draw about how to improve public participation in
agency decision making.
Challenge.gov is one example of institutionalized public engagement or what is sometimes
referred to as “CrowdLaw,” namely the use of technology to engage the public in law-, rule-, or
policy-making. It is the idea that public institutions work better when they increase citizen
engagement by using new technologies to obtain diverse sources of information, insight and
164

As Michael Herz wrote in a 2013 ACUS report:
[T]he online world in general has come to be increasingly characterized by participatory and
dialogic activities, with a move from static, text-based websites to dynamic, multi-media platforms
with large amounts of user-generated content. At the heart of this move to “Web 2.0” have been
social media, blogs, Twitter, Facebook, YouTube, IdeaScale, wikis, Flickr, Tumblr, and the like.
Outside the rulemaking setting, federal, state, and local governments have enthusiastically jumped
on the social media bandwagon.

Michael Herz, USING SOCIAL MEDIA IN RULEMAKING: POSSIBILITIES AND BARRIERS (Nov. 2013) (report to Admin.
Conf.
of
the
U.S.),
https://www.acus.gov/sites/default/files/documents/Herz%20Social%20Media
%20Final%20Report.pdf.
165

America Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science
Reauthorization Act of 2010, Pub. L. No. 111-358, 124 Sta. 3982 (Jan. 4, 2011).

166

About, CHALLENGE.GOV, https://www.challenge.gov/about (last visited Apr. 1, 2021).

167

Asteroid Grand Challenge, NASA.GOV, https://www.nasa.gov/content/asteroid-grand-challenge (last visited Apr.
1, 2021).

47

expertise at each stage of the law and policymaking cycle to improve the quality as well as the
legitimacy of the resulting laws, regulations, and policies, especially by engaging with
underrepresented communities.168 CrowdLaw does not describe one form of participation. Rather,
it describes a variety of different methods, tools and platforms that institutions use.
Expert sourcing, where officials crowdsource expert advice, is one example of how
government bodies are implementing more citizen engagement. The Federation of American
Scientists’ Congressional Science Policy Initiative invites hundreds of scientists to help draft
questions for Members of Congress to ask of committee witnesses. Such crowdsourcing, facilitated
by new technology, helps beleaguered staffers write more informed questions.169 The Governance
Lab at NYU uses videoconferencing to help coordinate online dialogues among experts to advise
government officials on a variety of topics. In Fall 2020, for example, it ran six deliberative
sessions at the behest of seven governments in Latin America to help them develop implementable
strategies for responding to specific public health challenges, including the improvement of mental
health services and combating misinformation.170
Some jurisdictions have used online collaborative drafting processes and platforms to write
policies and rules with the public, especially with expert members of the public. Instead of an
advisory committee or hearing with a handful of experts or writing rules entirely behind closed
doors, online collaborative annotation makes it possible to hear from a broader and deeper range
of experts and to focus their participation on specific comments on a document. In 2018, the
German government used an annotation platform to “expert source” feedback on its draft artificial
intelligence policy. By putting the draft on Hypothes.is, a free and open-source annotation tool,
the German Chancellor’s Office, working in collaboration with Harvard University’s Berkman
Center for Internet and Society, was able to solicit the input of global legal, technology and policy
experts. Using an annotation platform also made it possible for people to see one another’s
feedback, instead of a series of disconnected comments. One could envision an agency using
collaborative annotation to invite experts to annotate and comment on the text of a draft rule.
Many governments are experimenting with the use of random samples of members of the
public as a mechanism to obtain more legitimate forms of participation. New technology is making
it easier to assemble these representative samples of citizens, known as mini-publics, to weigh in
on a governing process. Small groups are known as citizen juries while larger random samples are
called citizen assemblies. For example, in the Brussels-Capitol region, a random sample of citizen
representatives serves on each parliamentary committee. Citizens ask questions and provide

168

Victòria Alsina & José Luis Martí, The Birth of the CrowdLaw Movement: Tech-Based Citizen Participation,
Legitimacy and the Quality of Lawmaking, 40 ANALYSE & KRITIK 337 (2018); see also Beth Simone Noveck,
CrowdLaw, in THE PALGRAVE ENCYCLOPEDIA OF INTEREST GROUPS, LOBBYING AND PUBLIC AFFAIRS (Phil Harris et
al. eds.) (forthcoming 2021).

169

CONGRESSIONAL SCIENCE POLICY INITIATIVE, https://fas.org/congressional-science-policy-initiative/ (last visited
Apr. 1, 2021).

170

See Smarter Crowdsourcing: Coronavirus, THE GOVERNANCE LAB, https://coronavirus.smartercrowdsourcing.org
(last visited Apr. 1, 2021).

48

advice.171 These processes could also be designed to elicit expertise and know-how relevant to
agency decision makers.
Similarly, some have suggested ideas such as administrative agencies empaneling a
thousand randomly selected citizens to provide oversight over agency decisionmaking.172 A
variation on this idea would use citizen juries to solicit information on agency agenda setting and
priorities,173 providing the citizen jurors with background materials generated by deliberative
polling before their discussions.174
Finally, instead of selecting a random sample, other institutions have relied on self-selected
participation using a variety of tools. In month-long online exercises known as “Evidence Checks,”
UK parliamentary committees invite experts, stakeholders, and members of the public to comment
on the validity of evidence on which a policy is based. The process begins when government
departments supply information to their respective committees about an issue. Each committee
publishes the information on a parliament.uk web page, and it is scrutinized by a wider pool of
invitees. The committee also presents specific questions and problems that it would like
participants to address. In contrast to a representative sample, this process allows a group of people
with relevant experience and expertise to identify gaps in research that require further review.175
Another example of self-selected participation was initiated by the New Jersey Department
of Education in March 2021 when that agency invited students, parents and educators across the
state to help inform the Department’s policymaking by responding to questions via All Our Ideas,
a free platform developed at Princeton University. All Our Ideas has been used in over 18,000
citizen engagement projects.176 The owner of the consultation uses the platform to write a series
of statements that are then randomly presented to the participant. People select the response they
prefer (or “I can’t decide” as a third answer) or they may submit their own response. As people
are repeatedly selecting between two randomly generated options, it is a faster and easier
mechanism for responding to a series of questions. This so-called “wiki survey” method of
showing people two pieces of information and having them choose between them and/or submit a
new item offers efficiency benefits over open-ended commenting and can be designed to draw on
participant expertise.

171

The Governance Lab, Belgian Sortition Models: Institutionalizing Deliberative Democracy, CROWDLAW FOR
CONGRESS, https://congress.crowd.law/case-belgian-sortition-models.html (last visited Apr. 1, 2021).

172

See David R. Arkush, Direct Republicanism in the Administrative Process, 81 GEO. WASH. L. REV. 1458 (2013).

173

BETH SIMONE NOVECK, SMART CITIZENS, SMARTER STATE: THE TECHNOLOGIES OF EXPERTISE AND THE FUTURE

OF GOVERNING 220 (2015).
174

See Reeve T. Bull, Making the Administrative State “Safe for Democracy”: A Theoretical and Practical Analysis
of Citizen Participation in Agency Decisionmaking, 65 ADMIN. L. REV. 611 (2014).

175

Nesta, UK Parliament Evidence Checks, https://www.nesta.org.uk/feature/six-pioneers-digital-democracy/ukparliament-evidence-checks (last visited Mar. 28, 2021).

176

MATTHEW SALGANIK, BIT BY BIT: SOCIAL RESEARCH IN THE DIGITAL AGE 111–15 (Princeton Univ. Press,
2017).

49

Recommendations
To help facilitate committee deliberations, we offer the following draft recommendations. The
general categories are technology, coordination and training, docket management, and
transparency. We could conceive of many variations of the ideas below, but offer these to help
the committee formulate its views.
Based on our research, we believe that mass, malattributed, and computer-generated comments
do not, at least currently, fundamentally undermine the notice-and-comment process. However,
such comments raise issues of sufficient significance that steps can and should be taken to
mitigate the difficulties emanating from them. Technology also presents opportunities for
enhancing public engagement in rulemaking.
The following recommendations lay out a variety of immediate and long-run actions for reducing
the challenges of mass, malattributed, and computer-generated comments and taking advantage
of technology-enabled participation.
Technology
Mass, malattributed, and computer-generated comments raise challenges for agencies at two
stages of the rulemaking process.
The first stage is comment submission. At this stage, one difficulty stems from the submission
of large numbers of comments. The GSA has recently taken important actions to help agencies
manage mass comments. For example, the current version of Regulations.gov includes an API
that facilitates the submission of comments in bulk. Another difficulty concerns the
authentication of the identity of the commenter. In this regard, Regulations.gov has implemented
two features. One feature is identity validation in the API, which enables authorized users to
submit comments in bulk. The other feature is reCAPTCHA, which is designed to screen out
commenters who are not humans.
We recommend that agencies, both those that use Regulations.gov and those that do not,
consider utilizing bulk submission, identity validation, reCAPTCHA, or similar tools in their
comment submission processes.
We recommend that agencies and relevant coordinating bodies stay abreast of developments in
the submission of mass, malattributed, and computer-generated comments, so that approaches to
combating difficulties arising from such developments can be implemented as needed.
The second stage at which mass, malattributed, and computer-generated comments raise
challenges is the processing of comments. One challenge in this regard is the submission of large
numbers of duplicate or near-identical comments. Our research indicates that it is commonplace
for agencies to use de-duplication software to identify the unique content. An additional
challenge concerns the identification of comments submitted under false identities and by
computers. Our research suggests that agencies to this point have devoted little attention to
identifying comments in their dockets that are malattributed or computer-generated.
We recommend that agencies continue to (or, if they have not already, begin to) utilize deduplication software to identify the unique content in submitted comments.
We recommend that agencies publish policies regarding the posting of duplicate and nearidentical comments. These policies should balance values such as user-friendliness,
transparency, and informational completeness. Options that could be considered include: posting
50

a single representative example with the count of the duplicates received and an option to view
all comments; breaking out non-identical content; and providing enhanced search options based
on the unique information content of comments.
We recommend that agencies and relevant coordinating bodies should encourage and stay
abreast of technology for identifying malattributed and computer-generated comments in the
docket.
We recommend that agencies and relevant coordinating bodies stay abreast of technologies that
can facilitate public participation outside the notice-and-comment process. Agencies often find
that supplemental public participation processes can be useful, and a wide range of technologies
can be used to structure meaningful dialogue between agencies and relevant publics.
Coordination and Training
We recommend that agencies and relevant coordinating bodies share best practices and relevant
innovations for addressing challenges and opportunities connected with mass, malattributed, and
computer-generated comments, and technologies related to supplemental public participation
processes.
We recommend that agencies work closely with relevant coordinating bodies to improve
existing technologies and develop new technologies to address issues associated with mass,
malattributed, and computer-generated comments.
•

The e-Rulemaking Program should provide a common de-duplication platform for
agencies to use, though agencies should be free to modify it or use another platform
as appropriate.

•

The e-Rulemaking Program and other relevant coordinating bodies should work with
agencies and private sector experts and vendors to develop technologies that respond
to common issues associated with mass, malattributed, and computer-generated
comments.

We recommend agencies offer opportunities for ongoing training and staff development to
respond to the rapidly evolving nature of technologies related to mass, malattributed, and
computer-generated comments, and supplemental public participation processes.
Docket Management
We recommend agencies post comments promptly to the rulemaking docket. If a comment is to
be included in the docket, it is important that it be posted promptly upon receipt. Evaluation and
control of mass, malattributed, and computer-generated comments depends in part on public
responses to them. Most obviously, someone whose identity has been used in a malattributed
comment may notice and flag the malattribution. Such external correction cannot occur unless
the problematic submission is made available.
We recommend that, if an agency decides to exclude or remove some or all duplicate,
malattributed, or computer-generated comments from the docket, it articulate such a policy in
advance [or provide a reasoned explanation after excluding]. The presumption is that an agency
must place all comments in the docket. To exclude a document requires an affirmative
justification that is articulated in law or in an agency policy. Existing examples include certain
agency prohibitions on threatening language or profanity, on anonymous comments, and on
copyrighted material.
51

We recommend that an agency policy against submission of malattributed comments provide
that if the agency is aware that it has received such a comment, it either retain the comment in
the docket but remove the malattribution (i.e., render it an anonymous submission) or remove the
comment from the docket altogether. Even if agencies adopt technological barriers to the
submission of malattributed comments, those methods are not likely to be perfect. Our analysis
finds that agencies do not have an obligation to affirmatively search the docket for malattributed
comments. But agencies are free to set reasonable policies concerning the public comment
process and reject comments that violate their policies. Agencies can also rely on comments that
violated their commenting policies (e.g. late comments) in some circumstances. If an agency
determines that a malattributed comment will remain in the docket, anonymization should be
used to protect the person whose identity has been used.
We recommend that agencies not discard computer-generated comments that it receives,
although those comments may be segregated and treated separately.
We recommend that any duplicative, malattributed, or computer-generated comment on which
an agency actually relies be placed and retained in the rulemaking docket. As noted, agencies
may choose to anonymize malattributed comments, and to segregate or flag computer-generated
comments, that are retained in the docket.
We recommend that, to the extent practicable, agencies should provide opportunities (including
potentially after the comment deadline) for individuals whose names have been attached to
comments they did not submit to identify and request removal of such comments from the
docket.
We recommend that agencies consider taking affirmative steps to identify comments that are
malattributed [or computer-generated] comments.
We recommend that, if an agency flags a comment as malattributed [or computer-generated] or
removes such a comment from the docket and the submitter provided electronic contact
information, the agency should notify the submitter of the agency’s action.
Transparency
We recommend agencies and relevant coordinating bodies consider providing materials that
explain to prospective commenters what information is useful to an agency in a public comment.
This could include various formats to reach different audiences (e.g., videos, FAQs).
We recommend that in NPRMs, NOIs, and ANPRs agencies ask specific questions and identify
particular information that would be useful in developing the proposal.
We recommend that, when publishing a final rule, agencies state whether they removed from
the docket any malattributed and/or computer-generated comments.

52