DOI: 10.7910/dvn/jb58on
title: Extracted Data for: Environmental Protection Agency (EPA) Science Inventory, Reports

description: This archive is a subset of the full Science Inventory archive; this archive includes all of the files marked as "Reports".
&lt;br&gt;
&lt;br&gt;
From the Original URL website (https://cfpub.epa.gov/si/index.cfm):
&lt;br&gt;
&lt;blockquote&gt;
 “The Science Inventory is a searchable database of research products primarily from EPA's Office of Research and Development. Science Inventory records provide descriptions of the product, contact information, and links to available printed material or websites.”
&lt;/blockquote&gt;
Also documented in the epaSI_readme.docx in this archive, the Science Inventory consists of 94,428 records. This repository captured PUBLISHED REPORTs as well as PRESENTATIONs, although in the end the files were not clearly categorized in any one way. The records were uncategorized in the above way upon download. Instead, the records are organized by index (Title and Entry ID number, plus page number of entry). &lt;br&gt;
&lt;br&gt;
The downloads from the Science Inventory (SI) occurred using a set of scripts that scraped the website, then downloaded, and then catalogued. The website is separated into three main categories:
&lt;ul&gt;
 &lt;li&gt;Reports&lt;/li&gt;
 &lt;li&gt;Presentations&lt;/li&gt;
 &lt;li&gt;All others&lt;/li&gt;
&lt;/ul&gt;
&lt;br&gt;
All of the types were downloaded together. The contents of each category are also presented in duplicate using their category, for findability. However, the exception to this are the NEPIS files (National Service Center for Environmental Publications, also known as NSCEP). These files have a special download mechanism, where downloads are restricted by the hour and frequency of download request. Thus, it is not clear that all NEPIS files are captured. The files are presented in archives by category (report, presentation, NEPIS), and then by page count, and then by all together. For files with duplicate filenames, the title of the item was used (from the scraped records), with “filler” words stripped out. 
&lt;br&gt;
&lt;br&gt;
Scripts:
&lt;br&gt;
&lt;br&gt;
&lt;b&gt;epa_SI_uncat_scraper1.py&lt;/b&gt; (Oct 29, 16KB)
&lt;br&gt;
Purpose: Scrapes EPA Science Inventory for all document types EXCEPT journals, creates a CSV index &lt;br&gt; Scope: Configurable page ranges (default 1-10) Features: Extracts metadata and download URLs from multiple document types (reports, books, data/software, etc.)
&lt;br&gt;
Main Features:
&lt;br&gt;
&lt;ol&gt;
 &lt;li&gt;&lt;b&gt;Multiple URL Columns:&lt;/b&gt; Auto-detects columns starting with 'url' (or you can specify them manually). Handles up to 31 URL columns.&lt;/li&gt;
 &lt;li&gt;&lt;b&gt;Intelligent Filename Logic:&lt;/b&gt;&lt;/li&gt;
 &lt;ul&gt;
 &lt;li&gt;&lt;b&gt;Single download per row:&lt;/b&gt; Uses native filename&lt;/li&gt;
 &lt;li&gt;&lt;b&gt;Multiple downloads per row:&lt;/b&gt; Adds title prefix (first 4 non-filler words)&lt;/li&gt;
 &lt;li&gt;&lt;b&gt;Duplicate filenames across records:&lt;/b&gt; Adds letter suffix (a, b, c, etc.)&lt;/li&gt;
 &lt;/ul&gt;
 &lt;li&gt;&lt;b&gt;Filler Word Filtering:&lt;/b&gt; Removes common filler words (to, the, of, and, at, in, for, a, an) when creating title prefixes.&lt;/li&gt;
 &lt;li&gt;&lt;b&gt;Two-Pass Processing:&lt;/b&gt;&lt;/li&gt;
 &lt;ul&gt;
 &lt;li&gt;Pass 1: Collects all download information&lt;/li&gt;
 &lt;li&gt;Pass 2: Downloads files and applies appropriate naming&lt;/li&gt;
 &lt;/ul&gt;
&lt;/ol&gt;
Key Changes from Original:
&lt;ul&gt;
 &lt;li&gt;&lt;code&gt;download()&lt;/code&gt; now returns &lt;code&gt;(success, native_filename)&lt;/code&gt; tuple&lt;/li&gt;
 &lt;li&gt;New &lt;code&gt;extract_title_prefix()&lt;/code&gt; function for creating prefixes&lt;/li&gt;
 &lt;li&gt;Tracks filename usage with &lt;code&gt;filename_counter&lt;/code&gt; and &lt;code&gt;filename_usage&lt;/code&gt; dictionaries&lt;/li&gt;
 &lt;li&gt;Automatically renames files after download based on conflict detection&lt;/li&gt;
 &lt;li&gt;Provides a conflict report at the end showing which filenames were duplicated&lt;/li&gt;
&lt;/ul&gt;
Usage:
&lt;pre&gt;&lt;code&gt;# python
process_csv(
 csv_file="your_file.csv",
 url_columns=None, # Auto-detect or specify ['url', 'url1', 'url2', ...]
 title_column='title', # Column with record titles
 output_dir='downloads',
 max_rows=None # For testing, set to a number
)&lt;/code&gt;&lt;/pre&gt;
&lt;br&gt;
&lt;b&gt;epa_multi_url_and_nepis_download3.py&lt;/b&gt; (Nov 5, 28KB)
&lt;br&gt;
Purpose: Purpose: Downloads files from CSV created by scrapers + handles NEPIS URLs. Enhanced version of the downloader with better NEPIS handling
&lt;br&gt;
Key feature: Has get_nepis_download_url() function that parses NEPIS popup pages to find actual PDF download links NEPIS handling: Constructs popup URL (Display=p%7Cf), parses HTML for PDF link, handles JavaScript patterns using BeautifulSoup/requests
&lt;br&gt;
Key additions: Handles multiple URL formats per record; Better filename sanitization; Title-based prefixes for files; Duplicate tracking; Statistics by domain
&lt;br&gt;
&lt;br&gt;
&lt;b&gt;epa_nepis_parse_manual1.py&lt;/b&gt; (Nov 6, 17KB)
&lt;br&gt;
Purpose: Separates NEPIS URLs from regular downloads for manual processing
&lt;br&gt;
Key feature: Creates separate CSV files for NEPIS vs non-NEPIS URLs
&lt;br&gt;
Use case: When automated NEPIS parsing fails, this prepares files for manual download
&lt;br&gt;
&lt;br&gt;
The workflow was:
&lt;ul&gt;
 &lt;li&gt;epa_SI_uncat_scraper1.py - Generate index of all documents&lt;/li&gt;
 &lt;li&gt;epa_multi_url_and_nepis_download3.py - Download most files (with best NEPIS handling)&lt;/li&gt;
 &lt;li&gt;epa_nepis_parse_manual1.py - For stubborn NEPIS files that fail automation, separate them for manual download Scraper scripts are available in the archive, and in GitHub.&lt;/li&gt;
&lt;/ul&gt;
&lt;br&gt;
As of 11/6/2025
&lt;ul&gt;
 &lt;li&gt;Scraped 94,031 entries&lt;/li&gt;
 &lt;li&gt;Of those, 1,718 are have NEPIS links associated with them&lt;/li&gt;
 &lt;li&gt;Of the non-NEPIS entries (92,313), there are 12,002 files from entries.&lt;/li&gt;
 &lt;li&gt; Of the NEPIS associated entries (that is, NEPIS links in any download column), 225 were download urls in columns that were not NEPIS, and of those, only 70 were actual downloads. The remainder of the 225 did not contain valuable content (that is, they either have “NTIS contact” listed, or a “follow the URL” sentence, or instruction to “contact the Program Officer” as the only page in a pdf.&lt;/li&gt;
&lt;/ul&gt;
&lt;br&gt;
Originally, an advanced search revealed that the records belonged to the following categories (number of records, category):
&lt;br&gt;232 ASSESSMENT DOCUMENTs
&lt;br&gt;1380 BOOKs
&lt;br&gt;1340 BOOK CHAPTERs
&lt;br&gt;276 COMMUNICATION PRODUCTs
&lt;br&gt;25 CRITERIA DOCUMENTs
&lt;br&gt;0 DATA/SOFTWARE
&lt;br&gt;177 EPA PUBLISHED PROCEEDINGS
&lt;br&gt;233 ETV DOCUMENTs
&lt;br&gt;147 EXTRAMURAL DOCUMENTs
&lt;br&gt;232 IRIS ASSESSMENTs
&lt;br&gt;23395 JOURNALs
&lt;br&gt;289 NEWSLETTERs
&lt;br&gt;191 NEWSLETTER ARTICLEs
&lt;br&gt;0 NON-EPA PUBLISHED PROCEEDINGS
&lt;br&gt;15 PAPER IN EPA PROCEEDINGS
&lt;br&gt;0 PAPER IN NON-EPA PROCEEDINGS
&lt;br&gt;34077 PRESENTATIONs
&lt;br&gt;4029 PUBLISHED REPORTs
&lt;br&gt;5 RISK ASSESSMENT GUIDELINES
&lt;br&gt;2516 SCIENCE ACTIVITies
&lt;br&gt;524 SITE DOCUMENTs
&lt;br&gt;407 SUMMARies
(subtotal: 69,490)
&lt;br&gt;
This would leave 24,938 as uncategorized records. However, scraping and downloading scripts were not able to detect these categories; therefore, “Reports”, “Presentations” and the NEPIS catalogue were the three categories where files were archived into specific groupings. (2025-10-29)
&lt;br&gt;
&lt;br&gt;
description: The below was extracted directly from the EPA Disclaimers page (https://www.epa.gov/web-policies-and-procedures/epa-disclaimers):
<br>
<br>
“Disclaimer of Endorsement Mention of or referral to commercial products or services, and/or links to non-EPA sites does not imply official EPA endorsement of or responsibility for the opinions, ideas, data, or products presented at those locations, or guarantee the validity of the information provided. Mention of commercial products/services on non-EPA websites is provided solely as a pointer to information on topics related to environmental protection that may be useful to EPA staff and the public. Copyright Status The U.S. Government retains a nonexclusive, royalty-free license to publish or reproduce these documents, or allow others to do so, for U.S. Government purposes. These documents may be freely distributed and used for non-commercial, scientific and educational purposes. Commercial use of the documents available from the EPA websites may be protected under the U.S. and Foreign Copyright Laws. Individual documents on the EPA website may have different copyright conditions, and that will be noted in those documents. Disclaimer of Liability With respect to documents available from the EPA website, neither the United States Government nor any of their employees, makes any warranty, express or implied, including the warranties of merchantability and fitness for a particular purpose, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Exit Disclaimer The Exit icon denotes that you are leaving the EPA website and entering an external link or a third-party site. EPA's official web site is www.epa.gov. EPA has provided this link because it provides additional information that may be useful or interesting and is being provided in a manner consistent with the intended purpose of the EPA website. Please note that EPA uses third-party sites to provide EPA content already on www.epa.gov in a different format. EPA cannot attest to the accuracy of non-EPA information provided by these third-party sites or any other linked site. EPA is providing these links for your reference. In doing so, EPA does not endorse any non-government websites, companies or applications. Also, please be aware that the privacy protection provided on the EPA.gov domain (EPA Privacy and Security Notice) does not apply to these third-party sites. To learn more about EPA and social media, please refer to& our social media page. Notice Information from the EPA website resides on numerous computer systems funded by the Agency. The use of the EPA websites may be monitored for computer security purposes. Any unauthorized access to the EPA website is prohibited and is subject to criminal and civil penalties under federal laws including, but not limited to, Public Law 99-474, the Computer Fraud and Abuse Act of 1986. Terms of Use for Geospatial Data These geospatial data and corresponding cartographic materials have been approved for use by the U.S. Environmental Protection Agency (EPA) as advised by the EPA Geospatial Advisory Committee (EGAC). This approved release is on the condition that neither the EPA nor the U.S. Government may be held liable for any damages resulting from its authorized or unauthorized use. These data and any corresponding products, services, or materials do not necessarily represent the EPA’s official position or viewpoint, expressed or implied. These content items are not intended for use in establishing liability or calculating cost recovery statutes of limitations. They cannot be relied upon to create any rights, substantive or procedural, enforceable by any party in litigation with the United States or third parties. Additionally, although these data have been processed successfully on EPA computer systems, no warranty expressed or implied can be made regarding the accuracy or utility of the data on any other system or for general or scientific purposes, nor shall the act of distribution constitute any such warranty. The Agency reserves the right to revise EPA-stewarded datasets pursuant to further analysis and review without public notice. Unless otherwise specified, geospatial data produced by the EPA is by default in the public domain and is not subject to domestic copyright protection under 17 U.S.C. § 105. Referenced data from non-EPA sources are neither inherently verified nor independently tested by the Agency in all circumstances. Permission to reproduce copyrighted items not produced by the EPA must be secured from the copyright owner. EPA strongly recommends careful attention be paid to metadata files associated with these data to better understand limitations, restrictions or intended use. The U.S. EPA shall not be held liable for improper or incorrect use of the data."