Package: python-edgar
Build a master index of SEC filings since 1993 withpython-edgarThe SEC filings index is split in quarterly files since 1993 (1993-QTR1, 1993-QTR2...). By usingpython-edgarand some scripting, you can easily rebuild a master index of all filings since 1993 by stitching quarterly index files together. The master index file can be then feed to a database, a pandas dataframe, stata, etc...An index file is a csv-like (pipe|separated) file that contains the following information:Company name (eg.TWITTER, INC)Company CIK (eg.0001418091)Filling date (eg.2013-10-03)Filling type (eg.S1)Filling URL on EDGAR (edgar/data/1418091/0001193125-13-390321.txt)Oncepython-edgaris finished downloading index files, you can open an index file withcsv.csvreaderorpandas.read_csvto have the data programmatically usable. Remember that the delimiter character is|!python-edgarcan be used as a library called from another python script, or as a standalone script.FeaturesCompliant: Follows fair acces guidelines established by the SEC athttps://www.sec.gov/os/accessing-edgar-dataEfficient: retrieve compressed archives instead of raw index file that are 10 times biggerImport as a library in your python project or run as a standalone scriptPython 3 only with 0 external dependencies (Python 3 only as of v3.0.0)UsageUsing python-edgar as a libraryInstall from pip in a virtualenvpipinstallpython-edgarCall the libraryimportedgaredgar.download_index(download_directory,since_year,user_agent,skip_all_present_except_last=False)Output2018-06-2312:41:46,451-DEBUG-downloadswillbesavedto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3o2018-06-2312:41:46,451-DEBUG-downloadingfilessince20172018-06-2312:41:46,451-INFO-6indexfilestoretrieve2018-06-2312:41:46,465-DEBUG-workercount:42018-06-2312:41:48,359-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2017/QTR3/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3o/2017-QTR3.tsv2018-06-2312:41:48,611-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2018/QTR2/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3o/2018-QTR2.tsv2018-06-2312:41:48,649-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2017/QTR4/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3o/2017-QTR4.tsv2018-06-2312:41:48,935-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2018/QTR1/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3o/2018-QTR1.tsv2018-06-2312:41:49,750-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2017/QTR2/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3o/2017-QTR2.tsv2018-06-2312:41:50,237-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2017/QTR1/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3o/2017-QTR1.tsv2018-06-2312:41:50,376-INFO-complete2018-06-2312:41:50,377-INFO-Filesdownloadedin/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3oUsing python-edgar as a standalone scriptDownload this repository as a zip ("Clone or Download" green button, > Download as zip.)Open your terminal inside that directory and runpython run.py -h. You can specify a destination directory for downloaded index files like-d edgar-idx(defaults to a temporary directory) and/or specify the year from which you want to build the index with-y 2017(defaults to current year).$pythonrun.py-y2017-ua"MyCompany edward@mycompany.com"2018-06-2312:41:46,451-DEBUG-downloadswillbesavedto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3o2018-06-2312:41:46,451-DEBUG-downloadingfilessince20172018-06-2312:41:46,451-INFO-6indexfilestoretrieve2018-06-2312:41:46,465-DEBUG-workercount:42018-06-2312:41:48,359-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2017/QTR3/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3o/2017-QTR3.tsv2018-06-2312:41:48,611-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2018/QTR2/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3o/2018-QTR2.tsv2018-06-2312:41:48,649-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2017/QTR4/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3o/2017-QTR4.tsv2018-06-2312:41:48,935-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2018/QTR1/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3o/2018-QTR1.tsv2018-06-2312:41:49,750-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2017/QTR2/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3o/2017-QTR2.tsv2018-06-2312:41:50,237-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2017/QTR1/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3o/2017-QTR1.tsv2018-06-2312:41:50,376-INFO-complete2018-06-2312:41:50,377-INFO-Filesdownloadedin/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpr2Nk3oCommon issuesRuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.Seehttps://pytorch.org/docs/stable/notes/windows.html#multiprocessing-error-without-if-clause-protectionYou will need to wrap your code inmain()function to be able to run it on Windows:defmain():importedgaredgar.download_index(".",2020,skip_all_present_except_last=False)if__name__=='__main__':main()I am using python 2 ...Python 2 support has been dropped as of October 2019. Seehttps://pythonclock.org.Stitch quarterly files to a master filepython-edgardoes only one thing and does it well: getting and cleaning uncompressed quarterly index files to your computer. Use command line tools, in the spirit of unix philosophy, to stitch these index files together and create our master index file.In this example, we calledpython run.pywithout arguments. It'll download every quarterly index file since 1993.pythonrun.py-y19932018-06-2313:00:16,855-DEBUG-downloadswillbesavedto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpcF1rx72018-06-2313:00:16,855-DEBUG-downloadingfilessince19932018-06-2313:00:16,856-INFO-102indexfilestoretrieve2018-06-2313:00:16,879-DEBUG-workercount:42018-06-2313:00:18,814-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2017/QTR4/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpcF1rx7/2017-QTR4.tsv2018-06-2313:00:19,026-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2017/QTR3/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpcF1rx7/2017-QTR3.tsv2018-06-2313:00:19,157-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2018/QTR2/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpcF1rx7/2018-QTR2.tsv2018-06-2313:00:19,543-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2018/QTR1/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpcF1rx7/2018-QTR1.tsv2018-06-2313:00:20,521-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2017/QTR2/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpcF1rx7/2017-QTR2.tsv2018-06-2313:00:20,719-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2016/QTR4/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpcF1rx7/2016-QTR4.tsv2018-06-2313:00:21,016-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2016/QTR3/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpcF1rx7/2016-QTR3.tsv2018-06-2313:00:21,134-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2017/QTR1/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpcF1rx7/2017-QTR1.tsv2018-06-2313:00:22,099-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/2016/QTR2/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpcF1rx7/2016-QTR2.tsv(...)dcw07x6zrrr0000gn/T/tmpcF1rx7/1993-QTR2.tsv2018-06-2313:00:54,378-INFO->downloadedhttps://www.sec.gov/Archives/edgar/full-index/1993/QTR1/master.zipto/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpcF1rx7/1993-QTR1.tsv2018-06-2313:00:54,423-INFO-complete2018-06-2313:00:54,424-INFO-Filesdownloadedin/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpcF1rx7Inspect the directory where our files where downloaded:$ls-lh/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpcF1rx7
total4964656drwx------104eswiacstaff3.3KJun2313:00.
drwxr-xr-x342eswiacstaff11KJun2313:01..
-rw-r--r--1eswiacstaff585BJun2313:001993-QTR1.tsv
-rw-r--r--1eswiacstaff580BJun2313:001993-QTR2.tsv
-rw-r--r--1eswiacstaff1.0KJun2313:001993-QTR3.tsv
-rw-r--r--1eswiacstaff2.8KJun2313:001993-QTR4.tsv
-rw-r--r--1eswiacstaff2.9MJun2313:001994-QTR1.tsv
-rw-r--r--1eswiacstaff2.3MJun2313:001994-QTR2.tsv(...)-rw-r--r--1eswiacstaff27MJun2313:002017-QTR3.tsv
-rw-r--r--1eswiacstaff27MJun2313:002017-QTR4.tsv
-rw-r--r--1eswiacstaff41MJun2313:002018-QTR1.tsv
-rw-r--r--1eswiacstaff31MJun2313:002018-QTR2.tsvHead to that directory so we can merge these files into a master file usingcat$cd/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpcF1rx7
$cat*.tsv>master.tsv
$du-hmaster.tsv2.3Gmaster.tsvNow you have this master index file. It's not sorted but that's easy to do (hint: Look into thesortcommand)Grab filings from a specific companyNow that we have downloaded the index files it becomes easy, with a bit of command line scripting, to quickly filter by company and extract URLs to the filings we want withgrep. In the following example we grep by CIK (1000045), store the output in an intermediate text file, which we re-open with cat and grep again by form 10-K. Prefix the paths withhttps://www.sec.gov/Archives/and you'll get the full URL.eswiac@mbppython-edgar(master)$grep-h1000045/var/folders/bv/2zbdkyyj14766dcw07x6zrrr0000gn/T/tmpvwOzOU/*>1000045.txt
eswiac@mbppython-edgar(master)$cat1000045.txt|grep-h10-K1000045|NICHOLASFINANCIALINC|10-K|2015-06-15|edgar/data/1000045/0001193125-15-223218.txt|edgar/data/1000045/0001193125-15-223218-index.html1000045|NICHOLASFINANCIALINC|10-K|2016-06-14|edgar/data/1000045/0001193125-16-620952.txt|edgar/data/1000045/0001193125-16-620952-index.html1000045|NICHOLASFINANCIALINC|10-K|2017-06-14|edgar/data/1000045/0001193125-17-203193.txt|edgar/data/1000045/0001193125-17-203193-index.html1000045|NICHOLASFINANCIALINC|10-K|2018-06-27|edgar/data/1000045/0001193125-18-205637.txt|edgar/data/1000045/0001193125-18-205637-index.htmlQuery the master index withqhttps://github.com/harelba/qallows you to run SQL directly on tabular data.Use with caution: q does not use indexes so running queries against the master index will be very slow since it's rather large. Sorting the master index or narrowing the data to a smaller subset will make search faster. Ultimately you want to load the master index file into a proper database that's able to handle the size.Some queries you may want to tryq "SELECT COUNT(1) FROM 1999-QTR4.tsv"q -d"|" "SELECT * FROM master.tsv where c1 = 1418091 and c3 = '10-Q' order by c4"LicenseMIT