Package: pyedgar
pyedgarPython package for downloading EDGAR documents and dataUsageThere are two primary interfaces to this library, namely filings and indices.filing.pyfiling.pyis the main module for interacting with EDGAR forms.Simple example:frompyedgarimportFilingf=Filing(20,'0000893220-96-000500')print(f)#output: <EDGAR filing (20/0000893220-96-000500) Headers:False, Text:False, Documents:False>print(f.type,f)# output: 10-K <EDGAR filing (20/0000893220-96-000500) Headers:True, Text:True, Documents:False>print(f.documents[0]['full_text'][:800])# Output:#                         SECURITIES AND EXCHANGE COMMISSION#                               WASHINGTON, D.C. 20549##                                     FORM 10-K##  (Mark One)#  /X/  Annual report pursuant to section 13 or 15(d) of the Securities Exchange#       Act of 1934 [Fee Required] for the fiscal year ended December 30, 1995 or##  / / Transition report pursuant to section 13 or 15(d) of the Securities#      Exchange Act of 1934 [No Fee Required] for the transition period from#      ________ to ________##  COMMISSION FILE NUMBER 0-9576###                             K-TRON INTERNATIONAL, INC.#               (EXACT NAME OF REGISTRANT AS SPECIFIED IN ITS CHARTER)##                 New Jersey                                22-1759452#     (State or other jurisdiction of         (I.R.S. Employer Identification No.)The forms are loaded lazily, so only when you request the data is the file read from disk or downloaded from the EDGAR website.
Filing objects have the following properties:path: path to cached filing on diskurls: URLs the EDGAR website location for the full text file and the index filefull_text: Full text of the entire.ncfiling (not just the first document)headers: Dictionary of all the headers from the full filing (i.e. not the exhibits). E.g. CIK, ACCESSION, PERIOD, etc.type: The general type of the document, extracted from the TYPE header and cleaned up (so 10-K405 --> 10-K)type_exact: The exact text extracted from the TYPE fielddocuments: Array of all the documents (between  tags). 0th is typically the main form, i.e. the 10-K filing, subsequent documents are exhibits.Each document in this array is itself a dictionary, with fields: TYPE, SEQUENCE, DESCRIPTION (typically the file name), FULL_TEXT. The latter is the text of the exhibit, i.e. just the 10-K filing in text or HTML.index.pyindex.pyis the main module for accessing extracted EDGAR indices.
The indices are created inpyedgar.utilities.indicesby the IndexMaker class.
Once these indices are created (which you can do by settingforce_download=True), you can view them via theindicesproperty:frompyedgarimportEDGARIndexall_indices=EDGARIndex(force_download=False)print(all_indices.indices)# Output:# {'form_all.tab': '/data/storage/edgar/indices/form_all.tab',#  'form_10-Q.tab': '/data/storage/edgar/indices/form_10-Q.tab',#  'form_13s.tab': '/data/storage/edgar/indices/form_13s.tab',#  'form_DEF14A.tab': '/data/storage/edgar/indices/form_DEF14A.tab',#  'form_8-K.tab': '/data/storage/edgar/indices/form_8-K.tab',#  'form_20-F.tab': '/data/storage/edgar/indices/form_20-F.tab',#  'form_10-K.tab': '/data/storage/edgar/indices/form_10-K.tab'}These indices are accessible as a pandas dataframe via [] or theget_indexmethod, where the index is selected via the key above (with or without the form_ or .tab).form_10k=all_indices['10-K']print(form_10k.head(1))# Output:#       cik                      name  form    filedate             accession#    0   20  K TRON INTERNATIONAL INC  10-K  1996-03-28  0000893220-96-000500To get a type of form that isn't automatically extracted, you can use form_all:df_s1=EDGARIndex().get_index('all').query("form.str.startswith('S-1')")print(df_s1.head(1))# Output:#        cik        name form    filedate             accession# 5600  1961  WORLDS INC  S-1  2014-02-04  0001264931-14-000033All indices are loaded and saved by pandas, so pandas is a requirement for using this functionality.ConfigConfig files namedpyedgar.conf,.pyedgar,pyedgar.iniare searched for at (in order):os.environ.get("PYEDGAR_CONF", '.')<-- PYEDGAR_CONF environmental variable./~/.config/pyedgar~/AppData/Local/pyedgar~/AppData/Roaming/pyedgar~/Library/Preferences/pyedgar~/.config/~/~/Documents/os.path.abspath(os.path.dirname(__file__))<-- directory of the package. Default package ships with this existing.See theexample config filefor commented config settings.Running multiple configs is quite easy, by settingos.environmanually:importos# os.environ['PYEDGAR_CONF'] = os.path.expanduser('~/Dropbox/config/pyedgar/hades.local.pyedgar.conf')os.environ['PYEDGAR_CONF']=os.path.expanduser('~/Dropbox/config/pyedgar/hades.desb.pyedgar.conf')frompyedgarimportconfigprint(config.CONFIG_FILE)# Output:#     WARNING:pyedgar.config:Loaded config file from '[~]/Dropbox/config/pyedgar/hades.desb.pyedgar.conf'.#     ALERT!!!! FILING_PATH_FORMAT is '{accession[11:13]}/{accession}.nc'.#     [~]/Dropbox/config/pyedgar/hades.desb.pyedgar.confdownloaderThere is a convenience downloader script, for downloading filing feed files and indexes.To see the status of current cached downloads (shows the latest downloaded files) and to see the config setup:$python-mpyedgar.downloader--status--configTo download and extract index files:$python-mpyedgar.downloader-i--loginfoAnd to download and extract the last 30 days of filings:$python-mpyedgar.downloader-dTo download and extract filings since the beginning:$python-mpyedgar.downloader-d--start-date1995-01-01InstallPip installable:pipinstallpyedgarOr pip installable from github:pipinstallgit+https://github.com/gaulinmp/pyedgar#egg=pyedgaror by checking out from github and installing in editable mode:gitclonehttps://github.com/gaulinmp/pyedgarcdpyedgar
pipinstall-e./Requirementsw3m for converting HTML to plaintext (tested on Linux).
A fallback method might one day be added.Tested only on Python >3.4HTML parsing tested only on Linux.
Other HTML->text conversion methodologies were tried (html2text, BeautifulSoup, lxml) but w3m was fastest even with the subprocess calling.
Converting multiple HTML files could probably be optimized with one instance of w3m instead of spawning a subprocess for each call.
But that's for future Mac to work on.