Package: xbrl-reports-indexes
XBRL Reports Indexes (xri)xricreates and updates a database for indexes of ESEF filings and SEC XBRL filings and filers (only filings metadata NOT filings contents). SEC XBRL filings index is based on data extracted fromSEC Monthly RSS Feedsfor XBRL filings, ESEF XBRL filings index is based on data extracted fromfilings.xbrl.org.This package is useful for easily collecting and organizing up to date information about XBRL filings available on SEC, including latest filings (updated every 10 minutes during working hours), and the same for ESEF filings. Data can be stored insqlite,postgresormysqldatabases (sqliterecommended).A downloadable sqlite example database is available on google drivehere, contains filings data until 2022-09-14.Installationpipinstallxbrl-reports-indexFrom sourceClonerepo, then build and install locally or install editable:# editablepipinstall-e.# buildpython-mbuild
pipinstalldist/*.whlUsageCommand lineFor options and help:$ xri-db-tasks -h# initialize sqlite database update data (requires internet connection)$xri-db-task<database-name>--initialize-database--update-sec--update-secInitializing the database and loading data may take a long time for SEC information, but subsequent updates should take less than a minute. when usingpostgresormysql, database must be created on the server first and connection information should be provided as follows:$xri-db-taskdatabase_name,product,user,password,host,port,timeout--initialize-database--update-sec--update-secproduct: sqlite (default), postgres or mysql.To search filings:$xri-db-tasks<dbconnectionparams>--search-filingssec--form-type10-q--added-date-from2022-08-10--added-date-to2022-08-10--sec-industry-tree70--limit-result1000--output-filesearch.rss2022-09-1403:48:32,220[rss-db.initialize-db]Verified/Initialized2022-09-1403:48:33,269[rss-db.info]Createdsearch.rssfor186filings.The above command returns SEC filings for forms 10-K and 10-Q filed between 2022-08-10 and 2022-08-12 inclusive, result is saved as an rss file similar to SEC rss feed, this file can be used byarellefor further processing.Python scriptThis package usesSQLAlchemyas anORM. Usage is as follows:fromxbrlreportsindexes.core.index_dbimportXbrlIndexDBfromsqlalchemy.ormimportSession# create a db objectdb=XbrlIndexDB('scratches/xdb_test.db')# create a query objectqry=db.search_filings(filing_system='sec',publication_date_from='2022-08-10',publication_date_to='2022-08-12',form_type='10-q',industry_code_tree='70',limit=1000,)# execute query and print out countwithSession(db.engine)assession:result=qry.with_session(session)print(result.count())Result:>>> 186Or we can go directly to the industry table:# import SEC modelfromxbrlreportsindexes.modelimportSECsession=Session(db.engine)# filter for industry 70industry=session.query(SEC.SecIndustry).filter(SEC.SecIndustry.industry_classification=='SEC',SEC.SecIndustry.industry_code==70,).first()print(industry.industry_description)industry name>>> 'Services'# get filings of that industryfilings=industry.all_industry_tree_filings(session)# filter filingsfilings.filter(SEC.SecFiling.pub_date>='2022-08-10',SEC.SecFiling.pub_date<'2022-08-13',SEC.SecFiling.form_type.like('%10-Q%'),).count()result of count>>> 186All other tools fromSQLAlchemycan be used to query and analyze the data, for example, here is the to 5 forms by count of filings for August 2022:fromsqlalchemyimportfunc,descform_type=SEC.SecFiling.form_typefilings_august_qry=session.query(form_type,func.count(form_type).label('count_forms')).group_by(form_type).order_by(desc('count_forms')).limit(5)forfinfilings_august_qry:print(f[0],'->',f'{f[1]:,}','forms')8-K -> 8,193 forms
10-Q -> 5,343 forms
8-K/A -> 198 forms
485BPOS -> 173 forms
10-K -> 167 formsData updateA cron job can be setup to collect information about new filing from SEC site every 10 minutes during working hours as follows:*/106-22**1-5xri-db-task<dbconnectionparameters>--update-sec# note timezone differencesLicenseApache-2.0