Package: fecfile
This is a library for converting campaign finance filings stored in the .fec format into native python objects. It maps the comma/ASCII 28 delimited fields to canonical names based on the version the filing uses and then converts the values that are dates and numbers into the appropriateint,float, ordatetimeobjects.This library is in relatively early testing. I've used it on a couple of projects, but I wouldn't trust it to work on all filings. That said, if you do try using it, I'd love to hear about it!Why?The FEC makes a ton of data available via the "export" links on the main site and thedeveloper API. For cases where those data sources are sufficient, they are almost certainly the easiest/best way to go. A few cases where one might need to be digging into raw filings are:Getting information from individual itemizations including addresses. (The FEC doesn't include street addresses in bulk downloads.)Getting data as soon as it has been filed, instead of waiting for it to be coded. (The FEC generally codes all filings received by 7pm eastern by 7am the next day. However, that means that a filing received at 11:59pm on Monday wouldn't be available until 7am on Wednesday, for example.)Getting more data than the rate-limit on the developer API would allow.Maintaining one's own database with all relevant campaign finance data, perhaps synced with another data source.Raw filings can be found by either downloading thebulk datazip files or from http requests likethis. This library includes helper methods for both.InstallationTo get started, install frompypiby running the following command in your preferred terminal:pipinstallfecfileUsageFor the vast majority of filings, the easiest way to use this library will be to load filings all at once by using thefrom_http(file_number),from_file(file_path), orloads(input)methods.These methods will return a Python dictionary, with keys forheader,filing,itemizations, andtext. Theitemizationsdictionary contains lists of itemizations grouped by type (Schedule A,Schedule B, etc.).Examples:importfecfilefiling1=fecfile.from_file('1229017.fec')print('${:,.2f}'.format(filing1['filing']['col_a_total_receipts']))filing2=fecfile.from_http(1146148)print(filing2['filing']['committee_name'])filing3=fecfile.from_http(1146148)all_contributions=filing3['itemizations']['Schedule B']mid_size_contributions=[itemforiteminall_contributionsif500<=item[contribution_amount]<1000]print(len(mid_size_contributions))withopen('1229017.fec')asfile:parsed=fecfile.loads(file.read())num_disbursements=len(parsed['itemizations']['Schedule B'])print(num_disbursements)url='https://docquery.fec.gov/dcdev/posted/1229017.fec'r=requests.get(url,headers={'User-Agent':'Mozilla/5.0'})parsed=fecfile.loads(r.text)fecfile.print_example(parsed)Note: the docquery.fec.gov urls cause problems with the requests library when a user-agent is not supplied. There may be a cleaner fix to that though.Advanced UsageFEC filings can be arbitrarily large. Loading enormous filings into memory all at once can cause problems (including running out of memory).Thefecfilelibrary exposes theiter_fileanditer_httpmethods to read large filings one line at a time. Both are generator functions that yieldFecItemobjects, which consist ofdataanddata_typeattributes. The data_type attribute can be one of "header", "summary", "itemization", "text", or "F99_text". The data attribute is a dictionary for all data types except for "F99_text", for which it is a string.importfecfileimportimaginary_database# Sometimes we only care about summary data, but want to be able to handle all filings, without# knowing anything about them before we attempt to parse.no_itemizations={'filter_itemizations':[]}foriinrange(1300000,1320000):foriteminfecfile.iter_http(i,options=no_itemizations):ifitem.data_type=='summary':imaginary_database.add_to_db(item.data)# Sometimes we only care about one type of itemization, but from a very large filing.# In this example, we add up all the contributions from Delaware in ActBlue's 2018# post-general filingonly_contributions={'filter_itemizations':['SA']}de_total=0foriteminfecfile.iter_http(1300352,options=only_contributions):ifitem.data_type=='itemization':ifitem.data['contributor_state']=='DE':de_total+=item.data['contribution_amount']print(de_total)# Sometimes we want to maintain a database where different types of itemizations live in their own# tables and have foreign key relationships to a summary record.file_path='/path/to/99840.fec'filing=Noneforiteminfecfile.iter_file(file_path):ifitem.data_type=='summary':filing=imaginary_database.add_filing(file_number=99840,**item.data)ifitem.data_type=='itemization':ifitem.data['form_type'].startswith('SA'):imaginary_database.add_contribution(filing=filing,**item.data)ifitem.data['form_type'].startswith('SB'):imaginary_database.add_disbursement(filing=filing,**item.data)ifitem.data['form_type'].startswith('SC'):imaginary_database.add_loan(filing=filing,**item.data)You can also choose to use theparse_headerandparse_linemethods if you are implementing a different method ofiterating over a filing's content. Before version 0.6, the below example was the only way to usefecfileto parsefilings without loading the entire filing into memory. This approach should no longer be necessary, but is kept toshow how example usage for those methods.importfecfileversion=Nonewithopen('1263179.fec')asfile:forlineinfile:ifversionisNone:header,version=fecfile.parse_header(line)else:parsed=fecfile.parse_line(line,version)save_to_db(parsed)API Referenceloadsloads(input,options={})Deserializeinput(astrinstancecontaining an FEC document) to a Python object.Optionally, pass an array of strings tooptions['filter_itemizations'].If included,loadswill only parse lines that start with any of thestrings in that array. For example, passing{'filter_itemizations': ['SC', 'SD']}tooptions, will only includeSchedule C and Schedule D itemizations. Also, passing{'filter_itemizations': []}tooptionswill result in only the headerand the filing being parsed and returned.Including{'as_strings': True}in theoptionsdictionary will not attempt to convert values that are normally numeric or datetimes to their native python types and will return dictionaries with all values as strings.parse_headerparse_header(hdr)Deserialize astror a list ofstrinstances containingheader information for an FEC document. Returns an Python object, theversionstrused in the document, and the number of lines usedby the header.The third return value from parse_header--the number of lines used by the header--is onlyuseful for early versions of the FEC file format, typically predating 2001. Versions 1 and 2 of the FEC file format allowed headers to be a multiline string beginning and ending with/*.Returning the number of lines in the header allows us to know where the non-header lines begin.parse_lineparse_line(line,version,line_num=None)Deserialize aline(astrinstancecontaining a line from an FEC document) to a Python object.versionis astrinstance for the version of the FEC file formatto be used, and is required.line_numis optional and is used for debugging. If an error orwarning is encountered, whatever is passed in toline_numwill beincluded in the error/warning message. Normally the line number of the input file will be passed in, so that the user is shown the error and the line number in the original file that triggered the error.from_httpfrom_http(file_number,options={})Utility method for retrieving a parsed Python representation of an FECfiling when it is not available as a local file. This method takeseither astrorintas afile_numberand requests the corresponding filing fromthedocquery.fec.govserver. It returns the parsed response.Seeabovefor how documentation on how to use the optionaloptionsargument.from_filefrom_file(file_path,options={})Utility method for getting a parsed Python representation of an FECfiling that exists as a .fec file on a local machine. This method takesastrof the path to the file, and returns the parsed Python object.Seeabovefor how documentation on how to use the optionaloptionsargument.iter_httpiter_http(file_number,options={})Makes an http request for the givenfile_numberand iterates over the response, yieldingFecIteminstances, which consist ofdataanddata_typeattributes. Thedata_typeattribute can be one of "header", "summary", "itemization", "text", or "F99_text". Thedataattribute is a dictionary for all data types except for "F99_text", for which it is a string. This method avoids loading the entire filing into memory, as thefrom_httpmethod does.Seeabovefor how documentation on how to use the optionaloptionsargument.iter_fileiter_file(file_path,options={})Opens a file at the givenfile_pathand iterates over its contents, yieldingFecIteminstances, which consist ofdataanddata_typeattributes. Thedata_typeattribute can be one of "header", "summary", "itemization", "text", or "F99_text". Thedataattribute is a dictionary for all data types except for "F99_text", for which it is a string. This method avoids loading the entire filing into memory, as thefrom_filemethod does.Seeabovefor how documentation on how to use the optionaloptionsargument.print_exampleprint_example(parsed)Utility method for debugging - prints out a representative subset ofthe Python object returned by one of the deserialization methods. Forfilings with itemizations, it only prints the first of each type ofitemization included in the object.Developing locallyAssuming you already have Python3 and the ability to create virtual environments installed, first clone this repository from github and cd into it:gitclonehttps://github.com/esonderegger/fecfile.gitcdfecfileThen create a virtual environment for this project (I use the following commands, but there are several ways to get the desired result):python3-mvenvvenvsourcevenv/bin/activateNext, install the dependencies:pythonsetup.pyinstallFinally, make some changes, and run:pythontests.pyThanksThis project would be impossible without the work done by the kind folks at The New York TimesNewsdev team. In particular, this project relies heavily onfech.Many thanks toDerek Willisfor creating and maintainingfech-sourceswhich serves as the upstream source of truth for parsing libaries in various languages.A huge thank you toChris Zubak-Skees, whose work on thefec-parsejavascript library inspired this project.Many thanks toJacob Fentonfor writing the caching logic and for providing valuable feedback about the overall design of this library.ContributingI would love some help with this, particularly with the mapping from strings toint,float, anddatetimetypes. Pleasecreate an issueormake a pull request. Or reach out privately via email - that works too.To do:Almost too much to list:Handle files from before v6 when they were comma-delimitedcreate adumpsmethod for writing .fec files for round-trip testsadd more types to the types.json fileelegantly handle errorsChangesSee thechangelogfor a list of notable changes introduced in each version of fecfile.