seamusabshere/remote_table — pull request #14 by dwillis (2014-01-05 21:03:28+00:00)
Title: Added :replace_quotes option to delimited CSV parser

Federal Election Commission data often contains double-quoting within its pipe-delimited text files that causes Ruby's CSV parser to fail. For example, this raises an error:

1.9.3p448 :005 > t = RemoteTable.new "ftp://ftp.fec.gov/FEC/2014/cm14.zip", :filename => "cm.txt", :col_sep => "|", :quote_char => '"'
1.9.3p448 :006 > t.entries
CSV::MalformedCSVError: Illegal quoting in line 110.

This pull request contains an option, :replace_quotes, which defaults to False, that when enabled replaces all double quotes within the target file with single quotes. For the above example, this now parses the file successfully:

1.9.3p448 :008 > cols = [:committee_id, :committee_name, :treasurer, :street_one, :street_two, :city, :state, :zipcode, :designation, :type, :party, :filing_frequency, :category, :connected_organization, :candidate_id]
1.9.3p448 :009 > t = RemoteTable.new "ftp://ftp.fec.gov/FEC/2014/cm14.zip", :filename => "cm.txt", :headers => cols, :col_sep => "|", :replace_quotes => true
1.9.3p448 :010 > t.entries[110]
=> {:committee_id=>"C00008896", :committee_name=>"OPEIU LOCAL 153 'VOTE' (VOICE OF THE ELECTORATE) COMMITTEE", :treasurer=>"RICHARD J LANIGAN", :street_one=>"265 WEST 14TH ST.", :street_two=>"", :city=>"NEW YORK", :state=>"NY", :zipcode=>"10011", :designation=>"U", :type=>"Q", :party=>"", :filing_frequency=>"Q", :category=>"L", :connected_organization=>"EMPLOYEES INT'L UNION; OFFICE & PROF.", :candidate_id=>""}

I've added a test case for this that passes. Thanks for your consideration; RemoteTable is an incredibly useful library!