sangster/gtfs_reader — pull request #5 by chesterbr (2014-10-03 21:46:57+00:00)
Title: Support for UTF-8 in GTFS files

When called from [gtfs_engine](https://github.com/sangster/gtfs_engine) (within a Rails 4.1.6 / Ruby 2.1.2 app), I noticed some GTFS downloaded fine, but some (such as, [San Diego MTS](http://www.sdmts.com/google_transit_files/google_transit.zip)) threw the exception printed below.

The affected feed files contained non-ASCII characters (which is in line with the [GTFS File Requirements](https://developers.google.com/transit/gtfs/reference#FileRequirements): "Files should be encoded in UTF-8 to support all Unicode characters."). The exception happened when unzipping because, by default, internal and external encodings won't match:

```ruby
$ irb
2.1.2 :001 > Encoding.default_external
=> #
2.1.2 :002 > Encoding.default_internal
=> nil
```
and a conversion will be attempted upon writing to the tempfile. The fix was to use binary mode (just as it was done to download the zip).

For some reason, (possibly some file having explicit or implicit encoding set), it doesn't happen when running `rspec`, so I had to force that situation on the spec to ensure it fails like `rake` does without the fix.

Exception:

```
$ bundle exec rake 'gtfs_engine:update'
2014-10-03 00:01:55 [INFO]: Updating mts
2014-10-03 00:02:15 [ERROR]: "\xEF" from ASCII-8BIT to UTF-8
rake aborted!
Encoding::UndefinedConversionError: "\xEF" from ASCII-8BIT to UTF-8
/Users/chester/.rvm/gems/ruby-2.1.2/gems/gtfs-reader-0.2.11/lib/gtfs_reader/source_updater.rb:90:in `block in extract_to_tempfiles'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:42:in `call'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:42:in `block in each'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:41:in `each'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:41:in `each'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/rubyzip-1.1.6/lib/zip/central_directory.rb:182:in `each'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/gtfs-reader-0.2.11/lib/gtfs_reader/source_updater.rb:88:in `extract_to_tempfiles'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/gtfs-reader-0.2.11/lib/gtfs_reader/source_updater.rb:36:in `download_source'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/gtfs-reader-0.2.11/lib/gtfs_reader/core.rb:51:in `block in update_verbosely'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/gtfs-reader-0.2.11/lib/gtfs_reader/core.rb:48:in `instance_exec'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/gtfs-reader-0.2.11/lib/gtfs_reader/core.rb:48:in `update_verbosely'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/gtfs-reader-0.2.11/lib/gtfs_reader/core.rb:34:in `update'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/gtfs-reader-0.2.11/lib/gtfs_reader/core.rb:29:in `block in update_all!'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/gtfs-reader-0.2.11/lib/gtfs_reader/config/sources.rb:11:in `each'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/gtfs-reader-0.2.11/lib/gtfs_reader/config/sources.rb:11:in `each'
/Users/chester/.rvm/gems/ruby-2.1.2/gems/gtfs-reader-0.2.11/lib/gtfs_reader/core.rb:29:in `update_all!'
/Users/chester/code/gtfs_engine/lib/tasks/gtfs_engine_tasks.rake:28:in `block (2 levels) in '
/Users/chester/.rvm/gems/ruby-2.1.2/bin/ruby_executable_hooks:15:in `eval'
/Users/chester/.rvm/gems/ruby-2.1.2/bin/ruby_executable_hooks:15:in `

'
Tasks: TOP => gtfs_engine:update
(See full trace by running task with --trace)
```