MacRuby/MacRuby — pull request #187 by eitoball (2013-02-12 01:13:32+00:00)
Title: Change encoding of OpenSSL::HMAC.digest result to binary

I think that this should fix MacRuby/MacRuby#49 .

With version 0.12, simple script using net-ssh like following fails with an error
> /snip/gems/net-ssh-2.6.5/lib/net/ssh/transport/packet_stream.rb:146:in `enqueue_packet': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)

```ruby
require 'rubygems'
require 'net/ssh'
Net::SSH.start('hostname', 'username', :password => 'password') do |ssh|
end
```

The `packet_stream.rb:146` is like
```ruby
message = encrypted_data + mac
```
The encoding of `encrypted_data` is `ASCII-8BIT`, and encoding of `mac` is `UTF-8`. `mac` is generated at 3 lines above.
```ruby
mac = client.hmac.digest([client.sequence_number, unencrypted_data].pack("NA*"))
```
`client` is an instance of `Net::SSH::Transport::HMAC::*` (depends algorithm). All those classes are inherited from `Net::SSH::Transport::HMAC::Abstract`, and `Net::SSH::Transport::HMAC::Abstract#digest` simply calls `OpenSSL::HMAC.digest` like:
```ruby
def digest(data)
OpenSSL::HMAC.digest(digest_class.new, key, data)[0,mac_length]
end
```
Thus, I change the encoding of `OpenSSL::HMAC.digest` return value. Indeed, MRI returns `BINARY`.

```
$ macruby -v -ropenssl -e"p OpenSSL::HMAC.digest('sha1', '', '').encoding"
MacRuby 0.12 (ruby 1.9.2) [universal-darwin10.0, x86_64]
#
$ ruby -v -ropenssl -e"p OpenSSL::HMAC.digest('sha1', '', '').encoding"
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin11.4.0]
#
```