tenderlove/psych — Issue #39: Crashing on Non-UTF8 Character Encoding
Author: tadman
Created: 2011-11-17 16:59:00+00:00
State: open

I found a situation where the engine will crash hard on a specific kind of input. The result looks something like this:

ruby(48546,0x7fff7003cca0) malloc: *** error for object 0x101929e50: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

Since Rails uses Psych as the default serializer now, any data with irregular characters could trigger this kind of problem. Most parts of Ruby will throw an "invalid UTF-8" exception instead of exploding like this.

I've created a minimal test-case that blows up on Ruby 1.9.2-p290 on OS X (64-bit):

#!/usr/bin/env ruby
# encoding: BINARY

# MacRoman encoded text "naïve"
naive = "na\x95ve"

require 'psych'

Psych.dump([ 'test', naive ])

Using `rdebug` I was able to extract a stack trace showing how this trip-up occurred:

/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:21:in `scalar'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:21:in `visit_Psych_Nodes_Scalar'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/visitor.rb:6:in `accept'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:26:in `block in visit_Psych_Nodes_Sequence'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:26:in `each'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:26:in `visit_Psych_Nodes_Sequence'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/visitor.rb:8:in `accept'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:16:in `block in visit_Psych_Nodes_Document'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:16:in `each'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:16:in `visit_Psych_Nodes_Document'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/visitor.rb:10:in `accept'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:10:in `block in visit_Psych_Nodes_Stream'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:10:in `each'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/emitter.rb:10:in `visit_Psych_Nodes_Stream'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/visitors/visitor.rb:11:in `accept'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych/nodes/node.rb:36:in `to_yaml'
/opt/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/psych.rb:166:in `dump'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/base.rb:1720:in `block in arel_attributes_values'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/base.rb:1713:in `each'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/base.rb:1713:in `arel_attributes_values'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/persistence.rb:265:in `create'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/timestamp.rb:47:in `create'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/callbacks.rb:277:in `block in create'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activesupport-3.0.7/lib/active_support/callbacks.rb:414:in `_run_create_callbacks'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/callbacks.rb:277:in `create'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/persistence.rb:246:in `create_or_update'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/callbacks.rb:273:in `block in create_or_update'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activesupport-3.0.7/lib/active_support/callbacks.rb:419:in `_run_save_callbacks'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/callbacks.rb:273:in `create_or_update'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/persistence.rb:39:in `save'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/validations.rb:43:in `save'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/attribute_methods/dirty.rb:21:in `save'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/transactions.rb:240:in `block (2 levels) in save'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/transactions.rb:292:in `block in with_transaction_returning_status'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/connection_adapters/abstract/database_statements.rb:139:in `transaction'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/transactions.rb:207:in `transaction'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/transactions.rb:290:in `with_transaction_returning_status'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/transactions.rb:240:in `block in save'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/transactions.rb:251:in `rollback_active_record_state!'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/transactions.rb:239:in `save'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/associations/association_collection.rb:273:in `block in create'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/associations/association_collection.rb:503:in `block in create_record'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/associations/association_collection.rb:480:in `add_record_to_target_with_callbacks'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/associations/association_collection.rb:503:in `create_record'
/opt/local/rvm/gems/ruby-1.9.2-p290/gems/activerecord-3.0.7/lib/active_record/associations/association_collection.rb:271:in `create'