Do magic dance to cherry-pick core_ext reliably

See https://github.com/rails/rails/issues/14664#issuecomment-40183326