Remove executables to avoid bundler bug

https://github.com/bundler/bundler/issues/2078