Skip byebug on all non-MRI rubies, fix tests

The changes in #18149 added tests for the app generator, but only fixed
it for the plugin generator (I should have let CI finish though I think
it would have failed as an allowed failure).