Revert "Add same logic to only_with_rvm"

This reverts commit 99bbfbdd3db35c21a7e2940a01dc43f5710983c4.