improved removing of unwanted gems ignoring failures, fix #2484