Remove only refs to s4 classes that don't have backrefs. Fixes #168