Careful removal of S4 class references

This fixes #157 and fixes #168.  It will need some tests before it gets merged.