remove references when clearing cache

I didn't realize the references were keeping track of records as well.  This in turn leads to stale data in hasMany collections when they attempt to reload ids which were cleared out etc.