Switch to using a GenericForeignKey

This is so we can insert things in places they shouldn't go.