Use polymorphic variants for impl_type

This improves type-safety, allowing us to remove some asserts from the code.