Cleanup: remove deprecated fields the proper way

Also move converters to their own file to make it easy to move them into v1beta1 later.