Fix checking for field removal

No tests though #yolo