Clean up after hotfix to fix xml loading for import

@wedaly: This has the XML factories I was talking about, and unskips the failing test from after the hotfix
@cahrens: This addresses issues around the hotfix last week
@nedbat: This removes `AttrMap`, and just uses `deserialize_field` and `serialize_field` as needed.