Centralize errors for Swift

Fixes some TODOs in #1382.