serialize: use Result

All of Decoder and Encoder's methods now return a Result.

Encodable.encode() and Decodable.decode() return a Result as well.

fixes #12292