serialize: use Result

All of Decoder and Encoder's methods now return a Result.

Encodable.encode() and Decodable.decode() return a Result as well.

fixes #12292
auto merge of #13107 : seanmonstar/rust/encoder-errors, r=erickt

All of Decoder and Encoder's methods now return a Result.

Encodable.encode() and Decodable.decode() return a Result as well.

fixes #12292