Remove rustc-serialize as a dependency.

Requiring it brings in another dependency that can break
and needs to be updated, slowing development in downstream
packages which depend on time.