Remove Reference<T> hack (plus some other clean-up)

Remove Serializer<T> and only use TypeSerializer<T> which is now
Serializable.