Remove Reference<T> hack (plus some other clean-up)

Also remove Serializer<T> and only use TypeSerializer<T> which is now
Serializable.