There's really no case where we shouldn't follow a pointer during decoding

Except when the type itself is a pointer.