Don't free vectors in their dtor

Merge/rebase error from DST. Thanks to @eddyb for finding.

Closes #16826 (I hope)

r?
the len and dataptr could/should be inside the if