Remove a hacky buffer extraction from metadata
auto merge of #11295 : sfackler/rust/de-jank-encoder, r=brson
Disowned the Visitor.