Remove backend VT caching entirely

Wasn't paying its rent in complexity.