Check to make sure the host is in the inventory vars cache before clearing it

Fixes #6660