Fixes #2442 - Write the answers file _before_ running puppet
(cherry picked from commit c9d40d485254c72e02a59bdd36ac747f83c239da)