WL-3134 Make sure we don't leave templates around.

Originally done in https://github.com/ox-it/wl-site-manage/pull/12/files.
Must have been missed in the port.
Same fix as before (can't cherry pick as files are too different).