Make sure we clear the contents of an SV before we treat it as a reference

Otherwise under certain circumstances it leads to a segfault.