Fixed up a mistake in ref-removing

In transferring over from the filter version, an error was
introduced. This fixes that error.