fix ref check by removing else

we should only try and add refimprove e.t.c if we removed an unref tag