Make sure the have_unknown state is correct