Only set vary header if the filter check is passed