Ignore Allow/DenyQOS in part if QOS not configured

Previous logic would fail an assert