Fix ACL handling for never and empty cases