Side note: Efficiency-wise, we might want to track whether this is an inclusion or exclusion mask, instead of calling logical_not eagerly.