Title: What is a quick way to count the number of pairs in a list where a XOR b is greater than a AND b?
Tags: python,algorithm,time-complexity

Question: I have an array of numbers, I want to count all possible combination of pairs for which the xor operation for that pair is greater than and operation.
Example:
4,3,5,2

possible pairs are:
(4,3) -> xor=7, and = 0
(4,5) -> xor=1, and = 4
(4,2) -> xor=6, and = 0
(3,5) -> xor=6, and = 1
(3,2) -> xor=1, and = 2
(5,2) -> xor=7, and = 0

Valid pairs for which xor > and are (4,3), (4,2), (3,5), (5,2) so result is 4.

This is my program:
def solve(array):
    n = len(array)
    ans = 0
    for i in range(0, n):
        p1 = array[i]
        for j in range(i, n):
            p2 = array[j]
            if p1 ^ p2 > p1 & p2:
                ans +=1
    return ans

Time complexity is O(n^2) , but my array size is 1 to 10^5 and each element in array is 1 to 2^30. So how can I reduce time complexity of this program.

Answer: Say a and b are integers. Then a^b > a&b iff a and b have different highest set bits.
Solution: use a counting map where the keys are the highest set bits. Populate this in linear time.
Then, process the keys. Say there are n total integers, and some key has r integers (with the same highest set bit). Then that key adds r * (n-r) to the count of pairs where xor > and. That is, each of r integers can be paired with each of (n-r) with a different highest set bit.
This double-counts everything, so divide by 2 at the end.

Example:
Say we have 8 integers, 3 of which have the third bit as their highest set bit, 3 the fourth, and 2 the fifth.
So per my algorithm, we have three buckets of sizes 3, 3, and 2, and a solution of [3*(8-3) + 3*(8-3) + 2*(8-2)] / 2 = 42 / 2 = 21.
Here's a more detailed explanation of the approach:
Any two integers within the same bucket have a higher value under the AND operation than the XOR operation because the AND operation preserves the max set bit, and the XOR operation turns it to 0.
Now take two integers in separate buckets. One of them has a higher max set bit than the other. That means that the max set bit between the two numbers appears in one but not the other, so becomes a 0 under the AND operation, but is preserved under the XOR operation, thus XOR results in a higher max set bit.
The number of pairs where XOR yields a higher result than AND is exactly the number of pairs which are not both in the same bucket.
Say we have n integers total, and r in some bucket. Each of the r integers in that bucket can be paired with any of the (n-r) integers in the other buckets and not with any of the r-1 integers in the same bucket, contributing r * (n-r) to the count of pairs where XOR yields a higher integer than AND.
However, this counts every pair that contributes to our count exactly twice. E.g., when we're comparing 1001 and 110, our analysis of both the bucket with the 4th and 3rd highest set bit being 1 will be incremented by 1 for this pair. Thus at the end we have to divide by 2.

Further example:
Here are all integers with the third bit as their highest set bit: 4, 5, 6, 7, or 100, 101, 110, and 111 in binary.
Here are all with the second bit as their highest set bit: 2, 3 or 10, 11 in binary.
Take any pair with the same highest set bit, arbitrarily I'll choose 6 and 7. AND(110, 111) = 110 = 6. XOR(110, 111) = 001. So the AND operation produces a higher result than XOR. In every case, XOR will convert the highest set bit from 1 to 0, and AND will keep it at 1, so in every case AND will result in a higher result than XOR.
Taking pairs from separate bucket, whichever bit is the highest set bit among the pair is only set in one of the two integers (because this is what we're bucketing by), so under AND that bit becomes 0, and under XOR it remains 1. Since the XOR operation leaves the output with a bigger highest-set-bit than AND, the resulting integer is higher under XOR.