Remove redundant bitwise operations

These make no sense as we're checking with <= 2 anyway.