ENH: use smallest integer for tri ranges

reduces buffer copy and comparison overhead for boolean outer product