build-matrix: Speed up with unary notation

4.6s -> 3.9s.

Signed-off-by: Timothy Gu <timothygu99@gmail.com>