[MAT-437] fixup mldivide for triangular matrices

This patch fixes issues with identifying the type of triangular
layout in a square matrix.