Reduce allocation cost in house_matrix_columns.