resolved fdo#87237 propagate error values through matrix comparisons

Apparently introduced with 8e8b43a03e77dd251876c1de0ac06eeeb09192cd the
comparison results were stored as boolean values, effectively discarding
any infinite double values and error values encoded as NaN values.

Change-Id: I1fb6f46894a0bee02a37e28b7e6cc84f8c051f28
(cherry picked from commit 3c405ff82fcc9f8f044833420485c54658064636)