xrandr: Use rounding when converting from float to fixed point matrix

This prevents unnecessary accuracy loss when the '--transform' option
is used. Also the preparation of a transformation matrix is now moved
to its own separate function.

Signed-off-by: Siarhei Siamashka <siarhei.siamashka@gmail.com>