Compositor and viewport: misc bug-fixes

The classic GDI+ problem of partially transparent image edges strikes
again!