Imho we are only doing scale if m_destRect is different than source. Also scale down needs the same handling since otherwise we make the video worse than needed.

Ie we must pretty much always use the frame buffer, or add another pixel shader that can do linear scaling from two sources in one pass.