fix similarity rotate + scale

thanks Topochicho

see https://github.com/jcupitt/libvips/issues/154