transcode: cleanup channel setup and allow upmixing

Allow channel upmixing as we have channel mixer that is able to do it.

Fixes #7261
trivial: fix upmixing case and handle dualmono

Earlier commit for dualmono (from 2003) seemed to miss some cases
and reordering the if cases as I assume dualmono isn't most common
case in filter.

Ref #7261