DSM: more robust detection of resolution

This allows switching back and forth between 10 and 11 bit
code which prevents getting locked up in one or the other.
It also avoids a situation that can cause an infinite
recursion.