mon: OSDMonitor: disallow nonsensical cache-mode transitions

Fixes: 8155

also adds tests to qa/workunits/cephtool/test.sh