Make the first color the outermost scope, the 2nd color the next scope,
etc

The original behavior is a bit chaotic due to an interaction with the
max trimming function