Fix debug assertions properly once and for all

[#72706522]
This cleans up the logic to **ensure** that no greater than 4 are allowed here.  The ```== 4``` looked suspicious to me.