Fix ConcurrentReferenceHashMap max constraints

Update calculateShift to respect the maximum upper bound as well as the
minimum value.

Issue: SPR-11720

(cherry picked from commit 16410cad)