CID#736170, CID#736171, CID#736172 Out-of-Bounds read/write

Let's be sure that nMaxcolorIndex < 256

Change-Id: I349184ad92c8e7b10a90a32e093972bfaee52467
Reviewed-on: https://gerrit.libreoffice.org/6970
Reviewed-by: Caolán McNamara <caolanm@redhat.com>
Tested-by: Caolán McNamara <caolanm@redhat.com>