javaunohelper: enhancements in decodeUTF8

Change-Id: I452dbd54251eea683d307c346dc4949a42eedc30
Reviewed-on: https://gerrit.libreoffice.org/10862
Reviewed-by: David Tardon <dtardon@redhat.com>
Tested-by: David Tardon <dtardon@redhat.com>