don't crash in mixed reference/no reference cases, related fdo#59853

Change-Id: Iecb4841dae5f5df28c697e1ea9998251d9975b5d
Reviewed-on: https://gerrit.libreoffice.org/8039
Reviewed-by: Kohei Yoshida <libreoffice@kohei.us>
Tested-by: Kohei Yoshida <libreoffice@kohei.us>