Do not loose already stored references

If a same object have several reference fields, REFERENCE_QUEUE could already contains saved references for another field, make sure we do not loose it...