Avoid passing the external ref manager pointer to make it re-entrant.

Change-Id: I67a0df1dcd0635ea82bfc397041a9c43dedbe75d
Handle external reference bits too. This is the last missing piece.

Change-Id: Ib1dd18bb5a8d70c53722ac91e2340d05bbc7798a