Fix unpacking agent relationships

The switch to using only remote IPs broke a relationship unpacker. This
fixes it.