PassThroughManager won't pickle - use IDs instead

We'd need to convert to use IDs here anyway for when we switch to using celery for searches