Use weakref to maintain pointer to object map