Another option would be to use "weak sets" similar to how we did them here:
https://github.com/mozilla/addon-sdk/blob/master/lib/sdk/lang/weak-set.js