Ya, this turned out to be better then the reverse of asking all models for their arrays. 
Maybe we need a third entity to keep this tracking information?