Use SoftReferences instead of WeakReferences to hold the Provider classes. If one provider of the set gets collected by the garbage collector we refetch the provider.