Use weak references in WeakHashMaps to ensure memory can be properly GC'd