Revert "Fix a memory leak in RerferenceCache"

Reverts neo4j/neo4j#2686