Remove Weakref usage

- While it initially worked, usage has grown problematic as we've done more
  performance optimizations, and as Weakref itself has changed. What I found was
  that Weakref and the GC were cleaning up early, making listeners disappear -
  and thus making tests fail unexpectedly. Since we never want a listener to
  disappear, it makes sense to remove this functionality.
Merge branch 'hotfix/4373' of git://github.com/weierophinney/zf2 into hotfix/4373
Merge branch 'hotfix/4373'

Close #4373