quick-fix #168: avoid using stale cached zipped egg dist info in easy_install

When installing a zipped egg, into a Python environment with a same named zipped
egg already installed, the installation can fail with a zipimport.ZipImportError
complaining about a 'bad local header' encountered in the new zip archive.

This can occur if setuptools loads the original egg for some reason and the two
zip archives have different content. Then if Python attempts to read a file from
the new archive, it will expect it in a location pointed to by the original
archive's directory. This will report an error if zipimport does not encounter
the expected local file start header in the given location.

The mismatch between the two archives can be reproduced by installing the same
setuptools version (prior to this commit and after commit
f40b810acc5f6494735c912a625d647dc2a3c582 that first introduced the requires.txt
metadata information file into the setuptools project) twice from its sources -
which can randomly fail due to the scenario described above. That will package
the zipped egg archive twice, with each of the archives containing slightly
different Python modules. In case this causes any of the compressed modules to
have different size (easy_install.pyc is often the culprit here), then
attempting to read any later file in the zip archive will fail (requires.txt
package metadata file is often the culprit here). A similar scenario can be
reproduced more consistently by manually modifying the setuptools
easy_install.py source file before building the new egg, e.g. by adding some
additional empty lines to its start.

The underlying reason for this problem is setuptools using zipimporter instances
with cached zip archive content directory information from the older zip
archive, even after the old archive has been replaced.

This patch cleans up only one such old zipimporter instance - one referenced via
easy_install command's local_index attribute. That is the one that has been
causing all the currently reported/reproduced installation failures.

A clean solution needs to make certain there are no more zipimporter instances
with stale archive content directory caches left behind after replacing a zipped
egg archive with another. There are currently at least the following known
potential sources for such stale zipimporter instances (all holding references
to Distribution instances that can then hold a reference to a zipimporter
related to their zipped egg archive):
  easy_install command attributes:
      local_index (Environment with a list of Distributions)
      package_index (PackageIndex with a list of Distributions)
      pth_file (PthDistributions with a list of Distributions)
  global pkg_resources.working_set object (holds a list of Distributions)
  imported module's __loader__ attribute (zipimporter instance)
  zipimport._zip_directory_cache
  sys.path_importer_cache

Further debugging & development note: A complete list of all the currently
active stale zipimporter instances can be read using CPython's gc module and its
object reference introspection functionality (gc.get_objects() &
gc.get_referrers()) from inside the uncache_zipdir() method in the setuptools
easy_install.py module. That is the method called just after the old arhive has
been replaced by the new one and all the stale zipimporter instances were
supposed to have been released.
pkg_resources.Environment.__getitem__() code cleanup

Better commented the code, especially the result caching & lazy distribution
list sorting strategies implemented there. Added several TODO comments
indicating things to look into there in the future.

General code cleanup:
 - stopped reusing the project_name variable for different purposes
 - added an assertion to detect corrupt distribution list caches
clean up easy_install.uncache_zipdir() code & comments
make easy_install.uncache_zipdir() remove more stale zipimporter instances

Since paths are case-insensitive on Windows, zipped egg modules may be loaded
using different but equivalent paths. Importing each such different path causes
a new zipimporter to be instantiated. Removing cached zipimporter instances must
then not forget about removing those created for differently spelled paths to
the same replaced egg.

Other missed zipimporter instances are those used to access zipped eggs stored
inside zipped eggs. When clearing zipimporter instances got a given path, we
need to clear all the instances related to any of its subpaths as well.
Monkey-patch the write_pkg_info method on Python 3.1 DistributionMetadata. Fixes #197
Backed out changeset: b0a2fcc5275a Ref #193
Add test capturing failure when find_packages no longer follows symlinks. Ref #195
Restore traversal of symbolic links in find_packages. Fixes #195.
Update changelog
Bumped to 3.5 in preparation for next release.
Added tag 3.5 for changeset 98f29d521c3a
Bumped to 3.6 in preparation for next release.
Give comma-separated values a little room to breathe (use common style convention).
Restore install._install with a comment to capture its requirement on earlier NumPy versions. Fixes #199.
Bumped to 3.5.1 in preparation for next release.
Added tag 3.5.1 for changeset 254d8c625f46
Bumped to 3.5.2 in preparation for next release.
Sort _distmap entries on insert. Obviates need for a cache.
Correct sort order by using list.sort with support for 'key' and 'reverse'. This method is less efficient, but allows the tests to pass.
Remove _sort_dists (unused)