quick-fix #168: avoid using stale cached zipped egg dist info in easy_install

When installing a zipped egg, into a Python environment with a same named zipped
egg already installed, the installation can fail with a zipimport.ZipImportError
complaining about a 'bad local header' encountered in the new zip archive.

This can occur if setuptools loads the original egg for some reason and the two
zip archives have different content. Then if Python attempts to read a file from
the new archive, it will expect it in a location pointed to by the original
archive's directory. This will report an error if zipimport does not encounter
the expected local file start header in the given location.

The mismatch between the two archives can be reproduced by installing the same
setuptools version (prior to this commit and after commit
f40b810acc5f6494735c912a625d647dc2a3c582 that first introduced the requires.txt
metadata information file into the setuptools project) twice from its sources -
which can randomly fail due to the scenario described above. That will package
the zipped egg archive twice, with each of the archives containing slightly
different Python modules. In case this causes any of the compressed modules to
have different size (easy_install.pyc is often the culprit here), then
attempting to read any later file in the zip archive will fail (requires.txt
package metadata file is often the culprit here). A similar scenario can be
reproduced more consistently by manually modifying the setuptools
easy_install.py source file before building the new egg, e.g. by adding some
additional empty lines to its start.

The underlying reason for this problem is setuptools using zipimporter instances
with cached zip archive content directory information from the older zip
archive, even after the old archive has been replaced.

This patch cleans up only one such old zipimporter instance - one referenced via
easy_install command's local_index attribute. That is the one that has been
causing all the currently reported/reproduced installation failures.

A clean solution needs to make certain there are no more zipimporter instances
with stale archive content directory caches left behind after replacing a zipped
egg archive with another. There are currently at least the following known
potential sources for such stale zipimporter instances (all holding references
to Distribution instances that can then hold a reference to a zipimporter
related to their zipped egg archive):
  easy_install command attributes:
      local_index (Environment with a list of Distributions)
      package_index (PackageIndex with a list of Distributions)
      pth_file (PthDistributions with a list of Distributions)
  global pkg_resources.working_set object (holds a list of Distributions)
  imported module's __loader__ attribute (zipimporter instance)
  zipimport._zip_directory_cache
  sys.path_importer_cache

Further debugging & development note: A complete list of all the currently
active stale zipimporter instances can be read using CPython's gc module and its
object reference introspection functionality (gc.get_objects() &
gc.get_referrers()) from inside the uncache_zipdir() method in the setuptools
easy_install.py module. That is the method called just after the old arhive has
been replaced by the new one and all the stale zipimporter instances were
supposed to have been released.

--HG--
extra : rebase_source : 041d2819881b8f7e5c4da333a387fc86d4f7b791
pkg_resources.Environment.__getitem__() code cleanup

Better commented the code, especially the result caching & lazy distribution
list sorting strategies implemented there. Added several TODO comments
indicating things to look into there in the future.

General code cleanup:
 - stopped reusing the project_name variable for different purposes
 - added an assertion to detect corrupt distribution list caches

--HG--
extra : source : 86ef8f59ef4e1fc253c155b9ca0856497455372d
clean up easy_install.uncache_zipdir() code & comments

--HG--
extra : rebase_source : 79778a670897cb92c17307f2535fcac6447e16b4
make easy_install.uncache_zipdir() remove more stale zipimporter instances

Since paths are case-insensitive on Windows, zipped egg modules may be loaded
using different but equivalent paths. Importing each such different path causes
a new zipimporter to be instantiated. Removing cached zipimporter instances must
then not forget about removing those created for differently spelled paths to
the same replaced egg.

Other missed zipimporter instances are those used to access zipped eggs stored
inside zipped eggs. When clearing zipimporter instances got a given path, we
need to clear all the instances related to any of its subpaths as well.

--HG--
extra : rebase_source : 86aeadd1e639fbc83d27a0c551fdc2b8a68a6f85
Added tag 3.5.1 for changeset 254d8c625f46
Bumped to 3.5.2 in preparation for next release.
Sort _distmap entries on insert. Obviates need for a cache.
Correct sort order by using list.sort with support for 'key' and 'reverse'. This method is less efficient, but allows the tests to pass.
Remove _sort_dists (unused)
Merge pkg_resources.Environment.__getitem__() code cleanup (pull request #47 and subsequent changes).
Rewrite hashcmp using modern syntax
Reformat for consistency
Reindent long line
Use simpler syntax to get the same info
Reindent long lines in marker impl
Reindent more long lines. Other minor syntax modernization.
Remove unnecessary list comprehension
Now this syntax fits nicely on one line
Fix spacing in __all__
Avoid trailing comments