pypa/virtualenv — pull request #691 by dstufft (2014-12-25 02:47:05+00:00)
Title: Crazy Idea: Rewrite virtualenv

I don't think it's any secret that the current virtualenv code base is pretty terrible. It's not well factored, it's full of hacks (some of which I'm pretty sure no longer apply), there are no tests and I don't think anybody actually fully understands it. On top of that the ``-p`` argument works by essentially shelling out and executing the ``virtualenv.py`` script with the target interpreter which cases a bunch of problems with ``sys.path`` that any attempt to fully resolve seems to break another use case. In addition to all of that since the creation of virtualenv the stdlib has grown it's own ``venv`` module which offers built in isolation.

At first I debated whether or not virtualenv has a place at all in the future(tm) where Python has a venv module built into it. I think that it does. The built in venv module suffers from a common problem with the stdlib modules, while it provides some functionality updates to it require an update to the entire Python interpreter and in a world that often times gets it's Python from LTS releases like CentOS, Ubuntu, Debian, etc they are often times using an older version of Python. Another thing is that even if you have an updated venv that doesn't help you if you want to create a venv of an *older* Python. In contrast the virtualenv module can be updated independently which means we can push out new features, bug fixes and new versions of pip/setuptools quicker.

However currently virtualenv relies on a giant pile of hacks in order to create it's isolation. These hacks can require more hacks depending on Python version, the interpreter it's running under, and if the downstream distributor of Python (e.g. Debian) applies their own patches. In addition the hacks involve writing our own ``site.py`` file and our own ``distutils.py`` which means that the behavior of these files inside of a virtualenv is different than when not inside of a virtualenv and this itself can cause a number of problems.

Due to all of this, I think it's a good idea to re-use the isolation support of the venv module if it's available in the target Python and only use the "legacy" hacks as a fallback. I looked at doing something like this in the current virtualenv however the code is so intertwined and spaghetti that doing that almost requires a massive rewrite anyways, so I've decided to just do a from-scratch rewrite of virtualenv and port over the isolation hacks from the old version. This work represents that.

In this rewrite I have the following goals:

* Re-implement with a clean code base using modern best practices.
* As complete test coverage as possible.
* Use the venv module to handle isolation wherever possible, but handle the activation and installing of projects into the virtual environment ourselves.
* Remove the ``-p`` hack where we re-execute virtualenv with a different interpreter but still support ``-p``.

The actual work itself involves:
* [x] Create a command line interface that is similar to the existing interface.
* [x] Create a generic ``EnvironmentBuilder`` API that any isolation technique can implement.
* [x] Create a venv module style implementation of the ``EnvironmentBuilder`` API.
* [ ] Create a legacy isolation style implementation of the ``EnvironmentBuilder`` API.
* [x] Support overriding the prompt for a virtual environment at creation time.
* [x] Support installing pip/setuptools (and independently enabling/disabling them) into the virtual environment.
* [x] Support searching extra directories for pip/setuptools packages.
* [ ] Support controlling the verbosity of the output using the ``-v``, ``--verbose``, ``-q``, and ``--quiet`` flags.
* [x] Support selecting the best ``EnvironmentBuilder`` implementation that is available for the target Python.
* [ ] Write tests for everything.
* [ ] Ensure that everything works on platforms other than OS X, particularly Windows.

The work for the ``VenvBuilder`` involves:
* [x] Create the actual virtual environment.
* [x] Enabling creating a virtual environment with and without the global ``site-packages`` directory.
* [x] Support clearing an existing virtual environment.
* [ ] Support virtual environments using symlinks, copying, or "pick the best".
* [ ] Support making a virtual environment relocatable.

The work for the ``LegacyBuilder`` involves:
* [ ] Create the actual virtual environment.
* [ ] Enabling creating a virtual environment with and without the global ``site-packages`` directory.
* [ ] Support clearing an existing virtual environment.
* [ ] Support virtual environments using symlinks, copying, or "pick the best".
* [ ] Support making a virtual environment relocatable.

Obviously the biggest danger involved here is that we're going to lose something that somebody somewhere was depending on and things will break. This is especially bad because virtualenv does not have anything resembling a reasonable test suite and it is a giant ball of hacks, layered upon more hacks, with an extra helping of spaghetti. However I believe in the long run this will be a much more sustainable solution, especially since it allows us to delegate the isolation portion to the interpreter when possible. This will be very helpful to alternate interpreters too such as PyPy or Jython since they'll be able to implement and control the isolation on their own.

/cc @alex @arigo for the impact on PyPy.
/cc @jimbaker for impact on Jython.

I'm probably going to complete the work regardless just to see if it's even possible, but what do the @pypa/virtualenv-developers think about this? Is this something we want to do?