Repository: pypa/virtualenv

Issue:
virtualenv 20: is the symlink hack really worth it?
I did some timing and it seems like the trouble it causes is not really worth it -- at the very least I'd like an option which copies instead of symlinks

Here's some timing I did to try and guage the differences -- since there's no options I could find I toggled this line to `if False` to get my "copy" data: https://github.com/pypa/virtualenv/blob/8c2985c2946e767bb6f74a7e22f51add17b38987/src/virtualenv/seed/via_app_data/via_app_data.py#L92

### with symlinks

my platform for this example is relatively low powered, a 2015 MBP

```console
$ rm -rf vvv; time virtualenv vvv

real	0m0.128s
user	0m0.107s
sys	0m0.023s
$ rm -rf vvv; time virtualenv vvv

real	0m0.128s
user	0m0.118s
sys	0m0.012s
$ rm -rf vvv; time virtualenv vvv

real	0m0.123s
user	0m0.121s
sys	0m0.004s
$ rm -rf vvv; time virtualenv vvv

real	0m0.119s
user	0m0.117s
sys	0m0.004s
$ rm -rf vvv; time virtualenv vvv

real	0m0.127s
user	0m0.109s
sys	0m0.020s
```

disk usage:

```console
$ du -hs vvv
128K	vvv
```

problems this can cause:

```console
$ # copied to same path on other machine
$ ./vvv/bin/python -c 'import setuptools'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools'
$ ./vvv/bin/pip --help
Traceback (most recent call last):
  File "./vvv/bin/pip", line 6, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip'
```

### with copies

```console
$ rm -rf vvv; time virtualenv vvv

real	0m0.179s
user	0m0.155s
sys	0m0.050s
$ rm -rf vvv; time virtualenv vvv

real	0m0.185s
user	0m0.158s
sys	0m0.050s
$ rm -rf vvv; time virtualenv vvv

real	0m0.183s
user	0m0.160s
sys	0m0.048s
$ rm -rf vvv; time virtualenv vvv

real	0m0.172s
user	0m0.162s
sys	0m0.035s
$ rm -rf vvv; time virtualenv vvv

real	0m0.181s
user	0m0.142s
sys	0m0.065s
```

```console
$ du -hs vvv
7.5M	vvv
```

### trade off

so we're looking at ~60ms of time overhead -- which (imo) isn't that much -- the disk usage is another concern but we're still taking that usage one way or another

### other considerations

hardlinks would be another consideration -- it would alleviate the problems I have with symlinks (caches, using virtualenv as a deployment mechanism, etc.) -- I'd have to do some implementation work to verify that case


Discussion:
I'm experiencing the `ModuleNotFoundError: No module named 'pip'` error mentioned above in https://travis-ci.org/galaxyproject/galaxy/jobs/648435619 , hoping for a solution.
@nsoranzo your issue is separate from the topic of this discussion, please open a new issue for that. @asottile I'll address your point raised after there are no more bugfixes needed at a later time; but note you can use ``--copies`` to get the copy behaviour (for both python files, and the app-data part). That being said if the app-data folder is causing you issues and you don't care about performance you really should be using the pip seeder.
I do care about performance, the copies approach is ~180ms whereas the pip approach is >3s

I don't want copies of the python executable
also `--copies` does not do copies:

```console
$ virtualenv vvv --copies
$ tree vvv
vvv
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── activate.ps1
│   ├── activate_this.py
│   ├── activate.xsh
│   ├── easy_install
│   ├── easy_install3
│   ├── easy_install-3.6
│   ├── pip
│   ├── pip3
│   ├── pip-3.6
│   ├── python
│   ├── python3
│   ├── python3.6
│   ├── wheel
│   ├── wheel3
│   └── wheel-3.6
├── lib
│   └── python3.6
│       └── site-packages
│           ├── easy_install.py -> /home/asottile/.local/share/virtualenv/seed-v1/3.6/image/SymlinkPipInstall/setuptools-45.2.0-py3-none-any/easy_install.py
│           ├── pip -> /home/asottile/.local/share/virtualenv/seed-v1/3.6/image/SymlinkPipInstall/pip-20.0.2-py2.py3-none-any/pip
│           ├── pip-20.0.2.dist-info -> /home/asottile/.local/share/virtualenv/seed-v1/3.6/image/SymlinkPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info
│           ├── pip-20.0.2.dist-info.virtualenv -> /home/asottile/.local/share/virtualenv/seed-v1/3.6/image/SymlinkPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info.virtualenv
│           ├── pkg_resources -> /home/asottile/.local/share/virtualenv/seed-v1/3.6/image/SymlinkPipInstall/setuptools-45.2.0-py3-none-any/pkg_resources
│           ├── setuptools -> /home/asottile/.local/share/virtualenv/seed-v1/3.6/image/SymlinkPipInstall/setuptools-45.2.0-py3-none-any/setuptools
│           ├── setuptools-45.2.0.dist-info -> /home/asottile/.local/share/virtualenv/seed-v1/3.6/image/SymlinkPipInstall/setuptools-45.2.0-py3-none-any/setuptools-45.2.0.dist-info
│           ├── setuptools-45.2.0.dist-info.virtualenv -> /home/asottile/.local/share/virtualenv/seed-v1/3.6/image/SymlinkPipInstall/setuptools-45.2.0-py3-none-any/setuptools-45.2.0.dist-info.virtualenv
│           ├── wheel -> /home/asottile/.local/share/virtualenv/seed-v1/3.6/image/SymlinkPipInstall/wheel-0.34.2-py2.py3-none-any/wheel
│           ├── wheel-0.34.2.dist-info -> /home/asottile/.local/share/virtualenv/seed-v1/3.6/image/SymlinkPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info
│           └── wheel-0.34.2.dist-info.virtualenv -> /home/asottile/.local/share/virtualenv/seed-v1/3.6/image/SymlinkPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info.virtualenv
└── pyvenv.cfg

11 directories, 23 files
```
Yeah, fixed that part of #1575, haven't released yet as first want to get in https://github.com/pypa/virtualenv/pull/1571 with it, that is failing at the moment.
I think the symlink approach is very much worth it; especially on Windows with some anti-virus active. That being said, should it be the default option? Maybe not.
I guess my thought is that 60ms is not worth a sacrifice of correctness for sane defaults (whereas 3s+ is of course unacceptable)

In addition to breaking caching of virtualenvs in CI, I've also found it breaks our deployment system at lyft (which produces venvs at a well known location, then tars them up to deploy them)

I don't really want to continue having to "virtualenv's defaults are broken use --XXX" as I've had to do for `--no-download` for so long (thanks for fixing that by the way! 🙏)
I'll create a PR that adds a separate flag for controlling the app-data copy/symlink behaviour, and make it copy by default on all platforms. With a bit of good progress should be out in the next two hours together with some other fixes.
Just for reference on Windows with some more strict anti-virus (on non-SSD harddisks) the difference is more than 60ms; it's more in the realm of 10 seconds.
Maybe then we could make the default for seeder be pip on non-Windows and the default be the symlink thing on Windows?

Resolving patch:
diff --git a/src/virtualenv/seed/via_app_data/via_app_data.py b/src/virtualenv/seed/via_app_data/via_app_data.py
--- a/src/virtualenv/seed/via_app_data/via_app_data.py
+++ b/src/virtualenv/seed/via_app_data/via_app_data.py
@@ -9,6 +9,7 @@
 import six
 
 from virtualenv.dirs import default_data_dir
+from virtualenv.info import fs_supports_symlink
 from virtualenv.seed.embed.base_embed import BaseEmbed
 from virtualenv.seed.embed.wheels.acquire import get_wheels
 
@@ -21,7 +22,7 @@ def __init__(self, options):
         super(FromAppData, self).__init__(options)
         self.clear = options.clear_app_data
         self.app_data_dir = default_data_dir() / "seed-v1"
-        self.symlinks = getattr(options, "copies", False) is False
+        self.symlinks = options.symlink_app_data
 
     @classmethod
     def add_parser_arguments(cls, parser, interpreter):
@@ -33,6 +34,16 @@ def add_parser_arguments(cls, parser, interpreter):
             help="clear the app data folder of seed images ({})".format((default_data_dir() / "seed-v1").path),
             default=False,
         )
+        can_symlink = fs_supports_symlink()
+        parser.add_argument(
+            "--symlink-app-data",
+            dest="symlink_app_data",
+            action="store_true" if can_symlink else "store_false",
+            help="{} symlink the python packages from the app-data folder (requires seed pip>=19.3)".format(
+                "" if can_symlink else "not supported - "
+            ),
+            default=False,
+        )
 
     def run(self, creator):
         if not self.enabled:
@@ -95,4 +106,6 @@ def installer_class(self, pip_version):
         return CopyPipInstall
 
     def __unicode__(self):
-        return super(FromAppData, self).__unicode__() + " app_data_dir={}".format(self.app_data_dir.path)
+        return super(FromAppData, self).__unicode__() + " app_data_dir={} via={}".format(
+            self.app_data_dir.path, "symlink" if self.symlinks else "copy"
+        )


Test patch:
diff --git a/tests/unit/seed/test_boostrap_link_via_app_data.py b/tests/unit/seed/test_boostrap_link_via_app_data.py
--- a/tests/unit/seed/test_boostrap_link_via_app_data.py
+++ b/tests/unit/seed/test_boostrap_link_via_app_data.py
@@ -7,6 +7,7 @@
 import six
 
 from virtualenv.discovery.py_info import PythonInfo
+from virtualenv.info import fs_supports_symlink
 from virtualenv.run import run_via_cli
 from virtualenv.seed.embed.wheels import BUNDLE_SUPPORT
 from virtualenv.seed.embed.wheels.acquire import BUNDLE_FOLDER
@@ -14,7 +15,8 @@
 
 
 @pytest.mark.slow
-def test_base_bootstrap_link_via_app_data(tmp_path, coverage_env, current_fastest):
+@pytest.mark.parametrize("copies", [True, False] if fs_supports_symlink() else [True])
+def test_base_bootstrap_link_via_app_data(tmp_path, coverage_env, current_fastest, copies):
     current = PythonInfo.current_system()
     bundle_ver = BUNDLE_SUPPORT[current.version_release_str]
     create_cmd = [
@@ -33,6 +35,8 @@ def test_base_bootstrap_link_via_app_data(tmp_path, coverage_env, current_fastes
         current_fastest,
         "-vv",
     ]
+    if not copies:
+        create_cmd.append("--symlink-app-data")
     result = run_via_cli(create_cmd)
     coverage_env()
     assert result