Repository: conda/conda

Issue:
problem creating a prefixed environment on a fhgfs filesystem
<!--
Hi!  Read this; it's important.

This is an issue tracker for conda -- the package manager.  File feature requests
for conda here, as well as bug reports about something conda has messed up.

If your issue is a bug report or feature request for:
  * a specific conda package from Anaconda ('defaults' channel):
      ==> file at https://github.com/ContinuumIO/anaconda-issues
  * a specific conda package from conda-forge:
      ==> file at the corresponding feedstock under https://github.com/conda-forge
  * repo.anaconda.com access and service:
      ==> file at https://github.com/ContinuumIO/anaconda-issues
  * anaconda.org access and service:
      ==> file at https://anaconda.org/contact/report
  * commands under 'conda build':
      ==> file at https://github.com/conda/conda-build
  * commands under 'conda env':
      ==> please file it here!
  * all other conda commands that start with 'conda':
      ==> please file it here!

If you continue on, and especially if you are submitting a bug report,

   **please include all requested information below.**

If a maintainer determines the information is required to understand 
your issue, and if it is not provided, your issue may be
closed automatically.

-->

**I'm submitting a...**
  - [x] bug report
  - [ ] feature request


## Current Behavior
<!-- What actually happens?
     If you want to include console output, please use "Steps to Reproduce" below. -->

Creating a prefixed environment on a fhgfs partition does not work. It generates symbolic links to a temporary folder that doesn't exist. I can install on an ext4 partition without problem. In the example below I used xz, but I have observed this with other packages as well.

### Steps to Reproduce
<!-- If the current behavior is a bug, please provide specific, minimal steps to independently reproduce.
     Include the exact conda commands that reproduce the issue and their output between the ticks below. -->
```
bash Miniconda2-latest-Linux-x86_64.sh
conda create -p /tmp/conda/xz xz
ls -lh /tmp/conda/xz/bin/xz
#-rwxrwxr-x 1 user user 97K May  4 16:42 /tmp/conda/xz/bin/xz
conda create -p /DEEP_fhgfs/tmp/conda/xz xz
#lrwxrwxrwx 1 user user 62 May  4 16:43 /DEEP_fhgfs/tmp/conda/xz/bin/xz -> ../../../../tmp/tmpsl8kMX/23b991ec-39b8-48ac-bc89-79aaaeeef66c
df -Th /tmp /DEEP_fhgfs/tmp
#Filesystem     Type   Size  Used Avail Use% Mounted on
#/dev/md2       ext4   XXX  XXX  XXX   1% /local
#fhgfs_nodev    fhgfs  XXX  XXX  XXX  97% /DEEP_fhgfs

```


## Expected Behavior
<!-- What do you think should happen? -->
The installation should complete similar to how it does on the ext4 partition

## Environment Information
<details open><summary><code>`conda info`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda info' -->

```
     active environment : None
       user config file : /home/karln/.condarc
 populated config files : 
          conda version : 4.4.10
    conda-build version : not installed
         python version : 2.7.14.final.0
       base environment : /home/karln/miniconda2  (writable)
           channel URLs : https://repo.continuum.io/pkgs/main/linux-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
          package cache : /home/karln/miniconda2/pkgs
                          /home/karln/.conda/pkgs
       envs directories : /home/karln/miniconda2/envs
                          /home/karln/.conda/envs
               platform : linux-64
             user-agent : conda/4.4.10 requests/2.18.4 CPython/2.7.14 Linux/3.18.86.1.amd64-smp debian/7 glibc/2.13
                UID:GID : 21748:3370
             netrc file : None
           offline mode : False

```
</p></details>


<details open><summary><code>`conda config --show-sources`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda config --show-sources' -->

```
Nothing returned
```
</p></details>


<details><summary><code>`conda list --show-channel-urls`</code></summary><p>
<!-- between the ticks below, paste the output of 'conda list --show-channel-urls' -->

```
# packages in environment at /home/karln/miniconda2:
#
# Name                    Version                   Build  Channel
asn1crypto                0.24.0                   py27_0    defaults
ca-certificates           2017.08.26           h1d4fec5_0    defaults
certifi                   2018.1.18                py27_0    defaults
cffi                      1.11.4           py27h9745a5d_0    defaults
chardet                   3.0.4            py27hfa10054_1    defaults
conda                     4.4.10                   py27_0    defaults
conda-env                 2.6.0                h36134e3_1    defaults
cryptography              2.1.4            py27h6697b16_0    defaults
enum34                    1.1.6            py27h99a27e9_1    defaults
futures                   3.2.0            py27h7b459c0_0    defaults
idna                      2.6              py27h5722d68_1    defaults
ipaddress                 1.0.19                   py27_0    defaults
libedit                   3.1                  heed3624_0    defaults
libffi                    3.2.1                hd88cf55_4    defaults
libgcc-ng                 7.2.0                h7cc24e2_2    defaults
libstdcxx-ng              7.2.0                h7a57d05_2    defaults
ncurses                   6.0                  h9df7e31_2    defaults
openssl                   1.0.2n               hb7f436b_0    defaults
pip                       9.0.1            py27ha730c48_4    defaults
pycosat                   0.6.3            py27ha4109ae_0    defaults
pycparser                 2.18             py27hefa08c5_1    defaults
pyopenssl                 17.5.0           py27hcee3be0_0    defaults
pysocks                   1.6.7            py27he2db6d2_1    defaults
python                    2.7.14              h1571d57_29    defaults
readline                  7.0                  ha6073c6_4    defaults
requests                  2.18.4           py27hc5b0589_1    defaults
ruamel_yaml               0.15.35          py27h14c3975_1    defaults
setuptools                38.4.0                   py27_0    defaults
six                       1.11.0           py27h5f960f1_1    defaults
sqlite                    3.22.0               h1bed415_0    defaults
tk                        8.6.7                hc745277_3    defaults
urllib3                   1.22             py27ha55213b_0    defaults
wheel                     0.30.0           py27h2bc6bb2_1    defaults
yaml                      0.1.7                had09818_2    defaults
zlib                      1.2.11               ha838bed_2    defaults

```
</p></details>



Discussion:
Can you be more specific on the technical root cause of the problem, and what we need to change in conda code to fix it?
From my perspective, the root cause of the problem is the links created,
but I'm afraid my experience and understanding of conda is lacking. There
might be a reason for it, but that is beyond me. If I'm missing something
obvious, please let me know.

Why does conda create links to a temporary folder that does not exist?

I would understand if the link pointed to the location of the binary, in
this case xz. That way I would be able to use the environment. As it is
now, I get a natural `command not found` with the fhgfs environment.


On Fri, May 4, 2018 at 5:58 PM, Kale Franz <notifications@github.com> wrote:

> Can you be more specific on the technical root cause of the problem, and
> what we need to change in conda code to fix it?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/conda/conda/issues/7253#issuecomment-386646008>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ADsaNRa0lXZe8UHodG1sIDG9EjdSaQQCks5tvHqRgaJpZM4TyyNj>
> .
>

I don’t have a lot of experience with this file system. Actually, I don’t have any. Maybe we can figure all this out together though with some back-and-forth. 

As a starting point, see how far you get after doing

    conda config --set always_copy true
That looks like it will do it. Will try it beginning of next week and get
back to you.

On Fri, May 4, 2018 at 8:00 PM, Kale Franz <notifications@github.com> wrote:

> I don’t have a lot of experience with this file system. Actually, I don’t
> have any. Maybe we can figure all this out together though with some
> back-and-forth.
>
> As a starting point, see how far you get after doing
>
> conda config --set always_copy true
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/conda/conda/issues/7253#issuecomment-386685795>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ADsaNXN_COOC5T1nNKSw1fHwUmH5R1-2ks5tvJdBgaJpZM4TyyNj>
> .
>

I'm afraid this didn't work. I activated the setting. Not sure that it makes a difference, but I also tried to remove my conda installation and start from scratch (.conda and .condarc included)

I tried to follow the process through the trace file:
[xz.conda.trace.txt](https://github.com/conda/conda/files/1979151/xz.conda.trace.txt)
This is how I understand it:

1. find packages in cache
1. prepare transaction
1. verifying transaction: During this step a temporary folder is created and files are correctly copied to this location
1. Executing transaction: Here, most files are soft linked and this seems to be correctly translated into a copy action. Still, a few selected files are targeted to be hard linked, but for some reasons they are soft linked. That they are soft linked probably has something to do with the system. The thing is that they should be copied. Or do I understand the option wrong?

One thing I noticed trying to understand the code:
https://github.com/conda/conda/blob/c54d490419d79d07a618a2c3ddc0729f20560246/conda/core/path_actions.py#L360
was that the super call had one argument less than the init function (the prefix_placeholder is missing). Could this have an impact?

Around line 406 of your trace is

```
TRACE conda.gateways.disk.create:_do_copy(252): copying /home/user/miniconda2/pkgs/xz-5.2.3-h5e939de_4/bin/xz => /DEEP_fhgfs/tmp/tmpUAg7af/4ad6953a-3e9e-488f-a59b-c4a708d60584
TRACE conda.core.path_actions:verify(386): rewriting prefixes in /DEEP_fhgfs/tmp/conda/xz/bin/xz
TRACE conda.gateways.disk.update:update_file_in_place_as_binary(35): in-place update path locked for /DEEP_fhgfs/tmp/tmpUAg7af/4ad6953a-3e9e-488f-a59b-c4a708d60584
```

and then around line 636

```
TRACE conda.core.path_actions:execute(411): linking /DEEP_fhgfs/tmp/tmpUAg7af/4ad6953a-3e9e-488f-a59b-c4a708d60584 => /DEEP_fhgfs/tmp/conda/xz/bin/xz
TRACE conda.gateways.disk.create:create_link(298): hard linking /DEEP_fhgfs/tmp/tmpUAg7af/4ad6953a-3e9e-488f-a59b-c4a708d60584 => /DEEP_fhgfs/tmp/conda/xz/bin/xz
```

What's happening here is that the `xz` executable is modified at install time with prefix rewrites.  This happens in two phases.

During the verification phase of the install process, we detect that the file is going to need prefix rewrites, so we copy it to a temporary location and do the rewrites there.  That's so that, before we enter the transaction, we'll know the exact final state of the file, including the sha256 sum for the file with the prefix modification.

During the second phase, which occurs *during* the transaction, we link the file from the temporary location into the environment.  Here, at least for many filesystems, hard-linking *should* be fine, because what we're hard-linking is the new inode we created in the temporary location.  However if the temporary location is on a different mount, then hard-linking will obviously fail, and we should fall back to copying.  That's exactly what we do here:

https://github.com/conda/conda/blob/c54d490419d79d07a618a2c3ddc0729f20560246/conda/gateways/disk/create.py#L294-L306

(line 306 that github is truncating is `copy(src, dst)`)

The log information shows that there's no hard-link failure happening on your system.  But conda most definitely thinks what it's created is a hard link and not a symbolic link for `/DEEP_fhgfs/tmp/conda/xz/bin/xz`.  Can you poke at `/DEEP_fhgfs/tmp/conda/xz/bin/xz` more and verify that it indeed ends up being a symbolic link once the conda operation is done?  This is definitely a puzzle now.
Thanks for the explanation, I could see how things happened, but I didn't understand the reasoning behind it.

The file looks very much like a symbolic link:
```
file /DEEP_fhgfs/tmp/conda/xz/bin/xz
#/DEEP_fhgfs/tmp/conda/xz/bin/xz: broken symbolic link to `../../../../tmp/tmpUAg7af/4ad6953a-3e9e-488f-a59b-c4a708d60584'
```
It was a long time since the system was configured... there is an anecdote, but I skip over it... the file system has a configuration:
```
sysCreateHardlinksAsSymlinks  = true
```
I couldn't find a description for it, but as I read it, hardlinks are replaced by symlinks and I can confirm that a `ln` without extra flags generates a symbolic link... no error signal

I still don't understand why the always_copy flag doesn't take precedence. Just to discard my earlier suspicions; Is it correct that the create code is called from the PrefixReplaceLinkAction class in path_actions.py? And if so, isn't the functionality of the super call to override the link type with copy? Or is this overridden on L365? (Setting Linktype to copy and then setting it back?)

https://github.com/conda/conda/blob/c54d490419d79d07a618a2c3ddc0729f20560246/conda/core/path_actions.py#L356-L365
Does it matter that "prefix_placeholder" is missing although the arguments are given as positional arguments? (My python is to shaky to judge this without setting up a test case.)

If the linktype change happens, shouldn't the create section copy instead of hardlink?

> I still don't understand why the always_copy flag doesn't take precedence.

Before encountering this issue with you, I would have said the hard-link fall-back-to-copy case was just fine for this particular code path dealing with storing prefix-rewritten files in temporary locations.  Now I agree that `always_copy` should mean "always copy."  The down side is that it's a performance hit for everybody, even though this `sysCreateHardlinksAsSymlinks` situation only applies to a fairly niche group.

>  Is it correct that the create code is called from the PrefixReplaceLinkAction class in path_actions.py? 

Yes.

> And if so, isn't the functionality of the super call to override the link type with copy?

Yes, but that copy applies from to the operation that's going from the package cache to the temporary location.  What's problematic now, and needs to be changed per my first comment in this post, is

https://github.com/conda/conda/blob/c54d490419d79d07a618a2c3ddc0729f20560246/conda/core/path_actions.py#L412

Should be changed to

    create_link(source_path, self.target_full_path, LinkType.copy)

> Does it matter that "prefix_placeholder" is missing although the arguments are given as positional arguments?

The `__init__` method on the parent class does not take `prefix_placeholder`.  That argument is specific to `PrefixReplaceLinkAction`.
> The down side is that it's a performance hit for everybody, even though this `sysCreateHardlinksAsSymlinks` situation only applies to a fairly niche group.

Rather than make the change that'll be the performance hit for everybody, I can probably test if hard links *are* hard links for the filesystem of the target environment.  If they end up being symlinks, then we just switch to `always_copy` again.  I think that's the correct solution.  And in that case the function we make the code change in is

https://github.com/conda/conda/blob/c54d490419d79d07a618a2c3ddc0729f20560246/conda/core/link.py#L49-L59
Actually, that function doesn't need a change at all.  We just need to make the `hardlink_supported()` function aware of this edge case.

https://github.com/conda/conda/blob/c54d490419d79d07a618a2c3ddc0729f20560246/conda/gateways/disk/test.py#L42-L66
Oh, look at that comment...  Maybe we have a logic error in that function?  We don't run our CI on such a filesystem, so it's possible that an error crept in over time.
yes that comment does indeed fit.
In determine_link_type, always_copy is the first option. If that option is set, hardlink_supported should never happen?
Does all copy/linking actions pass through determine_link_type?

I pulled the repository and started to run the tests. I could not place my conda installation on the problematic file system, but the repository is located there. I'll get back when it gotten through
Most of the tests run through.

[test.log.txt](https://github.com/conda/conda/files/1983654/test.log.txt)

but there are four failures. The most obvious one is "test_simple_LinkPathAction_hardlink" which returns a link. Then there are three tests associated with trash that I cannot see the another connection between.

I tried to add an assert hardlink_supported to tests/gateways/disk/test_link::test_hard_link and it works properly. Add least it returns false on my system.

I tried, but failed to modify simple_LinkPathAction_hardlink to use the PrefixReplaceLinkAction, as I couldn't find a test for this function. I stumble on providing a context object.
I made an attempt at a fix. As mentioned, it allows me to create environments on my file system, but I'm unable to judge if it is too naive, inserted at the wrong place, have ugly side effects etc.

It doesn't fix the tests, only the symptomatic described in this issue.

Resolving patch:
diff --git a/conda/core/path_actions.py b/conda/core/path_actions.py
--- a/conda/core/path_actions.py
+++ b/conda/core/path_actions.py
@@ -206,6 +206,7 @@ def make_file_link_action(source_path_data):
                                                package_info.extracted_package_dir,
                                                source_path_data.path,
                                                target_prefix, target_short_path,
+                                               requested_link_type,
                                                placeholder, fmode, source_path_data)
             else:
                 return LinkPathAction(transaction_context, package_info,
@@ -356,11 +357,14 @@ class PrefixReplaceLinkAction(LinkPathAction):
     def __init__(self, transaction_context, package_info,
                  extracted_package_dir, source_short_path,
                  target_prefix, target_short_path,
+                 link_type,
                  prefix_placeholder, file_mode, source_path_data):
+        # This link_type used in execute(). Make sure we always respect LinkType.copy request.
+        link_type = LinkType.copy if link_type == LinkType.copy else LinkType.hardlink
         super(PrefixReplaceLinkAction, self).__init__(transaction_context, package_info,
                                                       extracted_package_dir, source_short_path,
                                                       target_prefix, target_short_path,
-                                                      LinkType.copy, source_path_data)
+                                                      link_type, source_path_data)
         self.prefix_placeholder = prefix_placeholder
         self.file_mode = file_mode
         self.intermediate_path = None
@@ -409,7 +413,7 @@ def execute(self):
             self.verify()
         source_path = self.intermediate_path or self.source_full_path
         log.trace("linking %s => %s", source_path, self.target_full_path)
-        create_link(source_path, self.target_full_path, LinkType.hardlink)
+        create_link(source_path, self.target_full_path, self.link_type)
         self._execute_successful = True
 
 


Test patch: