Repository: ipython/ipython

Issue:
Temporary files in FilesWriter
I am not sure if this is a bug or not, but it was unexpected behavior for me, so I figured I'd report it.

When using `FilesWriter` with nbconvert, if you are converting multiple notebooks at once that rely on the same set of supplementary files, they do get copied to the build directory, but so do temporary files (one version per extra notebook). For example, if I have the following directory/file structure:

```
$ ls -R
.:
bar.ipynb  files/  foo.ipynb  ipython_nbconvert_config.py

./files:
file1.txt  file2.txt
```

Where `ipython_nbconvert_config.py` is:

``` python
from glob import glob

c = get_config()

c.NbConvertApp.export_format = 'notebook'
c.NbConvertApp.notebooks = glob('*.ipynb')
c.NbConvertApp.writer_class = 'FilesWriter'

c.FilesWriter.build_directory = 'build'
c.FilesWriter.files = glob('files/*')
```

Then, if I run nbconvert, the `build/files` directory contains:

```
$ ls build/files/
file1.txt  file1.txt-temp-77A5  file2.txt  file2.txt-temp-CE50
```

I traced the root of the problem to `IPython.utils.path.link_or_copy`, which says "Because `os.link` does not overwrite files, a unique temporary file will be used if the target already exists, then that file will be moved into place." That's fine, except it seems like the `os.rename(new_dst, dst)` line isn't actually removing `new_dst` (which is the temporary file).



Discussion:
@jhamrick can you reproduce this with master? I can't seem to. If you can, can you post `--debug` output?

Yes, I can still reproduce it:

```
[jhamrick@octavia]:~/project/tmp
$ rm -r build/
[jhamrick@octavia]:~/project/tmp
$ ipython nbconvert --debug --ipython-dir=/tmp
[NbConvertApp] Config changed:
[NbConvertApp] {'NbConvertApp': {'ipython_dir': u'/tmp', 'log_level': 10}}
[NbConvertApp] Using existing profile dir: u'/tmp/profile_default'
[NbConvertApp] Searching path [u'/Users/jhamrick/project/tmp', u'/tmp/profile_default', '/usr/local/etc/ipython', '/etc/ipython'] for config files
[NbConvertApp] Attempting to load config file: ipython_config.py
[NbConvertApp] Attempting to load config file: ipython_nbconvert_config.py
[NbConvertApp] Loaded config file: /Users/jhamrick/project/tmp/ipython_nbconvert_config.py
[NbConvertApp] Config changed:
[NbConvertApp] {'NbConvertApp': {'log_level': 10, 'notebooks': ['bar.ipynb', 'foo.ipynb'], 'ipython_dir': u'/tmp', 'writer_class': 'FilesWriter', 'export_format': 'notebook'}, 'FilesWriter': {'files': ['files/file1.txt', 'files/file2.txt'], 'build_directory': 'build'}}
[NbConvertApp] Converting notebook bar.ipynb to notebook
[NbConvertApp] Support files will be in bar_files/
[NbConvertApp] Applying preprocessor: coalesce_streams
[NbConvertApp] Making directory build/files
[NbConvertApp] Linking files/file1.txt -> build/files/file1.txt
[NbConvertApp] Making directory build/files
[NbConvertApp] Linking files/file2.txt -> build/files/file2.txt
[NbConvertApp] Writing 556 bytes to build/bar.nbconvert.ipynb
[NbConvertApp] Converting notebook foo.ipynb to notebook
[NbConvertApp] Support files will be in foo_files/
[NbConvertApp] Applying preprocessor: coalesce_streams
[NbConvertApp] Making directory build/files
[NbConvertApp] Linking files/file1.txt -> build/files/file1.txt
[NbConvertApp] Making directory build/files
[NbConvertApp] Linking files/file2.txt -> build/files/file2.txt
[NbConvertApp] Writing 556 bytes to build/foo.nbconvert.ipynb
[jhamrick@octavia]:~/project/tmp
$ ls build/files/
file1.txt  file1.txt-temp-A1D3  file2.txt  file2.txt-temp-924D
```

I can send you a tar of the example directory I have if that helps, too.

Looking into this.

See also [Python bug 21876](http://bugs.python.org/issue21876) - os.rename() doesn't remove the source file if it and the destination are hard links to the same file. I believe we're running into this. The comments there mention that "It is arguably a bug in early Unix implementations that got mistakenly codified as part of POSIX."

I think we can work around this by removing the destination file first.

I can reproduce it now, not sure what I did wrong before.


Resolving patch:
diff --git a/IPython/utils/path.py b/IPython/utils/path.py
--- a/IPython/utils/path.py
+++ b/IPython/utils/path.py
@@ -554,6 +554,12 @@ def link_or_copy(src, dst):
 
     link_errno = link(src, dst)
     if link_errno == errno.EEXIST:
+        if os.stat(src).st_ino == os.stat(dst).st_ino:
+            # dst is already a hard link to the correct file, so we don't need
+            # to do anything else. If we try to link and rename the file
+            # anyway, we get duplicate files - see http://bugs.python.org/issue21876
+            return
+
         new_dst = dst + "-temp-%04X" %(random.randint(1, 16**4), )
         try:
             link_or_copy(src, new_dst)


Test patch:
diff --git a/IPython/utils/tests/test_path.py b/IPython/utils/tests/test_path.py
--- a/IPython/utils/tests/test_path.py
+++ b/IPython/utils/tests/test_path.py
@@ -676,3 +676,12 @@ def test_windows(self):
         dst = self.dst("target")
         path.link_or_copy(self.src, dst)
         self.assert_content_equal(self.src, dst)
+
+    def test_link_twice(self):
+        # Linking the same file twice shouldn't leave duplicates around.
+        # See https://github.com/ipython/ipython/issues/6450
+        dst = self.dst('target')
+        path.link_or_copy(self.src, dst)
+        path.link_or_copy(self.src, dst)
+        self.assert_inode_equal(self.src, dst)
+        nt.assert_equal(sorted(os.listdir(self.tempdir.name)), ['src', 'target'])