Repository: ipython/ipython

Issue:
Notebook.ipynb CR+LF turned into CR+CR+LF
Loading a Notebook created with IPython 2.2 under Windows
![before](https://cloud.githubusercontent.com/assets/6342379/4499728/cdf16498-4a86-11e4-9a00-6f54eaa7d1aa.png)
when saved with IPython 2.3 results in
![after](https://cloud.githubusercontent.com/assets/6342379/4499735/e3221402-4a86-11e4-9d42-8fcc59d2cb73.png)

Similar to #3819 and #4122



Discussion:
Does the behavior differ in 2.3 from 2.2?

Under 2.2 the line endings are correct (CR+LF).

ok, thanks.

For if this is still useful: a file before/after save in ipython 2.3 (so the 'before' is saved in 2.2): https://gist.github.com/jorisvandenbossche/faf74a08aa346b9593e1

In notepad++, it indicated that the file saved in 2.3 has Macintosh format EOL, instead of Windows/Dos.

@jorisvandenbossche can you maybe post a zipfile somewhere? when I download that gist, there are no CRs anywhere. I suspect GitHub is doing some cleaning.

Is this better? https://drive.google.com/file/d/0B3G70MlBnCgKaUgwdElEdkctWDQ/view?usp=sharing

Just to double check, @jorisvandenbossche and @xflr6, what Python version are you using?

I am using 2.7.8, but can check with python 3

Aha, I don't have the issue using python 3 (3.4.2)

So to verify, the entire reproduction process is:
1. open notebook with CRLF line endings in 2.3
2. save it

Is that correct?

The 2.2 notebook in your zipfile has just unix-style LF line-endings. the 2.3 notebook has Windows-style CRLF. There are no instances of CRCRLF in either.

I do know how to get it to use LF always, which may fix both cases, but I would like to be able to reproduce the CRCRLF situation to be sure.

Reproduction process was for me:
- in an ipython 2.2 env (just conda create -n ipy22 ipython-notebook=2.2): make and save a notebook (`test_ipynb_file_22.ipynb`)
- activate an 2.3 env ( conda create -n ipy22 ipython-notebook=2.3), run ipython notebook, make a copy of the existing notebook, rerun and save as `test_ipynb_file_23.ipynb`

@jorisvandenbossche can you try #6909?

@minrk that seems to fix it!

But it gives now Unix line endings on python 3

Is there an easy way to check for this on Windows (I used `file` on linux to check it, but that doesn't work on windows). Now I am just looking at the indication that Notepad++ gives:
- ipython 2.2:
  - python 2.7: Dos/Windows
  - python 3.4: Dos/Windows
- ipython 2.3:
  - python 2.7: Macintosh (CR CRLF)
  - python 3.4: Dos/Windows (CRLF)
  - patched:
    - python 2.7: Dos/Windows (CRLF)
    - python 3.4: Unix (LF)

> But it gives now Unix line endings on python 3

Yes, it should give unix line endings under all circumstances, now.

> Is there an easy way to check for this on Windows

``` python
import io
with io.open(path, newline='') as f:
    for line in f:
        print(repr(line))
```

> Yes, it should give unix line endings under all circumstances, now.

But it doesn't for python 2

> But it doesn't for python 2

Yup, apparently the fix isn't quite right. Still tracking that down.

The problematic behaviour seems to be located in `atomic_writing`.
The following produces the same results as in 2.2 on Python 2.7 under Windows (CR+LF):

``` python
from IPython.nbformat.current import read, write
from IPython.utils.io import atomic_writing

nb = read(io.open('spam.ipynb'), u'json')

with atomic_writing('spam_.ipynb', text=False, encoding='utf8') as f:
    write(nb, f, u'json')
```

With text=True as in 2.3, the resulting file has CR+CR+LF line endings.


Resolving patch:
diff --git a/IPython/utils/io.py b/IPython/utils/io.py
--- a/IPython/utils/io.py
+++ b/IPython/utils/io.py
@@ -267,7 +267,7 @@ def atomic_writing(path, text=True, encoding='utf-8', **kwargs):
         path = os.path.join(os.path.dirname(path), os.readlink(path))
 
     dirname, basename = os.path.split(path)
-    handle, tmp_path = tempfile.mkstemp(prefix=basename, dir=dirname, text=text)
+    handle, tmp_path = tempfile.mkstemp(prefix=basename, dir=dirname)
     if text:
         fileobj = io.open(handle, 'w', encoding=encoding, **kwargs)
     else:


Test patch:
diff --git a/IPython/utils/tests/test_io.py b/IPython/utils/tests/test_io.py
--- a/IPython/utils/tests/test_io.py
+++ b/IPython/utils/tests/test_io.py
@@ -175,4 +175,41 @@ class CustomExc(Exception): pass
                 f.write(u'written from symlink')
             
             with stdlib_io.open(f1, 'r') as f:
-                nt.assert_equal(f.read(), u'written from symlink')
\ No newline at end of file
+                nt.assert_equal(f.read(), u'written from symlink')
+
+def test_atomic_writing_newlines():
+    with TemporaryDirectory() as td:
+        path = os.path.join(td, 'testfile')
+        
+        lf = u'a\nb\nc\n'
+        plat = lf.replace(u'\n', os.linesep)
+        crlf = lf.replace(u'\n', u'\r\n')
+        
+        # test default
+        with stdlib_io.open(path, 'w') as f:
+            f.write(lf)
+        with stdlib_io.open(path, 'r', newline='') as f:
+            read = f.read()
+        nt.assert_equal(read, plat)
+        
+        # test newline=LF
+        with stdlib_io.open(path, 'w', newline='\n') as f:
+            f.write(lf)
+        with stdlib_io.open(path, 'r', newline='') as f:
+            read = f.read()
+        nt.assert_equal(read, lf)
+        
+        # test newline=CRLF
+        with atomic_writing(path, newline='\r\n') as f:
+            f.write(lf)
+        with stdlib_io.open(path, 'r', newline='') as f:
+            read = f.read()
+        nt.assert_equal(read, crlf)
+        
+        # test newline=no convert
+        text = u'crlf\r\ncr\rlf\n'
+        with atomic_writing(path, newline='') as f:
+            f.write(text)
+        with stdlib_io.open(path, 'r', newline='') as f:
+            read = f.read()
+        nt.assert_equal(read, text)