Repository: spyder-ide/spyder

Issue:
Mitigate Spyder wiping users' files with more robust atomic saves/autosaves
<!--- **PLEASE READ:** When submitting here, please ensure you've completed the following checklist and checked the boxes to confirm. Issue reports without it may be closed. Thanks! --->



## Problem Description


### What steps reproduce the problem?

1. Save file to disk with connection issues (confirmed)
2. Crashes during runfile or during saving immediatly prior (reported 2x)
3. Other problems during saving???



Given we've publicly advertised the autosave feature as the solution to the reports we've been getting of Spyder completely wiping users' files on occasion e.g. @ccordoba12 in #6571:

> I don't understand how Spyder can delete your script, but we're working to provide autosaving functionality for our next major release (Spyder 4).

and particularly now that our own developer @jnsebgosselin is regularly experiencing a such a problem and can reproduce it, we should ensure autosave is reasonably "fail-safer" in the event that saving the file normally fails (silently or otherwise) and wipes or corrupts the file on disk. As currently implemented by @jitseniesen following his hard work in #7660 , the feature is reasonably sophisticated with a nice UI. However, as an initial implementation, it does leave open several significant opportunities for catastrophic data loss in several major scenarios:


*Failure mode:*
Saving may fail silently, or in flushing the data to the storage medium, which would result in the autosave still being deleted and the save being corrupted or wiped (or at least out of date). This would possibly apply to the repeated instances of files being wiped that @jnsebgosselin is experiencing (depending on exactly how the failure occurs).

*Mitigations:*
* Hardening the e.g. FileInfo._write_to_file() function, e.g. with ``<file>.flush()`` then ``os.fsync()``; switching to the more modern ``io.open()`` on Python 2 and making sure files are explicitly closed might help a little too
* Writing to a tempfile when saving, then doing an atomic replace of the original by the appropriate platform-specific method
* ~~If there are significant cases where the OS cannot be trusted to throw an error if something goes wring, we might need a readback check for the tempfile vs. the in memory version (at minimum, that the file reads without error and is the correct length, and perhaps comparing the first line and last lines), though this could be user-optional if its not determined to be critical or it takes a non-trivial amount of time~~
* ~~Temporarily disable autosave and warn the user if a save failure is detected, and notify the user of this; only re-enable once a manual save is successful~~

I've included the full form of the discussion between various parties on this issue below, for reference in one place.


Discussion:
The initial discussion on Gitter with me and @jnsebgosselin :

@jitseniesen I can do some functional testing too, hopefully tomorrow.

I haven't gone over the code in detail nor yet tested this behavior, but given we've advertised this feature as the solution to the reports we've been getting of Spyder completely wiping users' files on occasion (including multiple times to our own @jnsebgosselin ), I wanted to cross-post the relevant parts of the discussion @jnsebgosselin and I had a couple weeks ago on Gitter about how autosave should work, in order to "fail safe" and properly mitigate such issues (one of, if not *the* primary motivation to add it currently), to make sure that's included here:

@jnsebgosselin said:

> I'm wondering about something regarding the auto-save functionality @jitseniesen is implementing.
This is also related to what people are saying in spyder-ide/spyder#2111 and spyder-ide/spyder#4409 . I'm saying this because on my new laptop, I have a small SSD and a large secondary HDD. I have a problem currently with the secondary HDD. From time to time, it simply disconnect. Du to this, it happened a couple of times that Spyder would wipe entirely the files that were opened when trying to save them or running them. Not only the unsaved changes were lost, but the entire content of the files were gone. I'm wondering if some of the people reporting in the aforementioned issues are experiencing that also.

@CAM-Gerlach said:

> I've seen a number of similar reports, that we typically dismissed as isolated incidents, user error or not Spyder related at the time. That's the real value of the autosave functionality, not just for the unsaved changes but having an extra, separate backup copy that gets touched separately to the main one and, to provide the best protection against this issue, is persistent during a session (not being overwritten/removed after a user-triggered save, only on exit if Spyder exits cleanly with everything successfully saved) and not being directly overwritten during each autosave, but rather an (e.g. timestamped) new version is saved, and the old one only deleted until after confirming the new autosave was fully successful, with perhaps a few second delay to ensure everything is flushed to disk).

@jnsebgosselin said:

> OK, that's what I was wondering. So it is more like an "autosave and recovery" feature than just a simple and dumb autosave functionality. If that's the case, we should change the title of the PR to take that into account because I think this is really important.

@CAM-Gerlach said:

> I have no idea if that's exactly how it works. That's my naive suggestion of how it should work to start, to minimize the chance of data loss like you and others have described. The recovery UI is pretty neat though, and is an important part of it. A more advanced version of the feature, not really in scope for that PR, would save multiple versions of the document on a timer or every time it was saved, like most other editors and IDEs (NP++, Sublime, Pycharm, Eclipse, etc) offer.

@jnsebgosselin said:

> Ah ok thanks! I agree with you on this. Anyway, I have a mean to reproduce the total wiping of file content until I send my laptop for repair if needed

@CAM-Gerlach said:

> Yeah, the versioning would be nice. But just making the basic implementation of the autosave itself work like I described (not overwriting/removing the autosave when the file is saved by the user, only on a clean Spyder exist, and writing to a new file each time and only on confirmed success deleting the old, or warning the user that it failed) should (if I understand the failure mode correctly) ensure at least one good copy exists for the great majority of these events other that specific corner cases or user mistakes.
>
> To really be safe against the file blanking problem, you would also want to include a check before deletion of an old autosave file and after writing the new one (or main copy, for when the autosave is deleted on a clean Spyder exit or manually closing a file in the Editor) that reads back the new/main one and refuses to delete the old/autosave and warns the user if either any error occurred during the test read of the new/main, or if the new/main save has >= ~3 bytes of content while the old one has e.g. > ~100 B; unless the user wiped an entire file of more than a few lines and left the empty file for some reason without deleting it entirely this would properly prevent meaningful data loss due to the file blanking problem and many other causes with few to no false positives; otherwise, the autosave feature might not protect against at least certain incarnations of the problem it was supposed to stop in cases where the error occurred silently, only to the main file or after the last autosave, was accidentally overwritten or corrupted, was due to some external hardware or software interaction with the file after save, etc.
@jitseniesen 's response:

> It's an autosave, it's not a backup. It is inspired by the autosave system in Emacs, though I think most autosave systems work similarly (or they save periodically to the actual file opened without user interaction, which I think would be against what our users expect).
>
> Work can be lost in all kind of ways and autosave does not protect against all failure modes. From the reports, the main failure mode (apart from user error) is that Spyder / the OS / the computer crashes and the user loses unsaved files. This should help against it.
> 
> It may not help against hard disk errors. Specifically, if Spyder saves the file and it thinks this is done succesfully, then the autosave file is deleted. So this autosave feature does not help if due to a hardware failure the file did not really save but Spyder thinks it did save. This may be what @jnsebgosselin experienced: Spyder think it saved the file, but in fact the file got wiped. I am not sure how often this happens, it seems pretty rare (I don't remember any report apart from his), but I think the solution would be hardening the save function, perhaps add an fsync or try reading back the file or I don't know.
>
> Keeping the autosave file after the file is saved makes everything more complicated. I'm not sure it's worth the effort, and in any case I think it's best left to another iteration. 
@ccordoba12 's responce:

> > Keeping the autosave file after the file is saved makes everything more complicated
>
> I don't like this either because it pollutes your filesystem with backup files. I also agree with @jitseniesen that we need to leave that for another PR and it needs to be opt-in, not opt-out.
My response:

> It's an autosave, it's not a backup.

Which technically *is* a backup, just a particular type of one (and not a substitute for a traditional one), intended specifically to protect against data loss or corruption due to application/OS hangs, crashes, power failure, and other such instances outside the user's control. From that viewpoint, crashes that wipe the data in user's open file(s) are intimately part of that narrow scope (as opposed to e.g. versioning, persistent backup, or other such protection against hardware failure or user error, etc).

> I am not sure how often this happens, it seems pretty rare (I don't remember any report apart from his)

Aside from the multiple times he experienced it, there were at least two others, #6567 and #6571 , and I'm pretty sure I saw at least one or two more in another thread. In both of those, the error occurred when running a file, before which all files are automatically saved by default (putting them all potentially at risk), and if the fatal crash occurred anytime after the ``FileInfo._write_to_file()`` call returns, then the autosave will fail to protect against this. Furthermore, it also isn't any help in any situation where the crash/bad write occurs when on or around a save where the autosave file had been removed by a previous save and not regenerated (the file was previously saved less than 60 seconds ago, or if the file was only modified in the 60 seconds before save).

Although this is certainly rarer than just loosing unsaved work, the consequences are vastly more severe (total data loss of one or more entire files) and for a non-VCS managed project (or one that is not committed frequently), and the same good user habits (saving frequently) that insure against the former type of data loss not only give no protection for this, but actually make it more likely (more saves, and less likely for autosave to kick in.

> I think the solution would be hardening the save function

Yes, we should certainly implement that as well, but it doesn't cover the window where the file is saved after Spyder has deleted the autosave and not regenerated one. Aside from not deleting the autosave copy after saves and only when Spyder is closed cleanly, a solution would be to have the save() function itself trigger an autosave before it executed if none existed already, do the save with flush and fsync, and then delete it after if no errors were detected (perhaps with a user-optional readback check). 

> I don't like this either because it pollutes your filesystem with backup files.

That's not really true for @jitseniesen 's implementation, since autosave files all go in the ``autosave`` directory under the hidden Spyder config directory, not scattered throughout the filesystem. Furthermore, they are still deleted on a clean exit and the only case where they are retained under our proposal relative to currently is if a file is saved, and then Spyder crashes sometime within 60 seconds (or before the user makes more changes), which is generally when a file wipe would be most likely to occur anyway.

> I think it's best left to another iteration.

> we need to leave that for another PR

Okay, sure, I guess. Perhaps @jnsebgosselin could do it if he has the time, since he's the one who can actually test if it works.

> it needs to be opt-in, not opt-out

Perhaps readback verification could be opt-in, but the changes necessary (more robust saves as described above, and/or not removing the autosave files after save until clean exit) to protect against Spyder completely wiping users' files on save should be at least opt-out, since as a rare but potentially catastrophic problem users are unlikely to know to enable it until its too late, and I haven't seen any reports of this happening in other major IDEs.
@jnsebgosselin 's response:

> I support @CAM-Gerlach view on this. Being able to protect against data corruption (backup & restore) seems like a very important feature to have. Even though it is not a frequent issue, when this happens, the consequences are  absolutely disastrous.

> Don't directly overwrite the old autosave; rather, write the new save to a new file (either a temporary one in the same location that's later moved to the old one's name [..]

This is actually something we should probably do when saving the original file: do not overwrite the file, but save to a temporary file and then rename it over the original file.

Some relevant links:
* http://stupidpythonideas.blogspot.com/2014/07/getting-atomic-writes-right.html
* https://blog.gocept.com/2013/07/15/reliable-file-updates-with-python/
* https://stackoverflow.com/questions/2333872/atomic-writing-to-file-with-python

@jnsebgosselin When your file got truncated, how did you find out? Did Spyder crash, did you get any error message, or were your files silently truncated?
> Another application, thread, VCS or the OS could modify, wipe or corrupt the file (or in turn break Spyder's saving), which Spyder would not not only not trigger the autosave to back up the current version, but assuming it detected the modification, it would automatically replace the contents in memory with the wiped/corrupted/reset/etc. version and possibly (not sure about this—@jitseniesen how does autosave behave when an external change is detected?) even delete any existing autosave and/or think the file is now unmodified.

At the moment, if Spyder detects that the file on disk is changed and there are no changes in Spyder, the file is silently reloaded. If Spyder detects changes on disk and the user also changed the file in Spyder, then the user is given the choice to either reload the file from disk or continue with the changes made in Spyder.

So there are three cases. Autosave as currently implemented behaves as follows:
* If there never were any changes in Spyder, there is no autosave file. Spyder reloads the file from disk and does not create an autosave file.
* If there were changes in Spyder but the user chooses to reload the file from disk, then Spyder deletes the corresponding autosave file and now thinks the file is unmodified, so no new autosave file would be created until the user makes further changes to the file.
* If there were changes in Spyder and the user chooses to keep them, then Spyder will keep the autosave file. When Spyder next saves the file, it will overwrite the changes made externally (and then delete the autosave file).
> This is actually something we should probably do when saving the original file: do not overwrite the file, but save to a temporary file and then rename it over the original file.

Yep, makes sense—I didn't suggest this earlier since I wasn't sure a replace operation could be atomic. It essentially achieves the same thing as triggering an autosave first, but in-place, more reliably and more cleanly (especially once we drop Python 2 support in a year). I'll update the above accordingly.

> Some relevant links:

Thanks, great resources!

> http://stupidpythonideas.blogspot.com

So it seems the basic outline is:

```python
with tempfile.NamedTemporaryFile(dir=os.path.dirname(original_file_path),
                                 delete=False) as fout:
    <the write>
os.replace(fout.name, original_file_path)
```

with the caveat being that os.replace is Python 3 only; we can substitute rename() on non-Windows platforms if replace is not found but we will need to depend on pywin32 on Windows (its already in the defaults channel and installed by default) to get ``MoveFileEx`` to do the same thing on Windows.

> https://blog.gocept.com

If we don't worry about multiple Spyder processes for now, the main relevant piece here is the necessity to do

```python
dirfd = os.open(os.path.dirname(original_file_path), os.O_DIRECTORY)
os.fsync(dirfd)
os.close(dirfd)
```

to flush the replace changes to the directory to disk.

> @jnsebgosselin When your file got truncated, how did you find out? Did Spyder crash, did you get any error message, or were your files silently truncated?

Yes, this is quite important—if we can't trust the OS to raise an error if something goes wrong during the write to the tempfile, then we would presumably want to do at least a basic readback check, that e.g. the file length is the same as the version in memory and not 0, and that e.g. the first and last lines agree (or perhaps something more sophisticated; that's just a naive suggestion).

> So there are three cases. 

Thanks for the detailed explanation! The only potential issue, then, is the first case, where if we didn't persist the autosave file after saves. it would be nice to trigger an autosave of the existing in-memory content before it is replaced, but if we did persist the autosave the behavior is perfect.
Instead of rolling something on our own to make saves more reliable, we could just use the [atomicwrites](https://pypi.org/project/atomicwrites/) package. Its already on ``conda defaults`` and its what ``pytest`` uses.
@CAM-Gerlach, sounds great! Could you create a PR to use `atomicwrites` instead? Thanks!
@ccordoba12 Sure, I can take a crack at it, but it might be a bit since I have a few other PRs in the queue, a big course project due next Monday, the website to finish and some catch up to do on my research.

EDIT: Not sure how I closed it...
@dalthviz, please help us with this one. The idea is to use the `atomicwrites` package to save files from the Editor to disk. We could also use it to save our config files, but in a first attempt let's try to use it only on the Editor.
For the record, my config file has gotten wiped a few times, although that's less critical since there's a backup and I'm not 100% sure it was caused by a failed write. Also, we should probably save autosave files by the same mechanism (if they aren't already being done by the same save function that would use ``atomicwrites``), to prevent them from being corrupted when they are needed.

@ccordoba12 Do you want me to split the other autosave-related stuff off to a separate issue, keeping this one more tightly focused on just the robust saves that using ``atomicwrites`` will mitigate?
> Do you want me to split the other autosave-related stuff off to a separate issue

You mean the backup stuff?
The persistent autosave, etc. stuff, which is (hopefully) mostly unnecessary now to avoid this specific problem if the  ``atomicwrites`` approach fully avoids overwriting and blanking files, and whose functionality could be rolled into a simple timestamped backup system (each autosave being timestamped on creation, not deleted on save, and the last ``n`` files kept) to be much more useful.
Yeah, please open a new issue about that.
:+1: 

@ccordoba12 Thanks; I've opened the issue as #8323 and removed the rest from here.
Thanks!

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -218,6 +218,7 @@ def run(self):
     'numpydoc',
     'spyder-kernels>=1.0',
     'qdarkstyle>=2.6.4',
+    'atomicwrites',
     # Don't require keyring for Python 2 and Linux
     # because it depends on system packages
     'keyring;sys_platform!="linux2"',
diff --git a/spyder/utils/encoding.py b/spyder/utils/encoding.py
--- a/spyder/utils/encoding.py
+++ b/spyder/utils/encoding.py
@@ -20,6 +20,7 @@
 
 # Third-party imports
 from chardet.universaldetector import UniversalDetector
+from atomicwrites import atomic_write
 
 # Local imports
 from spyder.py3compat import (is_string, to_text_string, is_binary_string,
@@ -226,14 +227,21 @@ def to_unicode(string):
 
 def write(text, filename, encoding='utf-8', mode='wb'):
     """
-    Write 'text' to file ('filename') assuming 'encoding'
+    Write 'text' to file ('filename') assuming 'encoding' in an atomic way
     Return (eventually new) encoding
     """
     text, encoding = encode(text, encoding)
-    with open(filename, mode) as textfile:
-        textfile.write(text)
+    if 'a' in mode:
+        with open(filename, mode) as textfile:
+            textfile.write(text)
+    else:
+        with atomic_write(filename,
+                          overwrite=True,
+                          mode=mode) as textfile:
+            textfile.write(text)
     return encoding
 
+
 def writelines(lines, filename, encoding='utf-8', mode='wb'):
     """
     Write 'lines' to file ('filename') assuming 'encoding'


Test patch: