Repository: OctoPrint/OctoPrint

Issue:
[Bug] Restore of backup on docker-mounted --basedir volume due to rename logic
<!--
READ THE FOLLOWING FIRST:

If not already done, please read the the Contribution Guidelines that
are linked to the right under "Helpful resources" > "Contributing".

Also read the FAQ: https://faq.octoprint.org

This is a bug and feature tracker, please only use it to report bugs
or request features within OctoPrint (not OctoPi, not any OctoPrint
plugins and not unofficial OctoPrint versions).

Do not seek support here ("I need help with ...", "I have a
question ..."), that belongs on the community forum at 
discourse.octoprint.org, NOT here.

Mark requests with a "[Request]" prefix in the title please. For bug
reports fully fill out the bug reporting template (if you don't know
where to find some information - it's all described in the Contribution
Guidelines linked up there in the big yellow box).

When reporting a bug do NOT delete ANY lines from the template.

Make sure any bug you want to report is still present with the CURRENT
OctoPrint version and that it does not vanish when you start OctoPrint
in safe mode - how to do that is also explained in the Contribution
Guidelines linked up there in the big yellow box.

Thank you!
-->

#### What were you doing?
Trying to restore a backup.

<!-- 
Please be as specific as possible here. The maintainers will need to
reproduce your issue in order to fix it and that is not possible if they
don't know what you did to get it to happen in the first place.

Ideally provide exact steps to follow in order to reproduce your problem:
-->

1. Create a backup using the backup mechanism and download it.
2. Start a second octoprint instance with the exact same versions/codebase of plugins etc.
3. Upload the backup zip file there
4. See error

```

Uploading backup, this can take a while. Please wait...
 
Stelle Sicherung wieder her...
 
Unpacking backup to /tmp/tmpYhY_SP...
Unpacked
Renaming /data/octorpint-data to /data/octorpint-data.bck...
Moving /tmp/tmpYhY_SP/basedir to /data/octorpint-data...
Error while restoring config data
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/OctoPrint-1.3.10-py2.7.egg/octoprint/plugins/backup/__init__.py", line 828, in _restore_backup
    os.rename(basedir_extracted, basedir)
OSError: [Errno 18] Invalid cross-device link
Rolling back old config data
 
Wiederherstellung fehlgeschlagen! Prüfe die obige Ausgabe sowie octoprint.log für mögliche Gründe dafür.
Removing temporary unpacked folder
Removing temporary zip

```
I started octoprint using this script inside a docker container on a raspberry pi 3b+:
```
#!/bin/bash

# ensure that octoprint data will be on 2nd+ level folder to not make restore of backups fail trying to rename the docker mapped volume /data
mkdir -p /data/octorpint-data

# start Octoprint
octoprint --iknowwhatimdoing --port=8080 --basedir /data/octorpint-data
```
As you can see in the script, I circumvented a related issue - the restore mechanism calls `os.rename` on a mounted volume that cannot be renamed resulting in `OSError: [Errno 18] Invalid cross-device link`. This is most probably related to docker, because I mount the /data directory from the host machine, as it is a common way to handle state with docker containers. See https://stackoverflow.com/questions/42392600/oserror-errno-18-invalid-cross-device-link for workaround suggestions. For me it was fine to move the data dir to `/data/octorpint-data` instead of `/data` to work around that issue. Sadly the `os.rename(a,b)` call failed again. Maybe you can just print out a warning in this case (Errno 18) or switch to a copy, then delete behaviour (here https://github.com/foosel/OctoPrint/blob/112a8b9c8bf7f5895f6a29dbe3d98ea4d9e0f400/src/octoprint/plugins/backup/__init__.py#L828)? Or maybe shutil will help working around that (as mentioned in the linked SO-link). 

<!--
If you encountered a problem with specific files of any sorts, make sure
to also include a link to a file with which to reproduce the problem.
-->

#### What did you expect to happen?
Restore of the state.

#### What happened instead?


#### Did the same happen when running OctoPrint in safe mode?

<!-- 
Test if you can reproduce your problem in safe mode. You can find information
on how to enable safe mode in the Contribution Guidelines.

If you can't reproduce in safe mode, this is a bug with one of your
installed third party plugins. Don't open a ticket here!

If you can't test this in safe mode, state why.
-->

#### Version of OctoPrint
1.3.10 (both instances)
<!--
Can be found in the lower left corner of the web interface. ALWAYS INCLUDE.
-->

#### Operating System running OctoPrint
raspian (not OctoPi)
<!--
OctoPi, Linux, Windows, MacOS, something else? With version please.
OctoPi's version can be found in /etc/octopi_version or in the lower left
corner of the web interface.
-->

#### Printer model & used firmware incl. version

<!--
If applicable, always include if unsure.
-->

#### Browser and version of browser, operating system running browser

<!--
If applicable, always include if unsure.
-->

#### Link to octoprint.log
https://gist.github.com/hgross/41fa4272b9549260c5a29a87b8a5ac03

<!--
On gist.github.com or pastebin.com. ALWAYS INCLUDE and never truncate.

The Contribution Guidelines tell you where to find that.
-->

#### Link to contents of terminal tab or serial.log

<!--
On gist.github.com or pastebin.com. If applicable, always include if unsure or
reporting communication issues. Never truncate.

serial.log is usually not written due to performance reasons and must be
enabled explicitly. Provide at the very least the FULL contents of your
terminal tab at the time of the bug occurrence, even if you do not have
a serial.log (which the Contribution Guidelines tell you where to find).
-->

#### Link to contents of Javascript console in the browser

<!--
On gist.github.com or pastebin.com or alternatively a screenshot. If applicable -
always include if unsure or reporting UI issues.

The Contribution Guidelines tell you where to find that.
-->

#### Screenshot(s)/video(s) showing the problem:

<!--
If applicable. Always include if unsure or reporting UI issues.
-->

I have read the FAQ.

[Bug] Restore of backup on docker-mounted --basedir volume due to rename logic
<!--
READ THE FOLLOWING FIRST:

If not already done, please read the the Contribution Guidelines that
are linked to the right under "Helpful resources" > "Contributing".

Also read the FAQ: https://faq.octoprint.org

This is a bug and feature tracker, please only use it to report bugs
or request features within OctoPrint (not OctoPi, not any OctoPrint
plugins and not unofficial OctoPrint versions).

Do not seek support here ("I need help with ...", "I have a
question ..."), that belongs on the community forum at 
discourse.octoprint.org, NOT here.

Mark requests with a "[Request]" prefix in the title please. For bug
reports fully fill out the bug reporting template (if you don't know
where to find some information - it's all described in the Contribution
Guidelines linked up there in the big yellow box).

When reporting a bug do NOT delete ANY lines from the template.

Make sure any bug you want to report is still present with the CURRENT
OctoPrint version and that it does not vanish when you start OctoPrint
in safe mode - how to do that is also explained in the Contribution
Guidelines linked up there in the big yellow box.

Thank you!
-->

#### What were you doing?
Trying to restore a backup.

<!-- 
Please be as specific as possible here. The maintainers will need to
reproduce your issue in order to fix it and that is not possible if they
don't know what you did to get it to happen in the first place.

Ideally provide exact steps to follow in order to reproduce your problem:
-->

1. Create a backup using the backup mechanism and download it.
2. Start a second octoprint instance with the exact same versions/codebase of plugins etc.
3. Upload the backup zip file there
4. See error

```

Uploading backup, this can take a while. Please wait...
 
Stelle Sicherung wieder her...
 
Unpacking backup to /tmp/tmpYhY_SP...
Unpacked
Renaming /data/octorpint-data to /data/octorpint-data.bck...
Moving /tmp/tmpYhY_SP/basedir to /data/octorpint-data...
Error while restoring config data
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/OctoPrint-1.3.10-py2.7.egg/octoprint/plugins/backup/__init__.py", line 828, in _restore_backup
    os.rename(basedir_extracted, basedir)
OSError: [Errno 18] Invalid cross-device link
Rolling back old config data
 
Wiederherstellung fehlgeschlagen! Prüfe die obige Ausgabe sowie octoprint.log für mögliche Gründe dafür.
Removing temporary unpacked folder
Removing temporary zip

```
I started octoprint using this script inside a docker container on a raspberry pi 3b+:
```
#!/bin/bash

# ensure that octoprint data will be on 2nd+ level folder to not make restore of backups fail trying to rename the docker mapped volume /data
mkdir -p /data/octorpint-data

# start Octoprint
octoprint --iknowwhatimdoing --port=8080 --basedir /data/octorpint-data
```
As you can see in the script, I circumvented a related issue - the restore mechanism calls `os.rename` on a mounted volume that cannot be renamed resulting in `OSError: [Errno 18] Invalid cross-device link`. This is most probably related to docker, because I mount the /data directory from the host machine, as it is a common way to handle state with docker containers. See https://stackoverflow.com/questions/42392600/oserror-errno-18-invalid-cross-device-link for workaround suggestions. For me it was fine to move the data dir to `/data/octorpint-data` instead of `/data` to work around that issue. Sadly the `os.rename(a,b)` call failed again. Maybe you can just print out a warning in this case (Errno 18) or switch to a copy, then delete behaviour (here https://github.com/foosel/OctoPrint/blob/112a8b9c8bf7f5895f6a29dbe3d98ea4d9e0f400/src/octoprint/plugins/backup/__init__.py#L828)? Or maybe shutil will help working around that (as mentioned in the linked SO-link). 

<!--
If you encountered a problem with specific files of any sorts, make sure
to also include a link to a file with which to reproduce the problem.
-->

#### What did you expect to happen?
Restore of the state.

#### What happened instead?


#### Did the same happen when running OctoPrint in safe mode?

<!-- 
Test if you can reproduce your problem in safe mode. You can find information
on how to enable safe mode in the Contribution Guidelines.

If you can't reproduce in safe mode, this is a bug with one of your
installed third party plugins. Don't open a ticket here!

If you can't test this in safe mode, state why.
-->

#### Version of OctoPrint
1.3.10 (both instances)
<!--
Can be found in the lower left corner of the web interface. ALWAYS INCLUDE.
-->

#### Operating System running OctoPrint
raspian (not OctoPi)
<!--
OctoPi, Linux, Windows, MacOS, something else? With version please.
OctoPi's version can be found in /etc/octopi_version or in the lower left
corner of the web interface.
-->

#### Printer model & used firmware incl. version

<!--
If applicable, always include if unsure.
-->

#### Browser and version of browser, operating system running browser

<!--
If applicable, always include if unsure.
-->

#### Link to octoprint.log
https://gist.github.com/hgross/41fa4272b9549260c5a29a87b8a5ac03

<!--
On gist.github.com or pastebin.com. ALWAYS INCLUDE and never truncate.

The Contribution Guidelines tell you where to find that.
-->

#### Link to contents of terminal tab or serial.log

<!--
On gist.github.com or pastebin.com. If applicable, always include if unsure or
reporting communication issues. Never truncate.

serial.log is usually not written due to performance reasons and must be
enabled explicitly. Provide at the very least the FULL contents of your
terminal tab at the time of the bug occurrence, even if you do not have
a serial.log (which the Contribution Guidelines tell you where to find).
-->

#### Link to contents of Javascript console in the browser

<!--
On gist.github.com or pastebin.com or alternatively a screenshot. If applicable -
always include if unsure or reporting UI issues.

The Contribution Guidelines tell you where to find that.
-->

#### Screenshot(s)/video(s) showing the problem:

<!--
If applicable. Always include if unsure or reporting UI issues.
-->

I have read the FAQ.



Discussion:

Resolving patch:
diff --git a/src/octoprint/plugins/backup/__init__.py b/src/octoprint/plugins/backup/__init__.py
--- a/src/octoprint/plugins/backup/__init__.py
+++ b/src/octoprint/plugins/backup/__init__.py
@@ -684,7 +684,7 @@ def add_to_zip(source, target, ignored=None):
 				if len(plugins):
 					zip.writestr("plugin_list.json", json.dumps(plugins))
 
-			os.rename(temporary_path, final_path)
+			shutil.move(temporary_path, final_path)
 
 			if callable(on_backup_done):
 				on_backup_done(name, final_path, exclude)
@@ -834,18 +834,18 @@ def _restore_backup(cls, path,
 
 					if callable(on_log_progress):
 						on_log_progress(u"Renaming {} to {}...".format(basedir, basedir_backup))
-					os.rename(basedir, basedir_backup)
+					shutil.move(basedir, basedir_backup)
 
 					try:
 						if callable(on_log_progress):
 							on_log_progress(u"Moving {} to {}...".format(basedir_extracted, basedir))
-						os.rename(basedir_extracted, basedir)
+						shutil.move(basedir_extracted, basedir)
 					except:
 						if callable(on_log_error):
 							on_log_error(u"Error while restoring config data", exc_info=sys.exc_info())
 							on_log_error(u"Rolling back old config data")
 
-						os.rename(basedir_backup, basedir)
+						shutil.move(basedir_backup, basedir)
 
 						if callable(on_restore_failed):
 							on_restore_failed(path)


Test patch: