Repository: beetbox/beets

Issue:
Unicode error in new atomic move
### Problem

I’ve stumbled upon a bug when importing any file, seemingly related to the use of the `os.replace` function and the `import: move` option. using the copy option instead fixes the bug. According to adrian, it would be related to [Introduce atomic move and write of file](https://github.com/beetbox/beets/pull/4060)

```sh
Traceback (most recent call last):
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\util\__init__.py", line 496, in move
    os.replace(path, dest)
OSError: [WinError 17] The system cannot move the file to a different disk drive: '\\\\?\\C:\\Users\\hydro\\AppData\\Local\\Temp\\tmp02lvwxx1.flac' -> '\\\\?\\M:\\Musique lossless\\Ancient Methods\\2021 - The Jericho Remixes\\01 Swordplay (Hypnoskull vocal remix).flac'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\hydro\anaconda3\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\hydro\anaconda3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\hydro\anaconda3\Scripts\beet.exe\__main__.py", line 7, in <module>
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\ui\__init__.py", line 1285, in main
    _raw_main(args)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\ui\__init__.py", line 1272, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\ui\commands.py", line 973, in import_func
    import_files(lib, paths, query)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\ui\commands.py", line 943, in import_files
    session.run()
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\importer.py", line 340, in run
    pl.run_parallel(QUEUE_SIZE)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\util\pipeline.py", line 446, in run_parallel
    raise exc_info[1].with_traceback(exc_info[2])
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\util\pipeline.py", line 358, in run
    self.coro.send(msg)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\util\pipeline.py", line 170, in coro
    task = func(*(args + (task,)))
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\importer.py", line 1566, in manipulate_files
    task.manipulate_files(
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\importer.py", line 757, in manipulate_files
    item.move(operation)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\library.py", line 922, in move
    self.move_file(dest, operation)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\library.py", line 821, in move_file
```

### Setup

* OS: Windows 10
* Python version: 3.9.7
* beets version: 1.6.0
* Turning off plugins made problem go away (yes/no): irrelevant

The relevant bit of my configuration (output of `beet config`) is:

```yaml
# Import options
import:
  write: yes
  # copy: yes
  move: yes
  resume: no
  autotag: yes
```

Fixed `can't concat str to byte` inside move
## Description

`os.path.basename` returns the same type that was used.

So, I have no guarantee that I needed here: `b'.''` or `'.'`.

Let use something universal ;)

Closes #4168


Discussion:
Thank you for filing! Is there any chance that traceback got cut off at the end? It doesn't seem to end in the line that's the origin of the second exception.
Same issue here:

The error im getting is always the same except for the name of the file that causes the error:

```
Traceback (most recent call last):

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\util\__init__.py", line 496, in move

os.replace(path, dest)

OSError: [WinError 17] The system cannot move the file to a different disk drive: '\\\\?\\C:\\Users\\Nuc8\\AppData\\Local\\Temp\\tmp_fkqq4ch.jpg' -> '\\\\?\\D:\\_Sorted\\#artists\\Animal Collective\\2007 - Strawberry Jam [Domino WIGCD199] [24 96kHz FLAC]\\cover.jpg'


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

File "C:\Users\Nuc8\AppData\Local\Programs\Python\Python37\Scripts\beet-script.py", line 11, in <module>

load_entry_point('beets==1.6.0', 'console_scripts', 'beet')()

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\ui\__init__.py", line 1285, in main

_raw_main(args)

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\ui\__init__.py", line 1272, in _raw_main

subcommand.func(lib, suboptions, subargs)

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\ui\commands.py", line 973, in import_func

import_files(lib, paths, query)

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\ui\commands.py", line 943, in import_files

session.run()

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\importer.py", line 340, in run

pl.run_parallel(QUEUE_SIZE)

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\util\pipeline.py", line 446, in run_parallel

raise exc_info[1].with_traceback(exc_info[2])

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\util\pipeline.py", line 358, in run

self.coro.send(msg)

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\util\pipeline.py", line 170, in coro

task = func(*(args + (task,)))

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\importer.py", line 1569, in manipulate_files

session=session,

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\importer.py", line 766, in manipulate_files

plugins.send('import_task_files', session=session, task=self)

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\plugins.py", line 488, in send

result = handler(**arguments)

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\plugins.py", line 145, in wrapper

return func(*args, **kwargs)

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beetsplug\fetchart.py", line 1063, in assign_art

self._set_art(task.album, candidate, not self.src_removed)

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beetsplug\fetchart.py", line 1048, in _set_art

album.set_art(candidate.path, delete)

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\library.py", line 1317, in set_art

util.move(path, artdest)

File "c:\users\nuc8\appdata\local\programs\python\python37\lib\site-packages\beets\util\__init__.py", line 499, in move

prefix=py3_path(b'.' + os.path.basename(dest)),

TypeError: can't concat str to bytes"
```


Here is my congif.yaml:

https://pastebin.com/9M03TXdU






I don't have enough knowledge of python to be able to figure this out on my own, any help is truly appreciated.


thanks in advance to anyone who can help.
Thanks for the complete traceback; this definitely looks like a bug!

@catap, this bug was introduced in your PR #4060. The concatenation here assumes incorrectly that `os.path.basename` will produce a bytes object:
https://github.com/beetbox/beets/pull/4060/files#diff-8c8be7ee9289c87f5f470e1b6edda131dbfa07272a9afaaa92ba42e7d766702eR499

Can you please take a look?
@LeoSebal this should fix your issue. Because it is a blind shot may I ask you to test it? :)
I tested this fix and it works for me. Windows 11, Python 3.11.
Hi @catap

Sorry for the delayed answer, last week was pretty busy on my end.

That does not seem to do it. Admittedly, I might be the problem as I'm not used to troubleshooting beets.

What I did:
1. Checked out your branch
2. Ran `python setup.py install` from the repo root
3. Edited my configuration to move instead of copy files
4. Tried to import

Windows 10, Python 3.9.7 (conda 4.10.3)

Note that I am trying to import from on disk (I:) to another (M:), neither of which is the System disk in which the Temp folder is located.

Full log:
```python
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\util\__init__.py", line 496, in move
    os.replace(path, dest)
OSError: [WinError 17] The system cannot move the file to a different disk drive: '\\\\?\\C:\\Users\\hydro\\AppData\\Local\\Temp\\tmpcxi4_q0j.flac' -> '\\\\?\\M:\\Musique lossless\\FTP\\2020 - Sacred Tranz Tanz\\01 Thoughts on Time.flac'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\hydro\anaconda3\Scripts\beet-script.py", line 33, in <module>
    sys.exit(load_entry_point('beets==1.6.1', 'console_scripts', 'beet')())
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\ui\__init__.py", line 1285, in main

    _raw_main(args)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\ui\__init__.py", line 1272, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\ui\commands.py", line 973, in import_func
    import_files(lib, paths, query)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\ui\commands.py", line 943, in import_files
    session.run()
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\importer.py", line 340, in run
    pl.run_parallel(QUEUE_SIZE)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\util\pipeline.py", line 446, in run_parallel
    raise exc_info[1].with_traceback(exc_info[2])
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\util\pipeline.py", line 358, in run

    self.coro.send(msg)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\util\pipeline.py", line 170, in coro
    task = func(*(args + (task,)))
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\importer.py", line 1566, in manipulate_files
    task.manipulate_files(
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\importer.py", line 757, in manipulate_files
    item.move(operation)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\library.py", line 922, in move
    self.move_file(dest, operation)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\library.py", line 821, in move_file

    util.move(self.path, dest)
  File "C:\Users\hydro\anaconda3\lib\site-packages\beets\util\__init__.py", line 499, in move
    prefix=py3_path(b'.' + os.path.basename(dest)),
TypeError: can't concat str to bytes
```

@LeoSebal are you sure that you have updated the code and used as local root a branch for this PR?

I'm asking because I see in your stack trace at the end: `prefix=py3_path(b'.' + os.path.basename(dest)),` and in this PR it was changed to `prefix = py3_path('.') + os.path.basename(py3_path(dest))`.
@catap  I believe so: I tried to fetch and checkout again, and I believe I did it correctly. The lines 498...501 from `beets/util/__init__.py` correspond to your modifications.

I used the command `git fetch origin pull/4192/head:atomic-followup` and then checked out the branch, but I'm not used to using Github so it might be incorrect?
@LeoSebal it is wired.

Anyway, I see that an error happened in file `C:\Users\hydro\anaconda3\lib\site-packages\beets\util\__init__.py`.

Can you confirm that you run `pip install` or something similar from branch?

Because right now from provided stacktrace I see that this is the same error that @aibradford  had reported in the issue and which I've tried to fix :) 
@sampsyo ping here! ;)
I used `python setup.py install`, which seems to work: I reran the install just now, and the executable called by the `beet` command has been modified a few minutes ago.
@LeoSebal have it failed after you last attempt to reinstall it?
(oops, edited my previous comment which contained an error)

Yes, imports with move still seem to be failing.
@LeoSebal can you share a new stackrace?
To eliminate the guesswork, I tried to confirm exactly what the Python 3 standard library wants here. It is of course fine with all Unicode strings:

```py
>>> tempfile.mktemp(suffix='a', prefix='b', dir='.')
'./bdwgl4nk4a'
```

And it, quite rightly, rejects a mixture of `bytes`s and `str`s:

```py
>>> tempfile.mktemp(suffix='a', prefix='b', dir=b'.')
[...]
    raise TypeError("Can't mix strings and bytes in path components") from None
TypeError: Can't mix strings and bytes in path components
```

But the sad part is that this API also rejects all-bytes invocations:

```py
>>> tempfile.mktemp(suffix=b'a', prefix=b'b', dir=b'.')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/tempfile.py", line 403, in mktemp
    file = _os.path.join(dir, prefix + name + suffix)
TypeError: can't concat str to bytes
```

I despise Python path APIs that insist on Unicode arguments, and I think any API that does this should be considered buggy, so this led me to actually read the docs. [`mktemp` is deprecated](https://docs.python.org/3/library/tempfile.html#tempfile.mktemp) and we should not be using it at all! I wish I had looked more closely at this when it was originally proposed. Let's instead use one of the supported interfaces, such as `mkstemp` or `TemporaryFile`, which—as an added bonus—actually seem to support a bytes interface:

```py
>>> tempfile.mkstemp(suffix=b'a', prefix=b'b', dir=b'.')
(3, b'/Users/asampson/Desktop/XXX/b_d49w89_a')
```

So we will not need any of this `py3_path` hackery at all.

Resolving patch:
diff --git a/beets/util/__init__.py b/beets/util/__init__.py
--- a/beets/util/__init__.py
+++ b/beets/util/__init__.py
@@ -479,31 +479,40 @@ def move(path, dest, replace=False):
     instead, in which case metadata will *not* be preserved. Paths are
     translated to system paths.
     """
-    if os.path.isdir(path):
+    if os.path.isdir(syspath(path)):
         raise FilesystemError(u'source is directory', 'move', (path, dest))
-    if os.path.isdir(dest):
+    if os.path.isdir(syspath(dest)):
         raise FilesystemError(u'destination is directory', 'move',
                               (path, dest))
     if samefile(path, dest):
         return
-    path = syspath(path)
-    dest = syspath(dest)
-    if os.path.exists(dest) and not replace:
+    if os.path.exists(syspath(dest)) and not replace:
         raise FilesystemError('file exists', 'rename', (path, dest))
 
     # First, try renaming the file.
     try:
-        os.replace(path, dest)
+        os.replace(syspath(path), syspath(dest))
     except OSError:
-        tmp = tempfile.mktemp(suffix='.beets',
-                              prefix=py3_path(b'.' + os.path.basename(dest)),
-                              dir=py3_path(os.path.dirname(dest)))
-        tmp = syspath(tmp)
+        # Copy the file to a temporary destination.
+        basename = os.path.basename(bytestring_path(dest))
+        dirname = os.path.dirname(bytestring_path(dest))
+        tmp = tempfile.NamedTemporaryFile(
+            suffix=syspath(b'.beets', prefix=False),
+            prefix=syspath(b'.' + basename, prefix=False),
+            dir=syspath(dirname),
+            delete=False,
+        )
+        try:
+            with open(syspath(path), 'rb') as f:
+                shutil.copyfileobj(f, tmp)
+        finally:
+            tmp.close()
+
+        # Move the copied file into place.
         try:
-            shutil.copyfile(path, tmp)
-            os.replace(tmp, dest)
+            os.replace(tmp.name, syspath(dest))
             tmp = None
-            os.remove(path)
+            os.remove(syspath(path))
         except OSError as exc:
             raise FilesystemError(exc, 'move', (path, dest),
                                   traceback.format_exc())


Test patch: