Repository: beetbox/beets

Issue:
UTF-8 Character Encoding Breaks Somewhere on Windows
### Problem

I've tried a few Unicode character replacements for the regular colon ":". None of them work. I have an example that creates a folder called:

日比野則彦 - 2004 - Metal Gear Solid 3**â€“** Snake Eaterâ€“ The First Bite

notice the `â€“` instead of `–` (em dash, not en dash or hyphen). But I can use that character with a normal rename on a file in Windows 10. I'm running Win10 x64, and beets in the normal Windows CMD program.

Running this command in verbose (`-vv`) mode:

```sh
beet -vv import "e:\use\test\01. Snake Eater (Abstracted Camouflage).opus"
plugin paths:
Sending event: pluginload
library database: e:\songs\beetslibrary.bib
library directory: e:\s2
Sending event: library_opened
Sending event: import_begin
Sending event: import_task_created
Sending event: import_task_start
Looking up: e:\use\test\01. Snake Eater (Abstracted Camouflage).opus
Tagging Hibino Norihiko - Metal Gear Solid 3: Snake Eater: The First Bite
Searching for discovered album ID: 48746b35-d2bf-4199-9e44-69c2869b03bc
Candidate: 日比野則彦 - Metal Gear Solid 3: Snake Eater: The First Bite
Success. Distance: 0.31
Album ID match recommendation is Recommendation.low
Search terms: Hibino Norihiko - Metal Gear Solid 3: Snake Eater: The First Bite
Album might be VA: False
Sending event: albuminfo_received
Candidate: 日比野則彦 - Metal Gear Solid 3: Snake Eater: The First Bite
Duplicate.
Sending event: albuminfo_received
Candidate: Harry Gregson‐Williams & 日比野則彦 - Metal Gear Solid 3: Snake Eater
Success. Distance: 0.90
Sending event: albuminfo_received
Candidate: Harry Gregson‐Williams & 日比野則彦 - Metal Gear Solid 3: Snake Eater
Success. Distance: 0.89
Sending event: albuminfo_received
Candidate: Dan Bull feat. VI Seconds - Metal Gear Solid
Success. Distance: 0.83
Sending event: albuminfo_received
Candidate: Cynthia Harrell - Snake Eater -abstracted camouflage-
Success. Distance: 0.70
Evaluating 5 candidates.

e:\use\test\01. Snake Eater (Abstracted Camouflage).opus (1 items)
Sending event: before_choose_candidate
Correcting tags from:
    Hibino Norihiko - Metal Gear Solid 3: Snake Eater: The First Bite
To:
    日比野則彦 - Metal Gear Solid 3: Snake Eater: The First Bite
URL:
    https://musicbrainz.org/release/48746b35-d2bf-4199-9e44-69c2869b03bc
(Similarity: 68.8%) (missing tracks, artist) (CD, 2004, JP, コナミ株式会社)
Missing tracks (5/6 - 83.3%):
 ! Infiltration Into the Jungle (#2) (4:13)
 ! Escape (#3) (2:00)
 ! Chivalry (#4) (3:20)
 ! The Treading Behemoth (#5) (2:07)
 ! Snake Eater (Japanese version) (#6) (5:00)
Apply, More candidates, Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort?
Enter one of A, M, S, U, T, G, E, I, B: a
Sending event: import_task_choice
Sending event: import_task_apply
0 of 1 items replaced
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: database_change
Sending event: before_item_moved
Sending event: item_moved
Sending event: database_change
Sending event: database_change
Sending event: write
Sending event: after_write
Sending event: database_change
Sending event: import_task_files
Sending event: album_imported
Sending event: import
Sending event: cli_exit
```

Here's a link to the music files that trigger the bug (if relevant): 
[01. Snake Eater (Abstracted Camouflage).zip](https://github.com/beetbox/beets/files/801456/01.Snake.Eater.Abstracted.Camouflage.zip)
 (dummy file)

### Setup

* OS: Win10 x64
* beets version 1.4.3
* Python version 3.6.0
* Turning off plugins made problem go away (yes/no): no

My configuration (output of `beet config`) is:

```yaml
directory: e:\s2\
library: e:\songs\beetslibrary.bib
import:
    copy: no
    move: yes
    write: yes
    timid: no
per_disc_numbering: yes


replace:
replace:
    '[\\/]': _
    '^\.': _
    '[\x00-\x1f]': _
    '[<>"\?\*\|]': _
    '\.$': _
    '\s+$': ''
    '^\s+': ''
    '\:': –

paths:
    default: $albumartist - $original_year - $album/$track. $title 
```

(A hasty copy paste led to my config having two "replace" lines but it shouldn't affect the bug).
Beets is haunted, reads info from comment lines?
### Problem

A lot is going on here. May be many bugs combining? Some thoughts:

* Beets reading info to the right of a comment indicator `#` which is unintuitive
* Information in replace somehow ruining musicbrainz.host
* Maybe something to do with using git branch rather than latest stable (1.4.3)?
* Maybe something to do with latest commit?

Running this command in verbose (`-vv`) mode:

```sh
beet -vv
configuration error: musicbrainz.host not found
```

### Setup

* OS: win10 x64
* Python version: 3.6
* beets version: Latest git (https://github.com/beetbox/beets/commit/9840964f51fed3b3d0dcc1db9c76f90be4c05e73)
* Turning off plugins made problem go away (yes/no): no

My configuration (output of `beet config`) is:

Causes issue:

```yaml
library: e:\songs\beetslibrary.bib

replace:
    'qqqqqqq': 'eeeeeeeee'
    #”
```

Works fine:

```yaml
library: e:\songs\beetslibrary.bib

replace:
    'qqqqqqq': 'eeeeeeeee'
    #
```


Discussion:
That is indeed very strange!

To help diagnose this, can you try running this incantation at the Python prompt to see what beets does? (Type `python` or `python3.6` in your console, and then use ctrl-Z to exit the prompt when you're done.)

```
>>> from beets import util
>>> from beets import ui
>>> ui.get_replacements()
(this will dump out a representation of your replace config)
>>> util.sanitize_path('abc:def', ui.get_replacements())
(this will try sanitizing a path using your replace config)
```

That way, we can check whether we're doing the right thing *before* shipping the path off to the OS. If that works, we can try investigating how the correct beets-internal path gets mangled as it gets encoded for the OS call.
This is from my main beets config, which has more elaborate replacements, but the bug is still present.

```
>>> from beets import util
>>> from beets import ui
>>> ui.get_replacements()
[(re.compile('[\\\\/]'), '_'), (re.compile('^\\.'), '_'), (re.compile('[\\x00-\\x1f]'), '_'), (re.compile('\\.$'), '_'), (re.compile('\\s+$'), ''), (re.compile('^\\s+'), ''), (re.compile('[<>\\|]'), '_'), (re.compile('\\:'), 'â€“'), (re.compile('"'), "\\'\\'"), (re.compile('[\\*]'), 'âœ¶'), (re.compile('[\\?]'), 'ï¼Ÿ')]
>>> util.sanitize_path('abc:def', ui.get_replacements())
'abcâ€“def'
```
Hmm, so it looks like the problem is reading the configuration itself. Could the problem be that your text editor (which you're using to edit config.yaml) is using some older encoding instead of UTF-8?
I guess we should also ask Python which encoding it's using for files. For example, on my machine:

```
>>> open('/dev/null', 'r').encoding
'UTF-8'
```

(On Windows, you may need to point it toward a real text file instead of using /dev/null.)
```
>>> open('e:\\blank.txt', 'r').encoding
'cp1252'
>>> open('e:\\config.yaml', 'r').encoding
'cp1252'
```

So there's the problem, but how to fix it?

edit: I've tried a few ways in Notepad++ to save it as UTF-8. Everything in Notepad++ says it's UTF-8, but Python returns the same result.
Huh, that'll do it. One option would be to write your config file using that encoding, somehow, or to find the magical Windows incantation that changes the default IO encoding to a modern one (i.e., UTF-8). [This SO answer](http://stackoverflow.com/a/11516682) suggests that the incantation might be `chcp 65001`.
Even when setting my console (cmd and powershell both) to UTF-8, the filename that is written ends up garbled. edit: This happens even when setting the config file to UTF-8 with Byte Order Mark (BOM).

edit:

```
>>> open('e:\\config.yaml', 'r').encoding
'cp1252'
>>> open('e:\\config.yaml', 'r', encoding="utf-8").encoding
'utf-8'
```

This would probably be a quicker workaround than anywhere else. Where does beets open the config file?
It's surprisingly hard to set the Windows configuration to get this right—I'm not sure why, but the settings for console I/O and file I/O all have separate, obscure incantations. [This SO question](http://stackoverflow.com/q/27452317) expresses the same confusion.

Here are the two lines where config files are opened for reading:
https://github.com/beetbox/beets/blob/1c9d42da95e09b67dc496a328d81ba79e179f84d/beets/util/confit.py#L671
https://github.com/beetbox/beets/blob/1c9d42da95e09b67dc496a328d81ba79e179f84d/beets/util/confit.py#L911

It might be a good idea to force those to be opened in *binary* mode, with no Python-provided default encoding. That way, PyYAML gets to choose the encoding—and it supports only reasonable encodings (UTF-8 and UTF-16, as described [here](http://pyyaml.org/wiki/PyYAMLDocumentation#LoadingYAML)). In fact, that was exactly the behavior we already had on Python 2, which never seemed to cause a problem…
@sampsyo: isn't some of that outdated for python 3.6.0? 

In any case we should force the encoding to utf-8 regardless for any yaml files.
@sampsyo : is there already proof that saving a yaml file with PyYaml will save it as utf-8 by default windows?
@RollingStar : can you try with conemu terminal forced to utf-8?
Yeah, the console encoding stuff certainly is. But I can't find information on what 3.6 did for the default file encoding for `open`.

On Python 3, I believe `yaml.dump` just gives you a Unicode string.
I tried with ConEmu around the time of my edit, actually. The result (ConEmu x64, Python x86):

```
>chcp 65001                                                                    
Active code page: 65001                                                          
> python                                                                        
Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 07:18:10) [MSC v.1900 32 bit (In
tel)] on win32                                                                  
Type "help", "copyright", "credits" or "license" for more information.          
>>> open('e:\\config.yaml', 'r').encoding                                       
'cp1252'
```

edit: Hacky workaround that looks like it will work for my purposes. I don't know if you want to investigate opening the configs as binaries.

        with open(filename, 'r', encoding="utf-8") as f:
        with open(default_source.filename, 'r', encoding="utf-8") as fp:

The result: 

![image](https://cloud.githubusercontent.com/assets/4810902/23337157/a8a43d0c-fbb2-11e6-8591-0604ee871a85.png)

Unicode trickery makes impossible filenames possible!
Making these two changes helps me handle Unicode on Windows. Should this be changed in the beets code?

    with open(filename, 'r', encoding="utf-8") as f:
    with open(default_source.filename, 'r', encoding="utf-8") as fp:
Intriguing that you found a fix… which file are you referring to, exactly? Is this in Confit, and is the change forcing the `encoding` in the `open` call?
Yes. I open confit.py and make the changes at the places where the config is loaded.

Change [this](https://github.com/beetbox/beets/blob/1c9d42da95e09b67dc496a328d81ba79e179f84d/beets/util/confit.py#L671) to `with open(filename, 'r', encoding="utf-8") as f:`

Change [this](https://github.com/beetbox/beets/blob/1c9d42da95e09b67dc496a328d81ba79e179f84d/beets/util/confit.py#L911) to `with open(default_source.filename, 'r', encoding="utf-8") as fp:`

Actually, just changing the first one fixed my issue from #2565. I suspect if there was unicode in the default_source.filename, you would need the second change (above) as well. This edit has worked fine for me in Beets since I made that edit. I only found a Unicode error today when I reinstalled Beets and forgot about my little hotfix.
Hmm; that's odd—this one I can't seem to reproduce.

```
$ cat config.yaml
library: e:\songs\beetslibrary.bib

replace:
    'qqqqqqq': 'eeeeeeeee'
    #”
$ beet -vv config
[...clip...]
Sending event: library_opened
verbose: 2
library: e:\songs\beetslibrary.bib

replace:
    qqqqqqq: eeeeeeeee

Sending event: cli_exit
```

Based on our experience last time, I also tried removing the newline from the end of the file—still no issue.

I'm kind of stumped. Maybe it has something to do with Windows?
Happens with or without the newline. Note the troublesome character is [U+201D](http://www.fileformat.info/info/unicode/char/201d/index.htm) and not a normal quote `"`. All these don't work.

(This first one fails on Powershell as well as normal CMD. I only tested with normal CMD for the others.)

```
library: e:\songs\beetslibrary.bib

replace:
    'q': '”'
```

```
library: e:\songs\beetslibrary.bib

paths:
    default: '”'
```

```
library: e:\songs\beetslibrary.bib

paths:
    default: ”
```

This works:

```
library: e:\songs\beetslibrary.bib

paths:
    default: a
```


So, maybe `”` is being treated as an escape character? If the text handler parses escape characters before comments `#` that would explain all the results thus far. edit: By escape character, I mean maybe the text handler stops reading the config once it gets to `”`. This leaves beets with a corrupted user config, and it can't find musicbrainz.host. Normally I assume it loads the user config in its entirety, then fills the remainder from the program default config. Or it does it the other way, and this half-loaded user config causes beets to not have a value saved for musicbrainz.host.

edit 2: Delete `␡` also causes problems, even in a comment. I thought I used Unicode like this without issue in previous months. 

Maybe the latest Win10 Creators Update changed my console settings? Maybe not.

```
>chcp 65001
Active code page: 65001

>beet -vv
configuration error: musicbrainz.host not found
```

Still, because it may be a clue, I'll link to #2456.

edit 3: I forgot that in #2456 I made my own patch to Beets and have been using it since today, when I updated from stable to latest Git. I'll check my fix against these latest issues. I'm guessing this issue is a dupe of 2456.
Okay. Applying my fix from #2456 fixes my issue with the quotes and DEL and everything else. However, could error handling be improved for other unforseeable errors?

* Beets reading info to the right of a comment indicator `#`. Is Beets reading line by line, or the entire file at once? Line by line would have a benefit of being more assured that beets is not reading any line after # appears.
* The "configuration error" that is thrown. What is happening between program defaults & user config that causes this error?
OK—that sounds like we're dealing with an encoding error.

I'm guessing that the YAML parser is somehow giving up entirely when it encounters a character it doesn't understand. I'm still at a loss to explain why that leads to the configuration error you're seeing—it seems to throw off the whole configuration system, rather than just exiting with a parse error. It's hard to narrow down what's wrong without a way to reproduce the problem.

Resolving patch:
diff --git a/beets/util/confit.py b/beets/util/confit.py
--- a/beets/util/confit.py
+++ b/beets/util/confit.py
@@ -668,7 +668,7 @@ def load_yaml(filename):
     parsed, a ConfigReadError is raised.
     """
     try:
-        with open(filename, 'r') as f:
+        with open(filename, 'rb') as f:
             return yaml.load(f, Loader=Loader)
     except (IOError, yaml.error.YAMLError) as exc:
         raise ConfigReadError(filename, exc)
@@ -908,9 +908,10 @@ def dump(self, full=True, redact=False):
                 default_source = source
                 break
         if default_source and default_source.filename:
-            with open(default_source.filename, 'r') as fp:
+            with open(default_source.filename, 'rb') as fp:
                 default_data = fp.read()
-            yaml_out = restore_yaml_comments(yaml_out, default_data)
+            yaml_out = restore_yaml_comments(yaml_out,
+                                             default_data.decode('utf8'))
 
         return yaml_out
 


Test patch: