Repository: beetbox/beets

Issue:
hook: Interpolate paths (and other bytestrings) correctly into commands
### Problem

I have the following configuration for the Hook plugin in my config.yaml:

```
hook:
    hooks:
        - event: album_imported
          command: /usr/bin/ls -l "{album.path}"
```

This is just a test to see how beets presents the path values.  It appears that the paths are returned as bytes objects rather than strings.  This is problematic when using the path values as arguments for external shell scripts.  As can be seen below, the shell is unable to use the value provided by {album.path}.


```sh
$ beet -vv import /tmp/music/new/Al\ Di\ Meola\ -\ Elegant\ Gypsy/
```

Led to this problem:

```
hook: running command "/usr/bin/ls -l b'/tmp/music/FLAC/Al Di Meola/Elegant Gypsy'" for event album_imported
/usr/bin/ls: cannot access "b'/tmp/music/FLAC/Al Di Meola/Elegant Gypsy'": No such file or directory

```

The path "/tmp/music/FLAC/Al Di Meola/Elegant Gypsy" does exist on the filesystem after the import is complete.



### Setup

* OS: Arch Linux
* Python version: 3.4.5
* beets version: 1.4.7
* Turning off plugins made problem go away (yes/no): This issue is related to a plugin, so I didn't turn them off

My configuration (output of `beet config`) is:

```yaml
plugins: inline convert badfiles info missing lastgenre fetchart mbsync scrub smartplaylist hook
directory: /tmp/music/FLAC
library: ~/.config/beets/library.db

import:
    copy: yes
    write: yes
    log: ~/.config/beets/import.log
    languages: en
per_disc_numbering: yes

paths:
    default: $albumartist/$album%aunique{}/$disc_and_track - $title
    comp: $albumartist/$album%aunique{}/$disc_and_track - $title
item_fields:
    disc_and_track: u'%01i-%02i' % (disc, track) if disctotal > 1 else u'%02i' % (track)

ui:
    color: yes

match:
    ignored: missing_tracks unmatched_tracks
convert:
    copy_album_art: yes
    dest: /tmp/music/ogg
    embed: yes
    never_convert_lossy_files: yes
    format: ogg
    formats:
        ogg:
            command: oggenc -Q -q 4 -o $dest $source
            extension: ogg
        aac:
            command: ffmpeg -i $source -y -vn -acodec aac -aq 1 $dest
            extension: m4a
        alac:
            command: ffmpeg -i $source -y -vn -acodec alac $dest
            extension: m4a
        flac: ffmpeg -i $source -y -vn -acodec flac $dest
        mp3: ffmpeg -i $source -y -vn -aq 2 $dest
        opus: ffmpeg -i $source -y -vn -acodec libopus -ab 96k $dest
        wma: ffmpeg -i $source -y -vn -acodec wmav2 -vn $dest
    pretend: no
    threads: 4
    max_bitrate: 500
    auto: no
    tmpdir:
    quiet: no

    paths: {}
    no_convert: ''
    album_art_maxwidth: 0
lastgenre:
    force: yes
    prefer_specific: no
    min_weight: 20
    count: 4
    separator: '; '
    whitelist: yes
    fallback:
    canonical: no
    source: album
    auto: yes
fetchart:
    sources: filesystem coverart amazon albumart
    auto: yes
    minwidth: 0
    maxwidth: 0
    enforce_ratio: no
    cautious: no
    cover_names:
    - cover
    - front
    - art
    - album
    - folder
    google_key: REDACTED
    google_engine: 001442825323518660753:hrh5ch1gjzm
    fanarttv_key: REDACTED
    store_source: no
hook:
    hooks: [{event: album_imported, command: '/usr/bin/ls -l "{album.path}"'}]
pathfields: {}
album_fields: {}
scrub:
    auto: yes
missing:
    count: no
    total: no
    album: no
smartplaylist:
    relative_to:
    playlist_dir: .
    auto: yes
    playlists: []
```

I created a Python 2 virtual environment, installed beets and any dependencies in to that virtualenv, cleaned my test library, and imported the same files using the same config.yaml.  This time the shell  was able to use the path value returned by the hook configuration:

```
hook: running command "/usr/bin/ls -l /tmp/music/FLAC/Al Di Meola/Elegant Gypsy" for event album_imported
total 254944
-rw-r--r-- 1 mike mike 50409756 Jun 24 13:46 01 - Flight Over Rio.flac
-rw-r--r-- 1 mike mike 43352354 Jun 24 13:46 02 - Midnight Tango.flac
-rw-r--r-- 1 mike mike  7726389 Jun 24 13:46 03 - Percussion Intro.flac
-rw-r--r-- 1 mike mike 32184646 Jun 24 13:46 04 - Mediterranean Sundance.flac
-rw-r--r-- 1 mike mike 45770796 Jun 24 13:46 05 - Race With Devil on Spanish Highway.flac
-rw-r--r-- 1 mike mike 10421006 Jun 24 13:46 06 - Lady of Rome, Sister of Brazil.flac
-rw-r--r-- 1 mike mike 65807504 Jun 24 13:46 07 - Elegant Gypsy Suite.flac
-rw-r--r-- 1 mike mike  5366515 Jun 24 13:46 cover.jpg
```
I'm guessing this is due to a data type difference between Python 2 and Python 3.


Discussion:
Thank you! This is indeed a Python 3 regression. The problem is that we’re interpolating paths (which are bytes) into commands (which are Unicode) stings. Rather than just crash, Python 3 opts to use `repr()` to convert the bytes objects to strings for interpolation. This obviously won’t work for executing hooks that need actual filenames.

Fixing this will require some care. We’ll need to convert the commands (and any string parameters) to bytes and include the raw, unconverted filenames in the output. This is not something that plain string templating can do, it will require some sort of hack to get right.
I'm hitting this issue as well. As a terrible temporary hacky workaround, I'm passing the filename to a bash script, where I'm doing `echo $1 | cut -c 3- | rev | cut -c 2- | rev | xargs -0 osascript /Users/evan/bin/iTunesRefresh.scpt`. This turns `b'foo/path'` into `foo/path`
@sampsyo I've not tried it out but is there a reason our custom `CodingFormatter` is only being used on Python 2? I recall writing it to specifically handle encoding interpolated parameters.

https://github.com/beetbox/beets/blob/be118b922a92a39469985a2f45766c0f094c1963/beetsplug/hook.py#L27-L70

https://github.com/beetbox/beets/blob/be118b922a92a39469985a2f45766c0f094c1963/beetsplug/hook.py#L99-L102
Huh! It definitely looks like it's my fault—I committed this in 039825e0e0eaf02404fd0ea2eb3272128c33cebe.

That does look wrong to me now—this formatter is exactly what you'd want if you were to be interpolating byte strings into Unicode templates, which is exactly what we're doing. 😳

I think I was probably just mistaken at the time! Can you please check whether everything just works if we enable the formatter on Python 3 too? That would be great but extremely embarrassing for me.

Resolving patch:
diff --git a/beetsplug/hook.py b/beetsplug/hook.py
--- a/beetsplug/hook.py
+++ b/beetsplug/hook.py
@@ -18,7 +18,6 @@
 
 import string
 import subprocess
-import six
 
 from beets.plugins import BeetsPlugin
 from beets.util import shlex_split, arg_encoding
@@ -46,10 +45,8 @@ def format(self, format_string, *args, **kwargs):
 
         See str.format and string.Formatter.format.
         """
-        try:
+        if isinstance(format_string, bytes):
             format_string = format_string.decode(self._coding)
-        except UnicodeEncodeError:
-            pass
 
         return super(CodingFormatter, self).format(format_string, *args,
                                                    **kwargs)
@@ -96,10 +93,7 @@ def hook_function(**kwargs):
                 return
 
             # Use a string formatter that works on Unicode strings.
-            if six.PY2:
-                formatter = CodingFormatter(arg_encoding())
-            else:
-                formatter = string.Formatter()
+            formatter = CodingFormatter(arg_encoding())
 
             command_pieces = shlex_split(command)
 


Test patch:
diff --git a/test/test_hook.py b/test/test_hook.py
--- a/test/test_hook.py
+++ b/test/test_hook.py
@@ -110,6 +110,25 @@ def test_hook_argument_substitution(self):
             self.assertTrue(os.path.isfile(path))
             os.remove(path)
 
+    def test_hook_bytes_interpolation(self):
+        temporary_paths = [
+            get_temporary_path().encode('utf-8')
+            for i in range(self.TEST_HOOK_COUNT)
+        ]
+
+        for index, path in enumerate(temporary_paths):
+            self._add_hook('test_bytes_event_{0}'.format(index),
+                           'touch "{path}"')
+
+        self.load_plugins('hook')
+
+        for index, path in enumerate(temporary_paths):
+            plugins.send('test_bytes_event_{0}'.format(index), path=path)
+
+        for path in temporary_paths:
+            self.assertTrue(os.path.isfile(path))
+            os.remove(path)
+
 
 def suite():
     return unittest.TestLoader().loadTestsFromName(__name__)