Repository: beetbox/beets

Issue:
MediaFile: use older text encodings in ID3v2.3 mode
I am trying to create an auto-tagging configuration in which my tags are saved in ID3v2.3 (as 2.4 lacks compatibility with some players I use) and I like the cover art to be embedded in each music file. However, the cover art of the output files is not recognised by OS X 10.9.4 (i.e. in Finder) or iTunes.

Here is a simple configuration with which the problem occurs:

``` yaml
directory: /mnt/data/home/Music

plugins: mbsync fetchart embedart
per_disc_numbering: yes
id3v23: yes

import:
    copy: yes
    write: yes

paths:
    default: $albumartist/$album%aunique{}/$disc-$track $title
```

When I comment the `id3v23: yes` option the covers of the output files are correctly recognised in Mac.

The cover art of the ID3v2.3 output files is recognised in Windows, so it seems a Mac-specific issue. Strangely enough, the input files I used already have ID3v2.3 tags and embedded cover art and are correctly recognised in Mac. Below you have a diff of the ID3v2.3 tags (sorted by name) between an input and an output file taken with `mid3v2`:

``` diff
1c1,2
< APIC= (image/jpeg, 32205 bytes)

---
> APIC= (image/jpeg, 111083 bytes)
> COMM=iTunNORM='eng'= 00001700 00001700 00003981 00003981 00000000 00000000 00008187 00008187 00000000 00000000
3a5,6
> TBPM=0
> TCMP=0
5a9
> TLAN=eng
12a17,19
> TXXX=Album Artist Credit=The Beatles
> TXXX=ALBUMARTISTSORT=Beatles, The
> TXXX=Artist Credit=The Beatles
14a22
> TXXX=MusicBrainz Album Comment=UK mono
17c25
< TXXX=MusicBrainz Album Status=official

---
> TXXX=MusicBrainz Album Status=Official
27a36
> USLT=[unrepresentable data]
```



Discussion:
Forgot to mention I'm using beets 1.3.6.

Thanks for the report! This does indeed seem to be a problem. Investigating now.

Got a reproducible test case. In [this zip](https://www.dropbox.com/s/wsced6rfl9o1a1o/id3v23_image_bug.zip), run the `foo.py` script to produce a `23.mp3` and a `24.mp3`, only one of which is recognized by iTunes as having artwork.

Hmm, so it seems that the problem is in mutagen (or in beets' usage of it) and not in beets itself.

My input files were tagged previously with MusicBrainz Picard, which also uses mutagen. I have just loaded your foo.mp3 in Picard, left all existing tags unchanged and added the image you provided. The result was a ID3v2.3 tagged file with the exact same tags (even in the order of them) as your 23.mp3, but whose album art is recognised in iTunes. I can't provide a script as I used Picard's UI, but here is its output file: [23_picard.mp3](https://dl.dropboxusercontent.com/u/6004950/23_picard.mp3).

I am not very fluent in Python, but here seems to be the place where APIC frames are being added in Picard: [id3.py](https://github.com/musicbrainz/picard/blob/4d4b762514aa2ccee97ebeed27acb03185f34368/picard/formats/id3.py).

Thanks for doing the investigation! It looks like this is related to the text encoding on the APIC frame. We encode the string as UTF-8 (i.e., "encoding 3" according the ID3v2.4). But UTF-8 was only added to the standard in v2.4 ([c.f.](https://en.wikipedia.org/wiki/ID3#ID3v2)), so iTunes seems to be rejecting the entire APIC because we've specified an invalid encoding for the format.

I suppose this means we need to encoding the string differently when targeting v2.3. :unamused: 

Alas, the fix for this will unfortunately be quite involved. The current MediaFile architecture doesn't let us adapt the way frames are constructed based on object's flags; this is currently abstracted too far away. We will need to do some heavy refactoring to fix this right. Sorry if this doesn't happen immediately!

---

The underlying technical issue: StorageStyle `deserialize` methods don't have access to anything but the value they need to serialize; they need access to the MediaFile that they're serializing for to make this work. This will require threading the MediaFile through a couple of layers of indirection (`__set__` on the descriptors, followed by `serialize` itself; not to mention `store` the symmetric `__get__`/`fetch`/`deserialize` progression). This interface is scattered throughout the code and will therefore require lots of local changes to fix.

It might be a good time to rethink this architecture for better maintainability.

It's I who should be thanking you for looking into the issue so quickly!

I understand that fixing the bug right will take time. Meanwhile, is there some dirty hack that I (and possibly others) can do on my side in order to get this to work, like hardcoding the UTF-16 encoding setting somewhere in the code? It doesn't matter if it breaks ID3v2.4 tagging, as I could easily revert the hack if I wanted later.

Sure—if you want to get down and dirty with the Python, you can replace `frame.encoding = 3` with `frame.encoding = 0` in `mediafile.py`. This should work fine, as beets currently always uses the empty string for the description anyway.

Thanks, it worked. Keep up the good work, I hope you can find a nice way to refactor the architecture in order to fix this later :wink: 

If you pass unicode to mutagen it should do the right thing when saving as 2.3 (utf-16 instead of utf-8)

Good to know, @lazka; thanks. I think our problem is that we're setting the encoding flag incorrectly (so even if the encoding is right, iTunes thinks it's wrong).

The encoding flag will also be "downgraded" to utf16 (encoding=1) behind the scenes if saved as v2.3. Maybe iTunes doesn't like that either.

Aha, thanks for clarifying.

I did a little more digging and you're right—Mutagen is correctly encoding the data using ID3 encoding 1 (UTF-16). Alas, iTunes seems to want the encoding to be 0 (ISO-8859-1). Even more perplexingly, even if you just lie and set the frame's encoding to 0 after using UTF-16, iTunes seems to be happy. (I think I have this right; there's some chance I have the actual encoding wrong.)

The Jaikoz people seem to have noticed this too: http://blog.jthink.net/2013/08/solving-bug-in-itunes-that-prevents.html

Did some testing with macosx 10.9.4:

![results_10 9 4](https://cloud.githubusercontent.com/assets/991986/4975527/cc191fbe-68d9-11e4-8a20-ff7a7c3291f1.png)

Results:
- utf16be doesn't work at all.
- utf16 only if not empty
- nonascii was "öäü"; I couldn't find a place where it displays it, so it might be broken with other strings and I was just lucky.

Conclusion:
- `desc=u" ", encoding=3` should work for both versions
- mutagen might do something stupid when writing empty utf16 values, I need to check that.

Wow—that's quite a thorough sweep. Thanks for doing the investigation! I'm surprised that even a whitespace desc field makes a difference. I'll do some fiddling on my end to reproduce.

Adding an extra image to the files with iTunes:

```
v23_LATIN1_empty.mp3: all the same
v24_UTF16_nonascii.mp3: all the same
v24_UTF16_space.mp3: all the same
v23_UTF16_nonascii.mp3: all the same
v23_UTF16_space.mp3: all the same
v24_LATIN1_empty.mp3: all the same

v23_UTF16_empty.mp3:  switched to latin1, corrupt image
v24_UTF16_empty.mp3: switched to latin1, corrupt image
v24_UTF16BE_empty.mp3:  switched to latin1, corrupt image

v24_UTF16BE_nonascii.mp3: switched to utf16, corrupt image
v24_UTF16BE_space.mp3: switched to utf16, corrupt image

v24_UTF8_nonascii.mp3: switched to latin1, desc gone, image ok
v24_UTF8_space.mp3: switched to latin1, desc gone, image ok
v24_UTF8_empty.mp3: switched to latin1, image ok

v24_LATIN1_nonascii.mp3: switched to utf16, image ok
v24_LATIN1_space.mp3: switched to utf16, image ok
v23_LATIN1_nonascii.mp3: switched to utf16, image ok
v23_LATIN1_space.mp3:  switched to utf16, image ok
```
- iTunes will add a second frame even with the same `desc` (not sure what to do there in mutagen..)
- It seems to have a valid latin1 reading logic, converting to utf16, but doesn't convert frames with an empty `desc`, which might explain why they never noticed the utf16 bug.
- utf16be is kinda there, the desc gets converted correctly, but the image is corrupt.
- utf8 throws aways desc but parses correctly

Given what iTunes converts to and what it doesn't corrupt, the best options for compat are:
- utf16 except empty strings
- latin1 for everything that fits in it
- never utf8/utf16be

Wow; that is convoluted. It seems like a good policy on beets' side might be to always include a `desc` string (even just something generic like "image") to avoid requiring a different encoding.

Hi @sampsyo! Is there any update about this issue? For now I'm applying the hack you mentioned [here](#issuecomment-52378181) every time I update beets, but it would be nice if this had a proper solution.

From what I understood from the discussion a revamp of the architecture would be the most elegant way to do that, but isn't there anything you can do to allow users to manually override the encoding in the configuration for now, even if it seems somewhat strange / specific for this problem?

Sorry, no update to speak of. Contributions would be very welcome here! (There's no shortcut, alas.)


Resolving patch:
diff --git a/beets/mediafile.py b/beets/mediafile.py
--- a/beets/mediafile.py
+++ b/beets/mediafile.py
@@ -920,7 +920,16 @@ def serialize(self, image):
         frame.data = image.data
         frame.mime = image.mime_type
         frame.desc = image.desc or u''
-        frame.encoding = 3  # UTF-8 encoding of desc
+
+        # For compatibility with OS X/iTunes prefer latin-1 if possible.
+        # See issue #899
+        try:
+            frame.desc.encode("latin-1")
+        except UnicodeEncodeError:
+            frame.encoding = mutagen.id3.Encoding.UTF16
+        else:
+            frame.encoding = mutagen.id3.Encoding.LATIN1
+
         frame.type = image.type_index
         return frame
 


Test patch:
diff --git a/test/test_mediafile_edge.py b/test/test_mediafile_edge.py
--- a/test/test_mediafile_edge.py
+++ b/test/test_mediafile_edge.py
@@ -19,6 +19,7 @@
 
 import os
 import shutil
+import mutagen.id3
 
 from test import _common
 from test._common import unittest
@@ -375,30 +376,30 @@ def test_v23_on_non_mp3_is_noop(self):
         finally:
             self._delete_test()
 
-    def test_v24_image_encoding(self):
-        mf = self._make_test(id3v23=False)
-        try:
-            mf.images = [beets.mediafile.Image(b'test data')]
-            mf.save()
-            frame = mf.mgfile.tags.getall('APIC')[0]
-            self.assertEqual(frame.encoding, 3)
-        finally:
-            self._delete_test()
+    def test_image_encoding(self):
+        """For compatibility with OS X/iTunes.
 
-    @unittest.skip("a bug, see #899")
-    def test_v23_image_encoding(self):
-        """For compatibility with OS X/iTunes (and strict adherence to
-        the standard), ID3v2.3 tags need to use an inferior text
-        encoding: UTF-8 is not supported.
+        See https://github.com/beetbox/beets/issues/899#issuecomment-62437773
         """
-        mf = self._make_test(id3v23=True)
-        try:
-            mf.images = [beets.mediafile.Image(b'test data')]
-            mf.save()
-            frame = mf.mgfile.tags.getall('APIC')[0]
-            self.assertEqual(frame.encoding, 1)
-        finally:
-            self._delete_test()
+
+        for v23 in [True, False]:
+            mf = self._make_test(id3v23=v23)
+            try:
+                mf.images = [
+                    beets.mediafile.Image(b'data', desc=u""),
+                    beets.mediafile.Image(b'data', desc=u"foo"),
+                    beets.mediafile.Image(b'data', desc=u"\u0185"),
+                ]
+                mf.save()
+                apic_frames = mf.mgfile.tags.getall('APIC')
+                encodings = dict([(f.desc, f.encoding) for f in apic_frames])
+                self.assertEqual(encodings, {
+                    u"": mutagen.id3.Encoding.LATIN1,
+                    u"foo": mutagen.id3.Encoding.LATIN1,
+                    u"\u0185": mutagen.id3.Encoding.UTF16,
+                })
+            finally:
+                self._delete_test()
 
 
 def suite():