Repository: beetbox/beets

Issue:
discogs: IndexError: list index out of range
### Problem

I'm trying to import this album :
https://www.discogs.com/Para-One-Ciel-Ether-Con-Citoyen-Crazy/master/690105

while doing a `beet import Unsorted/New\ Stuff` :

```
/data/Music/Unsorted/New Stuff/Para One - 2003 - Paraone - Iris - Le Sept - Flynt - Lyricson (Maxi) [CD - FLAC] (6 items)
Finding tags for album "Paraone - Iris - Le sept - Flynt - Lyricson - Paraone - Iris - Le Sept - Flynt - Lyricson".
Candidates:
1. The Kenny Clarke - Francy Boland Big Band - Paris Jazz Concert: TNP - Oct. 29th, 1969, Part 2 (35.9%) (tracks, artist, album, ...) (CD, 2002, DE, LaserLight Digital)
2. A - Z Consolidated - A - Z Consolidated (33.8%) (tracks, album, artist)
3. Μίκης Θεοδωράκης - Βασίλης Τσιτσάνης - Σταμάτης Κόκοτας - Θεοδωράκης - Τσιτσάνης - Κόκοτας (29.3%) (tracks, missing tracks, album, ...) (12" Vinyl, 1985, GR, Μίνως Μάτσας & Υιός Α.Ε.)
4. Pirkko Mannola - Brita Koivunen - Vieno Kekkonen - Pirkko - Brita - Vieno (20.2%) (missing tracks, tracks, artist, ...) (CD, 1989, FI, Finnlevy)
5. _ - _ (7.7%) (missing tracks, tracks, album, ...)
# selection (default 1), Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort? I
Enter release ID: 690105
Traceback (most recent call last):
  File "/home/cesar/.local/bin/beet", line 11, in <module>
    sys.exit(main())
  File "/home/cesar/.local/lib/python2.7/site-packages/beets/ui/__init__.py", line 1250, in main
    _raw_main(args)
  File "/home/cesar/.local/lib/python2.7/site-packages/beets/ui/__init__.py", line 1240, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/home/cesar/.local/lib/python2.7/site-packages/beets/ui/commands.py", line 967, in import_func
    import_files(lib, paths, query)
  File "/home/cesar/.local/lib/python2.7/site-packages/beets/ui/commands.py", line 944, in import_files
    session.run()
  File "/home/cesar/.local/lib/python2.7/site-packages/beets/importer.py", line 320, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/home/cesar/.local/lib/python2.7/site-packages/beets/util/pipeline.py", line 301, in run
    out = self.coro.send(msg)
  File "/home/cesar/.local/lib/python2.7/site-packages/beets/util/pipeline.py", line 160, in coro
    task = func(*(args + (task,)))
  File "/home/cesar/.local/lib/python2.7/site-packages/beets/importer.py", line 1285, in user_query
    task.choose_match(session)
  File "/home/cesar/.local/lib/python2.7/site-packages/beets/importer.py", line 779, in choose_match
    choice = session.choose_match(self)
  File "/home/cesar/.local/lib/python2.7/site-packages/beets/ui/commands.py", line 755, in choose_match
    task.items, search_ids=search_id.split()
  File "/home/cesar/.local/lib/python2.7/site-packages/beets/autotag/match.py", line 407, in tag_album
    search_cands.extend(hooks.albums_for_id(search_id))
  File "/home/cesar/.local/lib/python2.7/site-packages/beets/autotag/hooks.py", line 541, in albums_for_id
    plugin_albums = plugins.album_for_id(album_id)
  File "/home/cesar/.local/lib/python2.7/site-packages/beets/plugins.py", line 373, in album_for_id
    res = plugin.album_for_id(album_id)
  File "/home/cesar/.local/lib/python2.7/site-packages/beetsplug/discogs.py", line 188, in album_for_id
    return self.get_album_info(result)
  File "/home/cesar/.local/lib/python2.7/site-packages/beetsplug/discogs.py", line 225, in get_album_info
    result.data['formats'][0].get('descriptions', [])) or None
IndexError: list index out of range
```

### Setup

```
$ uname -a
Linux debian 4.7.0-1-amd64 #1 SMP Debian 4.7.6-1 (2016-10-07) x86_64 GNU/Linux
```
```
$ beet version
beets version 1.4.1
Python version 2.7.12+
plugins: discogs, lastgenre
```
```
$ beet config
directory: /data/Music/beets

import:
    link: yes
    write: no
    log: /data/Music/.beets_import.log
library: /data/Music/beetslibrary.blb

plugins: discogs lastgenre
lastgenre:
    count: 1
    source: album
    prefer_specific: no
    force: yes
    min_weight: 10
    auto: yes
    whitelist: yes
    separator: ', '
    fallback:
    canonical: no
discogs:
    tokenfile: discogs_token.json
    apikey: REDACTED
    apisecret: REDACTED
    source_weight: 0.5
```



Discussion:
It works fine if I use one of the actual release instead of the "master". Maybe that's what I was supposed to do. Still it probably shouldn't crash.
that's interesting because there haven't been any changes in something obviously related since 1.4.1 was release.  You are using 1.4.1 release (vs 1.4.1 in master)
I mean release and master on discogs :

https://www.discogs.com/Para-One-Ciel-Ether-Con-Citoyen-Crazy/master/690105
https://www.discogs.com/Para-One-Ciel-Ether-Con-Citoyen-Crazy/release/1261580
Thanks for reporting! It looks like we need to relax an assumption about what the Discogs API returns.
> It works fine if I use one of the actual release instead of the "master". Maybe that's what I was supposed to do. Still it probably shouldn't crash.

Actually, on your scenario, when you specify the ID manually (at `Enter release ID: 690105`) the plugin does not retrieve the *master* release of the Para One album - it retrieves the discogs *release* with that ID, which turns out to be https://www.discogs.com/release/690105. The discogs plugin at beets makes queries only to the discogs release API, which I believe it is by design (as a "master release", on discogs terminology, is basically a mechanism for grouping releases together as far as I can see - making sense not to use master releases at all).

> Thanks for reporting! It looks like we need to relax an assumption about what the Discogs API returns.

Absolutely - luckily (?) enough, the Red Hot Chili Peppers release seems to be missing the `format` information, which the plugin is not handling gracefully at all. `DiscogsPlugin.get_album_info()` seems to be making some assumptions about other fields as well that might be prone to causing similar crashes - perhaps it would be worth to try to revise those too?
Yes, good point—it would make sense to program "defensively" here, since it's clearly not obvious which fields are guaranteed to exist and which might be missing. For example, the `country` field is treated as optional:

```py
country = result.data.get('country')
```

Perhaps we should do that sort of thing elsewhere too.

Resolving patch:
diff --git a/beetsplug/discogs.py b/beetsplug/discogs.py
--- a/beetsplug/discogs.py
+++ b/beetsplug/discogs.py
@@ -205,11 +205,27 @@ def get_albums(self, query):
             self._log.debug(u"Communication error while searching for {0!r}",
                             query, exc_info=True)
             return []
-        return [self.get_album_info(release) for release in releases[:5]]
+        return [album for album in map(self.get_album_info, releases[:5])
+                if album]
 
     def get_album_info(self, result):
         """Returns an AlbumInfo object for a discogs Release object.
         """
+        # Explicitly reload the `Release` fields, as they might not be yet
+        # present if the result is from a `discogs_client.search()`.
+        if not result.data.get('artists'):
+            result.refresh()
+
+        # Sanity check for required fields. The list of required fields is
+        # defined at Guideline 1.3.1.a, but in practice some releases might be
+        # lacking some of these fields. This function expects at least:
+        # `artists` (>0), `title`, `id`, `tracklist` (>0)
+        # https://www.discogs.com/help/doc/submission-guidelines-general-rules
+        if not all([result.data.get(k) for k in ['artists', 'title', 'id',
+                                                 'tracklist']]):
+            self._log.warn(u"Release does not contain the required fields")
+            return None
+
         artist, artist_id = self.get_artist([a.data for a in result.artists])
         album = re.sub(r' +', ' ', result.title)
         album_id = result.data['id']
@@ -218,25 +234,36 @@ def get_album_info(self, result):
         # information and leave us with skeleton `Artist` objects that will
         # each make an API call just to get the same data back.
         tracks = self.get_tracks(result.data['tracklist'])
-        albumtype = ', '.join(
-            result.data['formats'][0].get('descriptions', [])) or None
-        va = result.data['artists'][0]['name'].lower() == 'various'
+
+        # Extract information for the optional AlbumInfo fields, if possible.
+        va = result.data['artists'][0].get('name', '').lower() == 'various'
+        year = result.data.get('year')
+        mediums = len(set(t.medium for t in tracks))
+        country = result.data.get('country')
+        data_url = result.data.get('uri')
+
+        # Extract information for the optional AlbumInfo fields that are
+        # contained on nested discogs fields.
+        albumtype = media = label = catalogno = None
+        if result.data.get('formats'):
+            albumtype = ', '.join(
+                result.data['formats'][0].get('descriptions', [])) or None
+            media = result.data['formats'][0]['name']
+        if result.data.get('labels'):
+            label = result.data['labels'][0].get('name')
+            catalogno = result.data['labels'][0].get('catno')
+
+        # Additional cleanups (various artists name, catalog number, media).
         if va:
             artist = config['va_name'].as_str()
-        year = result.data['year']
-        label = result.data['labels'][0]['name']
-        mediums = len(set(t.medium for t in tracks))
-        catalogno = result.data['labels'][0]['catno']
         if catalogno == 'none':
-            catalogno = None
-        country = result.data.get('country')
-        media = result.data['formats'][0]['name']
+                catalogno = None
         # Explicitly set the `media` for the tracks, since it is expected by
         # `autotag.apply_metadata`, and set `medium_total`.
         for track in tracks:
             track.media = media
             track.medium_total = mediums
-        data_url = result.data['uri']
+
         return AlbumInfo(album, album_id, artist, artist_id, tracks, asin=None,
                          albumtype=albumtype, va=va, year=year, month=None,
                          day=None, label=label, mediums=mediums,


Test patch:
diff --git a/test/test_discogs.py b/test/test_discogs.py
--- a/test/test_discogs.py
+++ b/test/test_discogs.py
@@ -20,6 +20,7 @@
 import unittest
 from test import _common
 from test._common import Bag
+from test.helper import capture_log
 
 from beetsplug.discogs import DiscogsPlugin
 
@@ -327,6 +328,29 @@ def test_parse_tracklist_disctitles(self):
         self.assertEqual(d.tracks[2].disctitle, 'MEDIUM TITLE CD2')
         self.assertEqual(len(d.tracks), 3)
 
+    def test_parse_minimal_release(self):
+        """Test parsing of a release with the minimal amount of information."""
+        data = {'id': 123,
+                'tracklist': [self._make_track('A', '1', '01:01')],
+                'artists': [{'name': 'ARTIST NAME', 'id': 321, 'join': ''}],
+                'title': 'TITLE'}
+        release = Bag(data=data,
+                      title=data['title'],
+                      artists=[Bag(data=d) for d in data['artists']])
+        d = DiscogsPlugin().get_album_info(release)
+        self.assertEqual(d.artist, 'ARTIST NAME')
+        self.assertEqual(d.album, 'TITLE')
+        self.assertEqual(len(d.tracks), 1)
+
+    def test_parse_release_without_required_fields(self):
+        """Test parsing of a release that does not have the required fields."""
+        release = Bag(data={}, refresh=lambda *args: None)
+        with capture_log() as logs:
+            d = DiscogsPlugin().get_album_info(release)
+
+        self.assertEqual(d, None)
+        self.assertIn('Release does not contain the required fields', logs[0])
+
 
 def suite():
     return unittest.TestLoader().loadTestsFromName(__name__)