Repository: beetbox/beets

Issue:
lastimport: Unable to read page
Hi everyone,

Is anyone else having issues with lastimport at all?

```
lastimport: Fetching last.fm library for @mysticg
lastimport: Querying page #1...
lastimport: ERROR: unable to read page #1
lastimport: Retrying page #1... (1/3 retry)
lastimport: ERROR: unable to read page #1
lastimport: Retrying page #1... (2/3 retry)
lastimport: ERROR: unable to read page #1
lastimport: Retrying page #1... (3/3 retry)
lastimport: ... done!
lastimport: finished processing 1 song pages
lastimport: 0 unknown play-counts
lastimport: 0 play-counts imported
```

My config as per the manual is just

```
lastfm:
    user: mysticg
```

Any suggestions?  I'm wondering if it might be related to the Last.fm site updates?



Discussion:
Thanks for reporting. It does look like something is broken in the Last.fm API. It's reporting an "Invalid Method" error for any call to `library.getTracks`. There's no indication in the API documentation that the method has gone away: http://www.last.fm/api/show/library.getTracks

Google hasn't turned anything up. Maybe this is a bug that they'll eventually fix? Or maybe their API is just being neglected, which would be bad news? Perhaps we should post on their developer forms to get more information.

I've posted on their getsatisfaction portal.  It looks like they may have messed up a whole heap of stuff... 

https://getsatisfaction.com/lastfm/topics/api-known-issues?topic-reply-list%5Bsettings%5D%5Bfilter_by%5D=all&topic-reply-list%5Bsettings%5D%5Breply_id%5D=16048253#reply_16048253

Tragic. We really need a legitimate alternative to Last.fm.

Seems like they messed up their whole api with the page relaunch. But they keep telling that all features will be back once all the migration is done.

I tried to get the lastimport plugin working for the past few days with the same issue mentioned above.  It seems a bit hit and miss with other applications that use the api.  For example, I was able to use lastexport.py (from the libre.fm conversion tools) to pull in my history in a large exported_tracks.txt text file.  Nothing else seemed to work.  At least I have a backup of my play history now if nothing else.  I'll wait, but if I get really bored I may just decide to write a small perl or python script to convert that exported_tracks data into the beets database.  sqllite isn't hard to fool with if it comes down to it.

So, I had a closer look into this whole thing, and the problem is that last.fm has for some reason removed the api-method that is used in this plugin, which is http://www.last.fm/api/show/library.getTracks. When calling it, an error "Invalid Method - No method with that name in this package" is thrown back. library.getartists, which is also in the last.fm API documentation, works fine. So bottom line, last.fm's API is currently just a mess, and nobody really knows when or if they think to ever make it work again. I haven't found a different method that could replace library.getTracks, because the API documentation is not really up to date, to say the least. IMHO, everybody needs an alternative to last.fm


Resolving patch:
diff --git a/beetsplug/lastimport.py b/beetsplug/lastimport.py
--- a/beetsplug/lastimport.py
+++ b/beetsplug/lastimport.py
@@ -16,7 +16,8 @@
 from __future__ import (division, absolute_import, print_function,
                         unicode_literals)
 
-import requests
+import pylast
+from pylast import TopItem, _extract, _number
 from beets import ui
 from beets import dbcore
 from beets import config
@@ -52,6 +53,63 @@ def func(lib, opts, args):
         return [cmd]
 
 
+class CustomUser(pylast.User):
+    """ Custom user class derived from pylast.User, and overriding the
+    _get_things method to return MBID and album. Also introduces new
+    get_top_tracks_by_page method to allow access to more than one page of top
+    tracks.
+    """
+    def __init__(self, *args, **kwargs):
+        super(CustomUser, self).__init__(*args, **kwargs)
+
+    def _get_things(self, method, thing, thing_type, params=None,
+                    cacheable=True):
+        """Returns a list of the most played thing_types by this thing, in a
+        tuple with the total number of pages of results. Includes an MBID, if
+        found.
+        """
+        doc = self._request(
+            self.ws_prefix + "." + method, cacheable, params)
+
+        toptracks_node = doc.getElementsByTagName('toptracks')[0]
+        total_pages = int(toptracks_node.getAttribute('totalPages'))
+
+        seq = []
+        for node in doc.getElementsByTagName(thing):
+            title = _extract(node, "name")
+            artist = _extract(node, "name", 1)
+            mbid = _extract(node, "mbid")
+            playcount = _number(_extract(node, "playcount"))
+
+            thing = thing_type(artist, title, self.network)
+            thing.mbid = mbid
+            seq.append(TopItem(thing, playcount))
+
+        return seq, total_pages
+
+    def get_top_tracks_by_page(self, period=pylast.PERIOD_OVERALL, limit=None,
+                               page=1, cacheable=True):
+        """Returns the top tracks played by a user, in a tuple with the total
+        number of pages of results.
+        * period: The period of time. Possible values:
+          o PERIOD_OVERALL
+          o PERIOD_7DAYS
+          o PERIOD_1MONTH
+          o PERIOD_3MONTHS
+          o PERIOD_6MONTHS
+          o PERIOD_12MONTHS
+        """
+
+        params = self._get_params()
+        params['period'] = period
+        params['page'] = page
+        if limit:
+            params['limit'] = limit
+
+        return self._get_things(
+            "getTopTracks", "track", pylast.Track, params, cacheable)
+
+
 def import_lastfm(lib, log):
     user = config['lastfm']['user'].get(unicode)
     per_page = config['lastimport']['per_page'].get(int)
@@ -73,23 +131,19 @@ def import_lastfm(lib, log):
                  '/{}'.format(page_total) if page_total > 1 else '')
 
         for retry in range(0, retry_limit):
-            page = fetch_tracks(user, page_current + 1, per_page)
-            if 'tracks' in page:
-                # Let us the reveal the holy total pages!
-                page_total = int(page['tracks']['@attr']['totalPages'])
-                if page_total < 1:
-                    # It means nothing to us!
-                    raise ui.UserError('Last.fm reported no data.')
-
-                track = page['tracks']['track']
-                found, unknown = process_tracks(lib, track, log)
+            tracks, page_total = fetch_tracks(user, page_current + 1, per_page)
+            if page_total < 1:
+                # It means nothing to us!
+                raise ui.UserError('Last.fm reported no data.')
+
+            if tracks:
+                found, unknown = process_tracks(lib, tracks, log)
                 found_total += found
                 unknown_total += unknown
                 break
             else:
                 log.error('ERROR: unable to read page #{0}',
                           page_current + 1)
-                log.debug('API response: {}', page)
                 if retry < retry_limit:
                     log.info(
                         'Retrying page #{0}... ({1}/{2} retry)',
@@ -107,14 +161,30 @@ def import_lastfm(lib, log):
 
 
 def fetch_tracks(user, page, limit):
-    return requests.get(API_URL, params={
-        'method': 'library.gettracks',
-        'user': user,
-        'api_key': plugins.LASTFM_KEY,
-        'page': bytes(page),
-        'limit': bytes(limit),
-        'format': 'json',
-    }).json()
+    """ JSON format:
+        [
+            {
+                "mbid": "...",
+                "artist": "...",
+                "title": "...",
+                "playcount": "..."
+            }
+        ]
+    """
+    network = pylast.LastFMNetwork(api_key=config['lastfm']['api_key'])
+    user_obj = CustomUser(user, network)
+    results, total_pages =\
+        user_obj.get_top_tracks_by_page(limit=limit, page=page)
+    return [
+        {
+            "mbid": track.item.mbid if track.item.mbid else '',
+            "artist": {
+                "name": track.item.artist.name
+            },
+            "name": track.item.title,
+            "playcount": track.weight
+        } for track in results
+    ], total_pages
 
 
 def process_tracks(lib, tracks, log):
@@ -124,7 +194,7 @@ def process_tracks(lib, tracks, log):
     log.info('Received {0} tracks in this page, processing...', total)
 
     for num in xrange(0, total):
-        song = ''
+        song = None
         trackid = tracks[num]['mbid'].strip()
         artist = tracks[num]['artist'].get('name', '').strip()
         title = tracks[num]['name'].strip()
@@ -140,19 +210,8 @@ def process_tracks(lib, tracks, log):
                 dbcore.query.MatchQuery('mb_trackid', trackid)
             ).get()
 
-        # Otherwise try artist/title/album
-        if not song:
-            log.debug(u'no match for mb_trackid {0}, trying by '
-                      u'artist/title/album', trackid)
-            query = dbcore.AndQuery([
-                dbcore.query.SubstringQuery('artist', artist),
-                dbcore.query.SubstringQuery('title', title),
-                dbcore.query.SubstringQuery('album', album)
-            ])
-            song = lib.items(query).get()
-
         # If not, try just artist/title
-        if not song:
+        if song is None:
             log.debug(u'no album match, trying by artist/title')
             query = dbcore.AndQuery([
                 dbcore.query.SubstringQuery('artist', artist),
@@ -161,7 +220,7 @@ def process_tracks(lib, tracks, log):
             song = lib.items(query).get()
 
         # Last resort, try just replacing to utf-8 quote
-        if not song:
+        if song is None:
             title = title.replace("'", u'\u2019')
             log.debug(u'no title match, trying utf-8 single quote')
             query = dbcore.AndQuery([
@@ -170,7 +229,7 @@ def process_tracks(lib, tracks, log):
             ])
             song = lib.items(query).get()
 
-        if song:
+        if song is not None:
             count = int(song.get('play_count', 0))
             new_count = int(tracks[num]['playcount'])
             log.debug(u'match: {0} - {1} ({2}) '


Test patch: