Repository: beetbox/beets

Issue:
lyrics: Use hyphens in place spaces, not underscores, to construct URL for Musixmatch
Hi,

I try to get lyrics for some songs. I know they exist on musixmatch (i.e https://www.musixmatch.com/lyrics/Dead-Astronauts/Part-Two) but beets says : 

> lyrics: lyrics not found: Dead Astronauts - Constellations - Part Two



Discussion:
Can you please include your configuration and the verbose output?

Config.yaml

```
directory: /mnt/MyBook/Musique/beet
library: /mnt/MyBook/Musique/beets-database.db
import:
    link: yes
    write: yes
paths:
    singleton: $artist/Non-Album/$title
plugins: inline web lyrics
lyrics:
    sources: musixmatch

```

Ouput : 

```
beet -v lyrics Part Two
user configuration: /home/pi/.config/beets/config.yaml
data directory: /home/pi/.config/beets
plugin paths:
Sending event: pluginload
library database: /mnt/MyBook/Musique/beets-database.db
library directory: /mnt/MyBook/Musique/beet
Sending event: library_opened
lyrics: failed to fetch: https://www.musixmatch.com/lyrics/Dead_Astronauts/Part_Two (404)
lyrics: lyrics not found: Dead Astronauts - Constellations - Part Two
Sending event: cli_exit
```

So it seems that spaces should be replaced by '-' instead of '_'

Aha; that definitely does help explain it!

Can you tell whether MusixMatch is working for _any_ songs with spaces in them (i.e., maybe it's inconsistent)? Or should we really be using dashes in place of spaces for _every single_ song?

Sadly, a cursory search found a song that uses dashes in the URL: https://www.musixmatch.com/lyrics/The-Beatles/Hey-Jude-2 :cry:

EDIT: Excuse the lapse in judgment; this _confirms_ your findings, of course. Ignore this comment. :flushed:

I tried to replace '_' by '-' on line 206 in lyrics.py, but I still got a 404. Copy/Pasting the link into my browser works, however.

I don't know python at all, but I don't understand why a simple :
`requests.get("https://www.musixmatch.com/lyrics/Dead-Astronauts/Part-Two")` 
always returns a 404, but this in shell
`curl https://www.musixmatch.com/lyrics/Dead-Astronauts/Part-Two`
 works correctly...

Whoa, weird. Maybe they're explicitly blocking automated requests, and checking for the `requests` User-Agent. Do you know whether using a different user agent for one or the other makes them behave more like each other?

Well I don't understand. 

I can get a 200 response for Musixmatch with `requests.get()` on my main computer (Windows 10), but not on my raspberry (Same network) ONLY for musixmatch. Every other URLs I tried work well... It's driving me crazy !

dashes and underscores don't seem to be the problem, musixmatch redirects.

EDIT : So... changing user-agent makes it work. I removed  CPython/2.7.9

Wow; very strange. We'll have to leave the User-Agent thing a (partial) mystery, but let's consider this a bug to swap underscores and dashes (after doing more investigation—are underscores _always_ wrong)?

So after doing some looking, it seems they just use hyphens.

I wrote this quick snippet and ran it on the the ["Top Lyrics"](https://www.musixmatch.com/explore) page:

``` js
delete console.log;

Array.prototype.slice.call(document.getElementsByClassName("title")).forEach(function(element) {
    var href = element.href;

    if (href.indexOf("_") != -1) {
        console.log("Contains underscore: " + href);
    }
});
```

And there was no output, so I assume that no lyric URLs contain underscores.

So after looking at the code base, I'm not entirely sure how we could customise the replacement. We're currently using `@classmethod` to define `_encode`.

One way we could do it is like this, however it feels a bit 'hacky':

``` python
class SymbolsReplaced(Backend):
    REPLACEMENTS = {
        r'\s+': '_',
        '<': 'Less_Than',
        '>': 'Greater_Than',
        '#': 'Number_',
        r'[\[\{]': '(',
        r'[\[\{]': ')'
    }

    @classmethod
    def _encode(cls, s):
        for old, new in cls.REPLACEMENTS.iteritems():
            s = re.sub(old, new, s)

        return super(SymbolsReplaced, cls)._encode(s)

class MusiXmatch(SymbolsReplaced):
    REPLACEMENTS = dict(SymbolsReplaced.REPLACEMENTS, **{
        r'\s+': '-'
    })

    URL_PATTERN = 'https://www.musixmatch.com/lyrics/%s/%s'

    def fetch(self, artist, title):
        url = self.build_url(artist, title)
        html = self.fetch_url(url)
        if not html:
            return
        lyrics = extract_text_between(html,
                                      '"lyrics_body":', '"lyrics_language":')
        return lyrics.strip(',"').replace('\\n', '\n')
```

After testing this, I seem to have found a new problem with the `MusiXmatch` class, and that is that the `lyrics` variable has 0 length even when the page exists. It seems that they may have updated their design, and the parameters passed to `extract_text_between` are out of date.

Changing the line to this seems to fix it:

``` python
        lyrics = extract_text_between(html,
                                      '"body":', '"language":')
```


Resolving patch:
diff --git a/beetsplug/lyrics.py b/beetsplug/lyrics.py
--- a/beetsplug/lyrics.py
+++ b/beetsplug/lyrics.py
@@ -200,18 +200,28 @@ def fetch(self, artist, title):
 
 
 class SymbolsReplaced(Backend):
+    REPLACEMENTS = {
+        r'\s+': '_',
+        '<': 'Less_Than',
+        '>': 'Greater_Than',
+        '#': 'Number_',
+        r'[\[\{]': '(',
+        r'[\[\{]': ')'
+    }
+
     @classmethod
     def _encode(cls, s):
-        s = re.sub(r'\s+', '_', s)
-        s = s.replace("<", "Less_Than")
-        s = s.replace(">", "Greater_Than")
-        s = s.replace("#", "Number_")
-        s = re.sub(r'[\[\{]', '(', s)
-        s = re.sub(r'[\]\}]', ')', s)
+        for old, new in cls.REPLACEMENTS.iteritems():
+            s = re.sub(old, new, s)
+
         return super(SymbolsReplaced, cls)._encode(s)
 
 
 class MusiXmatch(SymbolsReplaced):
+    REPLACEMENTS = dict(SymbolsReplaced.REPLACEMENTS, **{
+        r'\s+': '-'
+    })
+
     URL_PATTERN = 'https://www.musixmatch.com/lyrics/%s/%s'
 
     def fetch(self, artist, title):
@@ -220,7 +230,7 @@ def fetch(self, artist, title):
         if not html:
             return
         lyrics = extract_text_between(html,
-                                      '"lyrics_body":', '"lyrics_language":')
+                                      '"body":', '"language":')
         return lyrics.strip(',"').replace('\\n', '\n')
 
 


Test patch: