Repository: beetbox/beets

Issue:
lyrics: Strip \u2005 (four-per-em space) in lyrics
The current lyrics plugin returns lyrics from Genius which includes the unicode character u2005 (four-per-em space) rather than replacing this with regular space. This results in squares in the lyrics in players like Foobar.

### Problem

Example lyrics:

``` sh
$ beet -vv lyrics -f joshua radin only a wave better days
user configuration: /home/xxxx/.config/beets/config.yaml
data directory: /home/xxxx/.config/beets
plugin paths: 
Sending event: pluginload
inline: adding item field disc_and_track
inline: adding item field artist_and_title
inline: adding item field original_date
inline: adding item field album_year
inline: adding album field totaltracks
inline: adding album field album_ends_with_ep
lyrics: Disabling google source: no API key configured.
ImageMagick version check failed: [Errno 2] No such file or directory: 'magick'
artresizer: method is (2, (6, 9, 10), True)
library database: /home/xxxx/.config/beets/beetslibrary.db
library directory: /media/music/beets2
Sending event: library_opened
lyrics: failed to fetch: https://www.musixmatch.com/lyrics/Joshua-Radin/Only-A-Wave-%28Better-Days%29 (404)
lyrics: got lyrics from backend: Genius
lyrics: fetched lyrics: JOSHUA RADIN - Here, Right Now - 03. Only a Wave (Better Days) (3:58)
Sending event: write
Sending event: after_write
importadded: Write of item '/media/music/beets2/Album/G-J/Joshua Radin/Joshua Radin - Here, Right Now [2019]/Joshua Radin - 03 - Only a Wave (Better Days).mp3', selected item.added=1604661728.1660378
Sending event: database_change
Sending event: cli_exit
```

Led to this problem:
several u2005 characters in the first paragraph.
Different song but same problem:
![image](https://user-images.githubusercontent.com/40003252/98451234-c6618900-213b-11eb-9c2a-7bbab021bea9.png)

### Setup

* OS: Linux Mint 20
* Python version: 3.8.5
* beets version: 1.5.0
* Turning off plugins made problem go away (yes/no): no

My configuration (output of `beet config`) is:

```yaml
lyrics:
    bing_lang_from: []
    auto: no
    on_import: yes
    processes: 10
    force: no
    fallback: ''
    bing_client_secret: REDACTED
    bing_lang_to:
    google_API_key: REDACTED
    google_engine_ID: REDACTED
    genius_api_key: REDACTED
    local: no
    sources: [google, musixmatch, genius]
library: /home/xxxx/.config/beets/beetslibrary.db
directory: /media/music/beets2

import:
    write: yes
    copy: yes
    move: yes
    link: no
    hardlink: no
    resume: ask
    incremental: no
    incremental_skip_later: no
    from_scratch: no
    quiet_fallback: skip
    none_rec_action: ask
    timid: no
    log: /home/xxxx/.config/beets/beetslog.txt
    autotag: yes
    quiet: no
    singletons: no
    default_action: apply
    languages: []
    detail: no
    flat: no
    group_albums: no
    pretend: no
    search_ids: []
    duplicate_action: ask
    bell: no
    set_fields: {}
    art: yes

clutter:
- Thumbs.DB
- .DS_Store
- desktop.ini
- '*.m3u'
- '*.nfo'
- '*.pls'
- '*.torrent'
ignore:
- .*
- '*~'
- System Volume Information
- lost+found
- cover.jpg
ignore_hidden: yes

replace:
    '[\\/]': '-'
    ^\.: _
    '[\x00-\x1f]': _
    '[<>\?\*\|]': _
    '[:]': ' -'
    \.$: _
    \s+$: ''
    ^\s+: ''
    ^-: _
    '[\u2018\u2019]': ''''
    '[\u201c\u201d]': '"'
    '"': ''''
path_sep_replace: '-'
drive_sep_replace: '-'
asciify_paths: yes
art_filename: folder
max_filename_length: 0

aunique:
    keys: albumartist album
    disambiguators: albumtype year label catalognum albumdisambig releasegroupdisambig
    bracket: '[]'

overwrite_null:
    album: []
    track: []

plugins:
- fetchart
- lyrics
- lastgenre
- embedart
- bucket
- the
- chroma
- ftintitle
- discogs
- scrub
- inline
- fromfilename
- edit
- importadded
- duplicates
- replaygain
pluginpath: []
threaded: yes
timeout: 5.0
per_disc_numbering: yes
verbose: 0
terminal_encoding: utf8
original_date: yes
artist_credit: no
id3v23: no
va_name: Various Artists

ui:
    terminal_width: 80
    length_diff_thresh: 10.0
    color: yes
    colors:
        text_success: green
        text_warning: yellow
        text_error: red
        text_highlight: red
        text_highlight_minor: lightgray
        action_default: turquoise
        action: blue

format_item: '%upper{$artist} - $album - $track. $title ($length)'
format_album: '%upper{$albumartist} - $album [$year] ($totaltracks)'
time_format: '%Y-%m-%d %H:%M:%S'
format_raw_length: no

sort_album: albumartist+ album+
sort_item: artist+ album+ disc+ track+
sort_case_insensitive: yes
item_fields:
    disc_and_track: f'{disc:02}-{track:02}' if disctotal > 1 else f'{track:02}'
    artist_and_title: f'{artist} - {title}' if artist != albumartist else title
    original_date: '''-''.join(filter(None, (original_year and f''{original_year:04}'', original_month and f''{original_month:02}'', original_day and f''{original_day:02}'')))'
    album_year: f'{year:04}'
album_fields:
    totaltracks: str(items[0].tracktotal) + ' tracks'
    album_ends_with_ep: 1 if str({album})[-2:] == "EP" else 0

paths:
    default: Album/%bucket{%the{$albumartist},alpha}/%the{$albumartist}/$albumartist - $album%aunique{} [$year]/$albumartist - $disc_and_track - $title
    albumtype:ep: Album/%bucket{%the{$albumartist},alpha}/%the{$albumartist}/$albumartist - $album%aunique{}%if{$album_ends_with_ep,, EP} [$year]/$albumartist - $disc_and_track - $title
    singleton: Non-Album/%bucket{%the{$artist}}/$artist - $title
    albumtype:soundtrack: Soundtracks/$album%aunique{} [$year]/$disc_and_track - $artist - $title
    comp: Compilations/%bucket{$year}/%the{$album%aunique{}}/$disc_and_track - $artist - $title

statefile: state.pickle

musicbrainz:
    host: musicbrainz.org
    ratelimit: 1
    ratelimit_interval: 1.0
    searchlimit: 10
    extra_tags: []

match:
    strong_rec_thresh: 0.1
    medium_rec_thresh: 0.25
    rec_gap_thresh: 0.25
    max_rec:
        missing_tracks: medium
        unmatched_tracks: medium
    distance_weights:
        source: 2.0
        artist: 3.0
        album: 3.0
        media: 1.0
        mediums: 1.0
        year: 1.0
        country: 0.5
        label: 0.5
        catalognum: 0.5
        albumdisambig: 0.5
        album_id: 5.0
        tracks: 4.0
        missing_tracks: 0.9
        unmatched_tracks: 0.6
        track_title: 3.0
        track_artist: 2.0
        track_index: 1.0
        track_length: 2.0
        track_id: 5.0
    preferred:
        countries:
        - US
        - GB|UK
        - JP
        - DE
        media: [CD, Digital Media|File]
        original_year: yes
    ignored: []
    required: []
    ignored_media:
    - Data CD
    - DVD
    - DVD-Video
    - Blu-ray
    - HD-DVD
    - VCD
    - SVCD
    - UMD
    - VHS
    ignore_data_tracks: yes
    ignore_video_tracks: yes
    track_length_grace: 10
    track_length_max: 30
embedart:
    auto: yes
    compare_threshold: 20
    ifempty: no
    maxwidth: 800
    remove_art_file: no
    quality: 0
fetchart:
    auto: yes
    cover_names: front folder
    minwidth: 0
    maxwidth: 0
    quality: 0
    enforce_ratio: no
    cautious: no
    sources:
    - filesystem
    - coverart
    - itunes
    - amazon
    - albumart
    google_key: REDACTED
    google_engine: 001442825323518660753:hrh5ch1gjzm
    fanarttv_key: REDACTED
    lastfm_key: REDACTED
    store_source: no
    high_resolution: no
bucket:
    bucket_alpha:
    - 0-9
    - A-C
    - D-F
    - G-J
    - K-N
    - O-R
    - S
    - T-Z
    bucket_year:
    - 1960s
    - 1970s
    - 1980s
    - 1990s
    - 2000s
    - 2010s
    bucket_alpha_regex:
        0-9: ^[^a-zA-Z]
    extrapolate: no
ftintitle:
    auto: yes
    drop: no
    format: feat. {0}
scrub:
    auto: no
copyartifacts:
    extensions: .cue .jpg .gif .png .pdf
    print_ignored: yes
edit:
    itemfields: track title artist album year
    albumfields: album albumartist year
    ignore_fields: id path
importadded:
    preserve_mtime: yes
    preserve_write_mtime: yes
    preserve_mtimes: no
    preserve_write_mtimes: no
duplicates:
    checksum: no
    tiebreak:
        items: [bitrate]
    album: no
    copy: ''
    count: no
    delete: no
    format: ''
    full: no
    keys: []
    merge: no
    move: ''
    path: no
    strict: no
    tag: ''
chroma:
    auto: yes
replaygain:
    auto: yes
    backend: gstreamer
    overwrite: yes
    targetlevel: 89
    per_disc: no
    peak: 'true'
    r128: [Opus]
    r128_targetlevel: 84
pathfields: {}
lastgenre:
    whitelist: yes
    min_weight: 10
    count: 1
    fallback:
    canonical: no
    source: album
    force: yes
    auto: yes
    separator: ', '
    prefer_specific: no
discogs:
    apikey: REDACTED
    apisecret: REDACTED
    tokenfile: discogs_token.json
    source_weight: 0.5
    user_token: REDACTED
    separator: ', '
    index_tracks: no
the:
    the: yes
    a: yes
    format: '{0}, {1}'
    strip: no
    patterns: []

```



Discussion:
Adding the following to line 429 in lyrics.py (_scrape_strip_cruft function) resolves the problem, but there may be a better way or place to add this to:
``` py
    html = re.sub(u'\u2005', " ", html) # replace Unicode Four-per-em space with regular space
```
Interesting; thanks! Any chance you could put together a PR with exactly that change?

Resolving patch:
diff --git a/beetsplug/lyrics.py b/beetsplug/lyrics.py
--- a/beetsplug/lyrics.py
+++ b/beetsplug/lyrics.py
@@ -424,6 +424,7 @@ def _scrape_strip_cruft(html, plain_text_out=False):
     html = re.sub(r' +', ' ', html)  # Whitespaces collapse.
     html = BREAK_RE.sub('\n', html)  # <br> eats up surrounding '\n'.
     html = re.sub(r'(?s)<(script).*?</\1>', '', html)  # Strip script tags.
+    html = re.sub(u'\u2005', " ", html)  # replace unicode with regular space
 
     if plain_text_out:  # Strip remaining HTML tags
         html = COMMENT_RE.sub('', html)


Test patch: