Repository: beetbox/beets

Issue:
chroma: Submission fails if fingerprint already exists
<!--
Describe your problem, feature request, or discussion topic here.

Please fill out this and the "Setup" section below and remember to include
enough detail so that other people can reproduce the problem.
-->

Hi there,




### Problem

Running this command in verbose (`-vv`) mode:

```sh
$ beet fingerprint # This works fine
$ beet -vv submit
```

Led to this problem:

```
<snip>
chroma: /run/media/john/pool/Music/John Zorn/The Big Gundown_ John Zorn Plays the Music of Ennio Morricone/02 Peur sur la ville.flac: fingerprint exists, skipping
chroma: submitting MBID                                                                                                                                                                                              
chroma: /run/media/john/pool/Music/John Zorn/The Big Gundown_ John Zorn Plays the Music of Ennio Morricone/03 Poverty (Once Upon a Time in America).flac: fingerprint exists, skipping
chroma: submitting MBID
chroma: submitting 64 fingerprints
chroma: acoustid submission error: error 2: missing required parameter "fingerprint"  
```

Here's a link to the music files that trigger the bug (if relevant): N/A, happens for each batch of IDs in my library, no IDs are submitted


### Setup

* OS: Arch linux
* Python version: Python 3.9.1 (default, Dec 13 2020, 11:55:53) 
* beets version:
```
beets version 1.5.0                                                                                                                                                                                                 
Python version 3.9.1                                                                                                                                                                                                
plugins: badfiles, chroma, fetchart, ftintitle, missing, the, zero 
```

```
❯ yay -Qi beets
Name            : beets
Version         : 1.4.9.r1031.cbc045f1c-1
Description     : Flexible music library manager and tagger
Architecture    : any
URL             : https://beets.io/
Licenses        : MIT
Groups          : None
Provides        : None
Depends On      : python-confuse  python-jellyfish  python-mediafile  python-munkres  python-musicbrainzngs  python-setuptools  python-six  python-unidecode  python-yaml
Optional Deps   : bash-completion: Bash completion
                  bs1770gain: ReplayGain plugin [installed]
                  chromaprint: Chromaprint/Acoustid plugin [installed]
                  ffmpeg: Convert, ReplayGain plugins [installed]
                  gst-plugins-bad: Chromaprint/Acoustid, BPD, ReplayGain plugins [installed]
                  gst-plugins-good: Chromaprint/Acoustid, BPD, ReplayGain plugins [installed]
                  gst-plugins-ugly: Chromaprint/Acoustid, BPD, ReplayGain plugins [installed]
                  gst-libav: Chromaprint/Acoustid, BPD, ReplayGain plugins [installed]
                  gst-python: Chromaprint/Acoustid, BPD, ReplayGain plugins [installed]
                  imagemagick: Thumbnails plugin [installed]
                  python-beautifulsoup4: Lyrics plugin [installed]
                  python-flask: Web plugin [installed]
                  python-gobject: ReplayGain plugin [installed]
                  python-mpd2: MPDStats plugin
                  python-pyacoustid: Chromaprint/Acoustid plugin [installed]
                  python-pylast: LastGenre plugin [installed]
                  python-requests: Chromaprint/Acoustid, BPD, FetchArt plugins [installed]
                  python-xdg: Thumbnails plugin [installed]
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 3.37 MiB
Packager        : Maxime Gauduin <alucryd@archlinux.org>
Build Date      : Wed 13 Jan 2021 02:39:44 PM GMT
Install Date    : Wed 13 Jan 2021 05:58:32 PM GMT
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : Signature
```
* Turning off plugins made problem go away (yes/no): No (obviously while keeping chroma on)

<!--
You can turn off plugins temporarily by passing --plugins= on the command line:

$ beet --plugins= version
-->

My configuration (output of `beet config`) is:

```yaml
directory: /run/media/john/pool/Music
library: ~/.config/beets/library.db
import:
        move: yes
plugins: fetchart chroma ftintitle missing zero badfiles the
acoustid:
        apikey: <redacted>
zero:
        fields: comments
        update_database: true
```



Discussion:
Huh, that definitely seems bad! The weird thing is that the library should already be defensively checking whether we're missing the fingerprint parameter:
https://github.com/beetbox/pyacoustid/blob/dd69cfd1b68ac15bf84597b6ab5ab0aae3226a83/acoustid.py#L384

It might be helpful to try snooping the HTTP traffic to the Acoustid web server to see what the request _actually_ contains… any chance you'd be willing to fire up Wireshark or similar to give that a shot?
Certainly!

I've captured the packets in wireshark - see attached :+1: 

[acoustid.pcapng.gz](https://github.com/beetbox/beets/files/5814655/acoustid.pcapng.gz)

Here's another dump which I think might be more helpful for you - I'm not super familiar with wireshark as most of my HTTP snooping is done these days with mitmproxy on mobile :innocent: 

[acoustid2.pcapng.gz](https://github.com/beetbox/beets/files/5814661/acoustid2.pcapng.gz)

Thanks!!! It's quite strange; the request is indeed missing the parameter… the request data for `/v2/submit` looks like this:

```
format=json&client=<redacted>&user=<redacted>&duration.0=464&track.0=Banner+%28Original+Mix%29&artist.0=%26lez&album.0=Essential+Afro+House&albumartist.0=Various+Artists&year.0=2018&trackno.0=2&discno.0=0&duration.1=549&mbid.1=10117223&duration.2=481&mbid.2=9405423&duration.3=434&…
```

That is, there are MBIDs and durations in there but not fingerprints. I'm a little stumped as to why this could happen!

I think my next step would be to do a little print-debugging to understand what's going on inside the pyacoustid library. In this `submit` function:
https://github.com/beetbox/pyacoustid/blob/dd69cfd1b68ac15bf84597b6ab5ab0aae3226a83/acoustid.py#L355

I'd print out `data`, and then `args` after the main loop that builds up the `args` dict, just to see if we can get a handle on what's going wrong. To reiterate, the weird thing is that the check `"fingerprint" not in d` should not allow the function get as far as actually making the API request if the fingerprints are missing.
@sampsyo no problem, I'll have a crack at adding this into my copy of beets & report back within the hour!
@sampsyo so I made the changes as you describe:

Then I ran `beet -vv submit > out 2>&1`: https://termsend.pl/o/gdaef
OK, something I notice here is that we have entries like `fingerprint.28': None` to be submitted to the server. Can you check whether these files actually have fingerprints, perhaps using `beet info` or similar? I know it says `fingerprint exists, skipping`, but something isn't connecting when we actually make the relevant API call.

This also makes me think we need better validation, either on the beets side or in the library. Probably in the plugin itself, we should check that the fingerprint value is not `None`.
It does seem like they do indeed have a fingerprint:

```
❯ beet info Liwung
/run/media/john/pool/Music/[unknown]/Java - The Jasmine Isle_ Gamelan Music/10 Liwung.mp3
acoustid_fingerprint: AQADtFKShkoYJXjiDObR43yQPGKR40d65UiaH7PARiGFe4AbF8eR5cWYR4RzdUKPJ8Ef9AkNLUeeH8eTHCfRhUqFlnxw4RUqI2U2HqUiJUR-Bf_RjGiTZMTnJkPiXgjz6GjxR8QbPA6aKkfFNHiY4_XwRuiRVoMdQod9uEeo4w1OEXa04dlwG07oDt-IB3-LhE06psh1PMcPJ_ux5zj3oPmRKEf64ZieY0qU2HiiuEkM_niO4mCO2FDD5Mh9PBqa5-iPg9vh5DL24GF-NOGJ3sNz_Cxe4emKL0dzODmqQzncFKxsoXaeoWmC_fgxHc3BK0dkZcrx7PgR6kfz4_vwRcPhBzn05JjyB2GuC3N1PESeo2tyiHmOXkY_aE6kDxGpHFwOH_vx6OglND1-HD-2HFx87MnQ9IF2C8dF_Lh4_IuO_kLT6Xj64HSwPUMifTmiPsMT_MEOV8el4-A1mHGGlS3qCc2P5EfazfhRH-aPn3AvuCH640hJjkj6H__x6grRX8d3NMfxAIePB9HCDEnTgDrB58GuH8Z1tIl4NF2SFyfOw2N4PBGHSKkKKXLGI2p84jr-40bTLUedBz9-VCmNKfkyHFQj7Bp6NM-RH8n54bh0POiMT4WF68gJLT_yD89DfEbyNAjToQfw48GJ5kdZ5HmgK0Ne9IOPHtEj4Yc--MYVPK0SUPShvsgFN6IH-hjKo39wSfvxpUa1FT4Z9C9eD08iPGKGhD2i5zjK5wGTcC9wjNg-PFgupaKQP0gqJUlCPIoYCs1IDsUR6NqOHL_hFz-eH9tBGLOk4Am-9Mj5wD3x44T4HnlzLMeeCD1zvAkP_PCPPjge_fiC5kX1URjTPvAPnnhQu0GyPEd2zD-OHzf68EHzE3hyNOUxyVE4ZIxHqFuHPTYuaPkRensoPMeP8_CDHRkfKMovhMfF4MlkNC_6IwyXExry7LiG5wl-BQ4pvIidF7p6hD-aP3geNOj144mOR0VzVBSPPNuh6si3NPBxBWeC8zmabpbQ6hGuPDhxhnAM-Tjxszinh7D9BN9xHD6uE_ZxH-_xI02Ob3igE6G3oMeHH1dyJWi6fXh2NM-JG1PKIWeQKKk-XHj4GFOey-jz4Hh2NNuDaRNT4zv4YzoaHb_wKVdwRYuRX4OQo8dVuMkBy58QHiqDMB364x6aX6gfNGH0GeFXaPmJkJdwHU9QfzKat3hG4g6-48Wr49VhJiJ2HoeX1-CVa9ijCw3CZx-Sb0Ej_ej1pFh-RdiRvAibHO6z4pmRHgk_WujxJMcuXWgWH_3BI-GXIzjAPDGmHD-SMjryB-fRJO2UoH-Op8XxH9ePj8RTnPiXZUiTnsGxQ3rCIPrw4DqaSjl6XbiOZnnaoeQzLHOw70QY8xmSM3guvA_yHNqRp0GZCM0e3NjV40Ui_UGe4zy8-CAb7Bd6vEjVItmH4_izIjGZB_GP7eFxg28V7C2aM6PR6sEVHMeWZ1iW41OZoU5yFemJZH_x43g0XC9OtFG3o0mTGT_-C1sjIiGy48YV3Az8seAx3EdCJj5ydM2LJttT_CmuE09-uFeGz6iV8RL6C9_y4QkuovmCI1lUBD9RJtIJN9rxlsDR63he3Cd-JP2RJpHxGB9-o3eO6-izB8_xH3-GB_-RsEH0g4T_HI8r7CyanCOm68GD6yD8BqOOzmjCFD_6o3lCJM84IsePOkUTLgqOHs16POGD50PyIw_OyMjDQzOP9PDxwc2D-niIHmGO5DkuKehK_DBXTEaeIgHzxWgq5tjhd8GP5HmRVsHL4nMVXMeSE1P-4jyDx8f_4Xh8_Mcf5EmO69ByRLpwKSfo5oL7Yl9O4uaxH96bgt-xZw-SF3l1NPsyVEofBd_x5Qh9JPxhBvZRb8eDU4eTRViOXDkUMUeY4zseFncibA8uJJoj5IePY3sS5OyPZM_RnHgi3EPPLEOj_fgiXGhiPXiOpsePH5cCN7UQ56kCx8Sxywafi4H_4szx7IWeCPGDbcScRcfX4DlcB2z2Y5ayNHiO5kRf5EkeJHN2NMXOo_NX-KiPmFlyJFdeofnB9NiPKwvyF8rBD9svyA9y8biyGE8AOz3yHVo4HuEJP8Z_nCmsOXiOnkEDLVxRJxkuLceTNUezK0L7HEGSJwmu4sH647lh_TMeZcV71AnRMDt-NI-E_iF-4jv0JDH8HX_wD2dQJdfRvBUe3ETyJ8hNwd0e3AIu6UXzo2cexPoDjVVShKkk_PgWHz_a6EKzfvinoXlY9MeDXEOSacfRrMlR5gefGt22PGiuzPiyFCd-LFq44PisoGnyEBWR49aJT5GFH3qOeguaHzeuI0zyLELzLBOe4ngP5NCjE_6O_F3QHk0kLSmeB08RZszQPkiu42kOXznuC4ef49HhKY3xWcNxbUjmKEeqoMzxJOiP5gFmEdbyYM-RP8FjQUvFHPJy5JOOnsXvo2meZCiTffh7NH_Q4_GFDw16PMGjJB-uhUKOl_iO59ByAKoIUIQBCAgA1qwJBDFIKaSEYEkAAIZRAkgihHAIAEMEYIQrBBwFBiFwgFDMMAMQwQAhYQAUDDDBIDJFEKMAodgJAAwxihFGkEIGOmcAogw5YowzimgDABRMgASgAgoIYQwyhFgDHFZcGEOghM4gBIRTFjmECAJACIQEAMAQzgxkhiIDBFJKAaIAUIyoIIwgBAAEgDMKLqupEEYYAYUwhCFggACAIUABY0YKAIYBQCkEOHDIQoGIE444RyxyCAzCiABECQKJUk4RabgkCAghkGBIGWUgFQCBQYkUgACFEFPAA-OAcUIRphQABFDQBQCEGWEVQkQwAIAhgCGkFWFKgQGAQkYKooBAihmmkKHACGKUQMQw5IRhghCCBEDOGCCQMJgAIwxFCgnBEAEFEEcAAAjBqJCySBEEEEEUGQYNMYggQACShBACFKDEISeM0hopQoQSjCECGBNQICQYUBgZBRBDSpoCDDAADGCIAIwAQwASCEHFlBCICACEYEIwhARAUgBggIDWAYGIsIgAIBwEgjQDAEMGGAUIIpoQB5RxVCAlEFJICGO4EiY0IwwChAICGDLUOEQAA0xShIQHAhsgHOVAIGCRQYYIpBQRhgGhAAFJsCAJZNII5JCwCiHqkFJAAiEUNYw4QoFRQhIhiAFGIISVQ4BQiASxoBFCEFJIGiOEUAYppilRxCEjGFHGGWMBA1IoJhBAQBCAAAJUGIAIFIQIBQgAABMjhCFASYGIMgAoxSiQWCGGDGQUG-UUqEAA4QCowhghDWKWMiYJcQIAAggghlsAETUCQaCZUIAYSg0DgEEHkHHsCwKFMSAYhiwyQABGBDFMOAAEAgMoAYBjAAIDlFDEUKQIBQgCAQBDxhiBiQBIAAEUQoAKhCgAglpkIXACACYQ0e4hRYShijAEGBAEIMQAM8IQQQBSlAGFgCBEoECNchAxBgJTyDEJJEMEEAMIQsIAQRRCToAikGlIIaMA
               album: Java - The Jasmine Isle: Gamelan Music
         albumartist: [unknown]
  albumartist_credit: [unknown]
    albumartist_sort: [unknown]
         albumstatus: Official
           albumtype: album
            arranger:
                 art: False
              artist: [unknown]
       artist_credit: [unknown]
         artist_sort: [unknown]
            bitdepth: 0
             bitrate: 238267
                 bpm: 74
          catalognum: H-72031
            channels: 2
                comp: False
             country: US
                date: 1969-01-01
                disc: 1
           disctotal: 1
              format: MP3
               genre: Rock
              genres: Rock
         initial_key: A#m
               label: Nonesuch
            language: eng
              length: 360.5420408163265
              lyrics:
    mb_albumartistid: 125ec42a-7229-4250-afc5-e057484327fe
          mb_albumid: e10955b4-fc5b-4b03-ae03-5f975233328c
         mb_artistid: 125ec42a-7229-4250-afc5-e057484327fe
   mb_releasegroupid: ae2192c4-e9b4-3310-81e7-cad8b302b986
   mb_releasetrackid: 958592e8-42fb-430b-a44d-472c300fff7d
          mb_trackid: 3bcaf6b1-81c3-46ad-a178-64d79189d117
               media: 12" Vinyl
       original_date: 1969-01-01
       original_year: 1969
     r128_album_gain: 0.0
     r128_track_gain: 0.0
          samplerate: 44100
              script: Latn
               title: Liwung
               track: 10
          tracktotal: 10
                year: 1969
```

Got it, thanks!! I think this is the problem:
https://github.com/beetbox/beets/blob/cbc045f1c8f898acd8798ff134a25557f1769525/beetsplug/chroma.py#L326-L332

That is, in the `fingerprint exists, skipping` case, we are missing a `return item.acoustid_fingerprint` that would would actually cause the fingerprint to be submitted!

Any chance you'd be interested in trying that out and, if it works, opening a PR?
Hey @sampsyo apologies for the wait; I'm going to look at this momentarily - I'll update this comment with my findings :)
Absolutely no worries whatsoever. Take your time! :smiley:

Resolving patch:
diff --git a/beetsplug/chroma.py b/beetsplug/chroma.py
--- a/beetsplug/chroma.py
+++ b/beetsplug/chroma.py
@@ -329,7 +329,7 @@ def fingerprint_item(log, item, write=False):
         else:
             log.info(u'{0}: using existing fingerprint',
                      util.displayable_path(item.path))
-            return item.acoustid_fingerprint
+        return item.acoustid_fingerprint
     else:
         log.info(u'{0}: fingerprinting',
                  util.displayable_path(item.path))


Test patch: