Repository: beetbox/beets

Issue:
Reimporting from another source should not remove the specific information from other sources
https://github.com/beetbox/beets/pull/4735 did a great job cleaning up the reimport process. However, now that I have been using this for some time, I realized that it introduced a bug. Specifically, we are not even preserving ids from any previous imports. This seems wrong. We need ids from every possible source (for specific purposes). For example, even if we use MusicBrainz, we may need `spotify_album_id` to obtain popularity or music attributes. 

The reimport process should only overwrite the common fields (and not touch any fields from other sources). Thus, reimporting with MB should not remove `spotify_album_id`

cc: @JOJ0 

Reimporting from another source should not remove the specific information from other sources
https://github.com/beetbox/beets/pull/4735 did a great job cleaning up the reimport process. However, now that I have been using this for some time, I realized that it introduced a bug. Specifically, we are not even preserving ids from any previous imports. This seems wrong. We need ids from every possible source (for specific purposes). For example, even if we use MusicBrainz, we may need `spotify_album_id` to obtain popularity or music attributes. 

The reimport process should only overwrite the common fields (and not touch any fields from other sources). Thus, reimporting with MB should not remove `spotify_album_id`

cc: @JOJ0 



Discussion:
Good catch. To clarify how it _should_ work: If a new/different spotify id is found it is overwritten, if none is found it should stay.

Maybe I screwed up on the latest refactoring commits.

Please try checkin out this commit and see if it still always overwrites with empty data/deletes existing: https://github.com/beetbox/beets/commit/04628ada96acaed57a4e78a67681de6ff724e6f8
Hmm...I am going to do some more investigation and report back. 
@JOJ0 I think (at least one) issue is with re-import with a plugin (JioSaavn in this case). See below:

Initial import with Spotify:
<img width="325" alt="image" src="https://user-images.githubusercontent.com/18319734/235371365-1ffd4920-2508-4359-863c-e1d23941e5cf.png">

Re-import with JioSaavn (relevant logs):

```
Reimported album 1. Preserving attribute ['added']. Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported album 1. Preserving flexible attributes []. Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported item 1. Preserving attribute ['added']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1. Not preserving flexible attributes ['data_source', 'spotify_album_id']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1. Preserving flexible attributes ['spotify_artist_id', 'spotify_track_id']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Sending event: database_change
```

<img width="377" alt="image" src="https://user-images.githubusercontent.com/18319734/235371419-8af1beb0-6225-4082-8f22-3669bea5ed97.png">

Reimport with MusicBrainz:
```
Reimported album 1. Preserving attribute ['added']. Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported album 1. Preserving flexible attributes []. Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported item 1. Preserving attribute ['added']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1. Not preserving flexible attributes ['data_source']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1. Preserving flexible attributes ['jiosaavn_album_id', 'jiosaavn_artist_id', 'jiosaavn_perma_url', 'cover_art_url', 'jiosaavn_track_id', 'jiosaavn_updated', 'spotify_artist_id', 'spotify_track_id']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Sending event: database_change
```
Output:
<img width="354" alt="image" src="https://user-images.githubusercontent.com/18319734/235371531-c8d3b68d-a798-4449-816c-bc7781a46c3d.png">

Reimport with JioSaavn
```
Reimported album 1. Preserving attribute ['added']. Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported album 1. Preserving flexible attributes []. Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported item 1. Preserving attribute ['added']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1. Not preserving flexible attributes ['data_source', 'spotify_album_id']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1. Preserving flexible attributes ['track_alt', 'jiosaavn_album_id', 'jiosaavn_artist_id', 'jiosaavn_perma_url', 'cover_art_url', 'jiosaavn_track_id', 'jiosaavn_updated', 'spotify_artist_id', 'spotify_track_id']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Sending event: database_change

```
Hi, thanks so much for the detailed report!!! Aprreciated! So to summarize:

With a **freshly** imported (never reimported) album:

- JioSaavn -> The spotify_album_id flex attr is entirely deleted from the DB.
- MusicBrainz -> Everything works as expected.

Did you do above test with a freshly imported album before you did the MusicBrainz test? Sorry for nitpicking but that would be an important fact for me to know!

Did you try checking out the commit I mentioned above? This commit also would help with seeing what we actually get from JioSaavn/MB. You will see key+value in the logs if a new spotify id is found during the reimport. I removed this information from the final logging. Checking out this commit will give us debug prints so to say :-)


So, I used the commit you linked earlier and here are the logs/output. It seems it is working fine with this commit. 

Freshly imported with MB:
<img width="284" alt="image" src="https://user-images.githubusercontent.com/18319734/235446289-93a55622-50e8-4077-b2da-bc59d6c9bc88.png">

Reimported with Spotify:
```
Reimported album 1 attribute: ['added'] from album 1, Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported album 1 flexible attributes: [] from album 1, Path:/home/arsaboo/shared/music/XYZ (2023)
Reimported item 1 attribute: ['added'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Not reimported item 1 flexible attribute: data_sourceNew value: Spotify, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1 flexible attributes: ['track_alt', 'spotify_album_id'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Sending event: database_change
```
<img width="307" alt="image" src="https://user-images.githubusercontent.com/18319734/235446449-5ae99ab5-e7e0-4eea-876e-9072da61c9b7.png">

Reimported again with JioSaavn:
```
Reimported album 1 attribute: ['added'] from album 1, Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported album 1 flexible attributes: [] from album 1, Path:/home/arsaboo/shared/music/XYZ (2023)
Reimported item 1 attribute: ['added'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Not reimported item 1 flexible attribute: data_sourceNew value: JioSaavn, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1 flexible attributes: ['spotify_album_id', 'spotify_artist_id', 'spotify_track_id', 'track_alt'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Sending event: database_change
```
<img width="350" alt="image" src="https://user-images.githubusercontent.com/18319734/235446593-b2a2256f-9f2b-4034-96d8-961859ca1f43.png">

Reimported again with Spotify:
```
Reimported album 1 attribute: ['added'] from album 1, Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported album 1 flexible attributes: [] from album 1, Path:/home/arsaboo/shared/music/XYZ (2023)
Reimported item 1 attribute: ['added'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Not reimported item 1 flexible attribute: data_sourceNew value: Spotify, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1 flexible attributes: ['jiosaavn_album_id', 'jiosaavn_artist_id', 'jiosaavn_perma_url', 'cover_art_url', 'jiosaavn_track_id', 'jiosaavn_updated', 'track_alt', 'spotify_album_id', 'spotify_artist_id', 'spotify_track_id'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Sending event: database_change
```

<img width="345" alt="image" src="https://user-images.githubusercontent.com/18319734/235446777-3f33ad81-13a1-4d91-bcda-24ad553ba32d.png">

Reimported again with JioSaavn:
```
Reimported album 1 attribute: ['added'] from album 1, Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported album 1 flexible attributes: [] from album 1, Path:/home/arsaboo/shared/music/XYZ (2023)
Reimported item 1 attribute: ['added'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Not reimported item 1 flexible attribute: data_sourceNew value: JioSaavn, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1 flexible attributes: ['spotify_album_id', 'spotify_artist_id', 'spotify_track_id', 'track_alt', 'jiosaavn_album_id', 'jiosaavn_artist_id', 'jiosaavn_perma_url', 'cover_art_url', 'jiosaavn_track_id', 'jiosaavn_updated'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Sending event: database_change
```
<img width="308" alt="image" src="https://user-images.githubusercontent.com/18319734/235446901-9d200b46-e35a-4a4f-8e08-72c3a8a10f62.png">

Let me know if you want me to test any other sequence of (re)imports. 
Thanks so much for the detailed testing! That helps! I don't find much time for testing myself currently.

I have an idea of what is wrong in this list comprehension:

https://github.com/beetbox/beets/blob/0753f47e3edd560260a1886e3718fbb3d2dfd83e/beets/importer.py#L827-L829

Can you try changing it to:

```
            overwritten_fields = [k for k in existing_fields
                                  if k in overwrite_keys
                                  and new_obj.get(k)
                                  and existing_fields.get(k) != new_obj.get(k)]
```
Good catch. To clarify how it _should_ work: If a new/different spotify id is found it is overwritten, if none is found it should stay.

Maybe I screwed up on the latest refactoring commits.

Please try checkin out this commit and see if it still always overwrites with empty data/deletes existing: https://github.com/beetbox/beets/commit/04628ada96acaed57a4e78a67681de6ff724e6f8
Hmm...I am going to do some more investigation and report back. 
@JOJ0 I think (at least one) issue is with re-import with a plugin (JioSaavn in this case). See below:

Initial import with Spotify:
<img width="325" alt="image" src="https://user-images.githubusercontent.com/18319734/235371365-1ffd4920-2508-4359-863c-e1d23941e5cf.png">

Re-import with JioSaavn (relevant logs):

```
Reimported album 1. Preserving attribute ['added']. Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported album 1. Preserving flexible attributes []. Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported item 1. Preserving attribute ['added']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1. Not preserving flexible attributes ['data_source', 'spotify_album_id']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1. Preserving flexible attributes ['spotify_artist_id', 'spotify_track_id']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Sending event: database_change
```

<img width="377" alt="image" src="https://user-images.githubusercontent.com/18319734/235371419-8af1beb0-6225-4082-8f22-3669bea5ed97.png">

Reimport with MusicBrainz:
```
Reimported album 1. Preserving attribute ['added']. Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported album 1. Preserving flexible attributes []. Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported item 1. Preserving attribute ['added']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1. Not preserving flexible attributes ['data_source']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1. Preserving flexible attributes ['jiosaavn_album_id', 'jiosaavn_artist_id', 'jiosaavn_perma_url', 'cover_art_url', 'jiosaavn_track_id', 'jiosaavn_updated', 'spotify_artist_id', 'spotify_track_id']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Sending event: database_change
```
Output:
<img width="354" alt="image" src="https://user-images.githubusercontent.com/18319734/235371531-c8d3b68d-a798-4449-816c-bc7781a46c3d.png">

Reimport with JioSaavn
```
Reimported album 1. Preserving attribute ['added']. Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported album 1. Preserving flexible attributes []. Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported item 1. Preserving attribute ['added']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1. Not preserving flexible attributes ['data_source', 'spotify_album_id']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1. Preserving flexible attributes ['track_alt', 'jiosaavn_album_id', 'jiosaavn_artist_id', 'jiosaavn_perma_url', 'cover_art_url', 'jiosaavn_track_id', 'jiosaavn_updated', 'spotify_artist_id', 'spotify_track_id']. Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Sending event: database_change

```
Hi, thanks so much for the detailed report!!! Aprreciated! So to summarize:

With a **freshly** imported (never reimported) album:

- JioSaavn -> The spotify_album_id flex attr is entirely deleted from the DB.
- MusicBrainz -> Everything works as expected.

Did you do above test with a freshly imported album before you did the MusicBrainz test? Sorry for nitpicking but that would be an important fact for me to know!

Did you try checking out the commit I mentioned above? This commit also would help with seeing what we actually get from JioSaavn/MB. You will see key+value in the logs if a new spotify id is found during the reimport. I removed this information from the final logging. Checking out this commit will give us debug prints so to say :-)


So, I used the commit you linked earlier and here are the logs/output. It seems it is working fine with this commit. 

Freshly imported with MB:
<img width="284" alt="image" src="https://user-images.githubusercontent.com/18319734/235446289-93a55622-50e8-4077-b2da-bc59d6c9bc88.png">

Reimported with Spotify:
```
Reimported album 1 attribute: ['added'] from album 1, Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported album 1 flexible attributes: [] from album 1, Path:/home/arsaboo/shared/music/XYZ (2023)
Reimported item 1 attribute: ['added'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Not reimported item 1 flexible attribute: data_sourceNew value: Spotify, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1 flexible attributes: ['track_alt', 'spotify_album_id'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Sending event: database_change
```
<img width="307" alt="image" src="https://user-images.githubusercontent.com/18319734/235446449-5ae99ab5-e7e0-4eea-876e-9072da61c9b7.png">

Reimported again with JioSaavn:
```
Reimported album 1 attribute: ['added'] from album 1, Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported album 1 flexible attributes: [] from album 1, Path:/home/arsaboo/shared/music/XYZ (2023)
Reimported item 1 attribute: ['added'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Not reimported item 1 flexible attribute: data_sourceNew value: JioSaavn, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1 flexible attributes: ['spotify_album_id', 'spotify_artist_id', 'spotify_track_id', 'track_alt'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Sending event: database_change
```
<img width="350" alt="image" src="https://user-images.githubusercontent.com/18319734/235446593-b2a2256f-9f2b-4034-96d8-961859ca1f43.png">

Reimported again with Spotify:
```
Reimported album 1 attribute: ['added'] from album 1, Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported album 1 flexible attributes: [] from album 1, Path:/home/arsaboo/shared/music/XYZ (2023)
Reimported item 1 attribute: ['added'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Not reimported item 1 flexible attribute: data_sourceNew value: Spotify, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1 flexible attributes: ['jiosaavn_album_id', 'jiosaavn_artist_id', 'jiosaavn_perma_url', 'cover_art_url', 'jiosaavn_track_id', 'jiosaavn_updated', 'track_alt', 'spotify_album_id', 'spotify_artist_id', 'spotify_track_id'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Sending event: database_change
```

<img width="345" alt="image" src="https://user-images.githubusercontent.com/18319734/235446777-3f33ad81-13a1-4d91-bcda-24ad553ba32d.png">

Reimported again with JioSaavn:
```
Reimported album 1 attribute: ['added'] from album 1, Path: /home/arsaboo/shared/music/XYZ (2023)
Reimported album 1 flexible attributes: [] from album 1, Path:/home/arsaboo/shared/music/XYZ (2023)
Reimported item 1 attribute: ['added'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Not reimported item 1 flexible attribute: data_sourceNew value: JioSaavn, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Reimported item 1 flexible attributes: ['spotify_album_id', 'spotify_artist_id', 'spotify_track_id', 'track_alt', 'jiosaavn_album_id', 'jiosaavn_artist_id', 'jiosaavn_perma_url', 'cover_art_url', 'jiosaavn_track_id', 'jiosaavn_updated'] from item 1, Path: /home/arsaboo/shared/music/XYZ (2023)/XYZ (2023) - Jaadui.mp3
Sending event: database_change
```
<img width="308" alt="image" src="https://user-images.githubusercontent.com/18319734/235446901-9d200b46-e35a-4a4f-8e08-72c3a8a10f62.png">

Let me know if you want me to test any other sequence of (re)imports. 
Thanks so much for the detailed testing! That helps! I don't find much time for testing myself currently.

I have an idea of what is wrong in this list comprehension:

https://github.com/beetbox/beets/blob/0753f47e3edd560260a1886e3718fbb3d2dfd83e/beets/importer.py#L827-L829

Can you try changing it to:

```
            overwritten_fields = [k for k in existing_fields
                                  if k in overwrite_keys
                                  and new_obj.get(k)
                                  and existing_fields.get(k) != new_obj.get(k)]
```

Resolving patch:
diff --git a/beets/importer.py b/beets/importer.py
--- a/beets/importer.py
+++ b/beets/importer.py
@@ -826,6 +826,7 @@ def _reduce_and_log(new_obj, existing_fields, overwrite_keys):
             existing_fields = dict(existing_fields)
             overwritten_fields = [k for k in existing_fields
                                   if k in overwrite_keys
+                                  and new_obj.get(k)
                                   and existing_fields.get(k) != new_obj.get(k)]
             if overwritten_fields:
                 log.debug(


Test patch: