Repository: beetbox/beets (Python)

Issue:
replaygain: bs1770gain v0.6.0 XML output can't be parsed
<!--
Describe your problem, feature request, or discussion topic here.

Please fill out this and the "Setup" section below and remember to include
enough detail so that other people can reproduce the problem.
-->

### replaygain: bs1770gain v0.6.0 XML output can't be parsed

Running `beet -vv replaygain` with the `bs1770gain` backend results in 
```
replaygain: analyzing Nyul -  When The Path Fades - Floatfall
replaygain: executing bs1770gain --ebu --xml -p /home/ybnd/code/beets/library/Nyul/2014 -  When The Path Fades/05 Floatfall.flac
replaygain: analysis finished: <!-- scanning 1 -->
<!-- analyzing ... -->
<bs1770gain norm="-23.00">
  <track total="1" number="1" file="05 Floatfall.flac">
    <integrated lufs="-10.99" lu="-12.01"/>
    <sample-peak spfs="-0.10" amplitude="0.99"/>
  </track>
  <integrated lufs="-10.99" lu="-12.01"/>
  <sample-peak spfs="-0.10" amplitude="0.99"/>
</bs1770gain>
<!-- done. -->

replaygain: ReplayGain error: The bs1770gain tool produced malformed XML. Using version >=0.4.10 may solve this problem.
```
for all tracks.


1. The actual XML output `text` that gets parsed at https://github.com/beetbox/beets/blob/ca3142b1d92d177a2d789775a4ff2ee5569b9ba6/beetsplug/replaygain.py#L318-L324 gets spliced with some progress percentages 
    ```
    b'<!-- scanning 1 -->
    <!-- analyzing ... -->
    <bs1770gain norm="-23.00">
      <track total="1" number="1" file="05 Floatfall.flac">
    0%\x08\x080%\x08\x080%\x08\x080%\x08\x08 ...
    ... 100%\x08\x08\x08\x08    \x08\x08\x08\x08    <integrated lufs="-10.99" lu="-12.01"/>
        <sample-peak spfs="-0.10" amplitude="0.99"/>
      </track>
      <integrated lufs="-10.99" lu="-12.01"/>
      <sample-peak spfs="-0.10" amplitude="0.99"/>
    </bs1770gain>
    <!-- done. -->'
    ```
2. The parser expects `factor` instead of `amplitude` in the `sample-peak` element, which *used to be* what the output looked like based on https://github.com/beetbox/beets/issues/2983): https://github.com/beetbox/beets/blob/ca3142b1d92d177a2d789775a4ff2ee5569b9ba6/beetsplug/replaygain.py#L296-L299
    

### Setup

* OS: 5.4.14-2-MANJARO
* Python version: 3.8.1
* beets version: [current master](https://github.com/beetbox/beets/tree/ca3142b1d92d177a2d789775a4ff2ee5569b9ba6)
* Turning off plugins made problem go away (yes/no): n/a

My configuration (output of `beet config`) is:

```yaml
replaygain:
    backend: bs1770gain
    method: ebu
```

with 

```
$ bs1770gain --version
bs1770gain 0.6.0, Copyright (C) Peter Belkner 2014-2019.
```





Discussion:
I can solve the progress percentage issue with
```python
temp = text.decode('ascii')
while '\x08' in temp:
    temp = re.sub('[^\x08]\x08', '', temp)
parser.Parse(text, True)
```
but this still leaves the second issue and the following error
```
start_element_handler
    state['peak'] = float(attrs[u'factor'])
KeyError: 'factor'
```



With other `bs1770gain` methods, the `integrated` element includes `db` instead of `lu`

* default, `method: replaygain`
    ```
  <integrated dbfs="-10.99" db="-7.01"/>
  <sample-peak dbfs="-0.10" amplitude="0.99"/>
    ```
* `method: atsc`
    ```
  <integrated dbfs="-10.99" db="-13.01"/>
  <sample-peak dbfs="-0.10" amplitude="0.99"/>
    ```
resulting in the following error
```
start_element_handler
    state['gain'] = float(attrs[u'lu'])
KeyError: 'lu'
```
Thanks for tracking it down; happy to merge a patch that adapts to the new names.



Patch:
diff --git a/beetsplug/replaygain.py b/beetsplug/replaygain.py
index 1076ac7145..646e3accea 100644
--- a/beetsplug/replaygain.py
+++ b/beetsplug/replaygain.py
@@ -22,6 +22,7 @@
 import sys
 import warnings
 import enum
+import re
 import xml.parsers.expat
 from six.moves import zip
 
@@ -66,6 +67,11 @@ def call(args, **kwargs):
         raise ReplayGainError(u"argument encoding failed")
 
 
+def after_version(version_a, version_b):
+    return tuple(int(s) for s in version_a.split('.')) \
+            >= tuple(int(s) for s in version_b.split('.'))
+
+
 def db_to_lufs(db):
     """Convert db to LUFS.
 
@@ -147,8 +153,12 @@ def __init__(self, config, log):
 
         cmd = 'bs1770gain'
         try:
-            call([cmd, "--help"])
+            version_out = call([cmd, '--version'])
             self.command = cmd
+            self.version = re.search(
+                'bs1770gain ([0-9]+.[0-9]+.[0-9]+), ',
+                version_out.stdout.decode('utf-8')
+            ).group(1)
         except OSError:
             raise FatalReplayGainError(
                 u'Is bs1770gain installed?'
@@ -241,17 +251,23 @@ def compute_chunk_gain(self, items, is_album, target_level):
         if self.__method != "":
             # backward compatibility to `method` option
             method = self.__method
+            gain_adjustment = target_level \
+                - [k for k, v in self.methods.items() if v == method][0]
         elif target_level in self.methods:
             method = self.methods[target_level]
             gain_adjustment = 0
         else:
-            method = self.methods[-23]
-            gain_adjustment = target_level - lufs_to_db(-23)
+            lufs_target = -23
+            method = self.methods[lufs_target]
+            gain_adjustment = target_level - lufs_target
 
         # Construct shell command.
         cmd = [self.command]
         cmd += ["--" + method]
         cmd += ['--xml', '-p']
+        if after_version(self.version, '0.6.0'):
+            cmd += ['--unit=ebu']            # set units to LU
+            cmd += ['--suppress-progress']   # don't print % to XML output
 
         # Workaround for Windows: the underlying tool fails on paths
         # with the \\?\ prefix, so we don't use it here. This
@@ -286,6 +302,7 @@ def parse_tool_output(self, text, path_list, is_album):
         album_gain = {}  # mutable variable so it can be set from handlers
         parser = xml.parsers.expat.ParserCreate(encoding='utf-8')
         state = {'file': None, 'gain': None, 'peak': None}
+        album_state = {'gain': None, 'peak': None}
 
         def start_element_handler(name, attrs):
             if name == u'track':
@@ -294,9 +311,13 @@ def start_element_handler(name, attrs):
                     raise ReplayGainError(
                         u'duplicate filename in bs1770gain output')
             elif name == u'integrated':
-                state['gain'] = float(attrs[u'lu'])
+                if 'lu' in attrs:
+                    state['gain'] = float(attrs[u'lu'])
             elif name == u'sample-peak':
-                state['peak'] = float(attrs[u'factor'])
+                if 'factor' in attrs:
+                    state['peak'] = float(attrs[u'factor'])
+                elif 'amplitude' in attrs:
+                    state['peak'] = float(attrs[u'amplitude'])
 
         def end_element_handler(name):
             if name == u'track':
@@ -312,6 +333,17 @@ def end_element_handler(name):
                                           'the output of bs1770gain')
                 album_gain["album"] = Gain(state['gain'], state['peak'])
                 state['gain'] = state['peak'] = None
+            elif len(per_file_gain) == len(path_list):
+                if state['gain'] is not None:
+                    album_state['gain'] = state['gain']
+                if state['peak'] is not None:
+                    album_state['peak'] = state['peak']
+                if album_state['gain'] is not None \
+                        and album_state['peak'] is not None:
+                    album_gain["album"] = Gain(
+                        album_state['gain'], album_state['peak'])
+                state['gain'] = state['peak'] = None
+
         parser.StartElementHandler = start_element_handler
         parser.EndElementHandler = end_element_handler
 
diff --git a/docs/changelog.rst b/docs/changelog.rst
index 9976e99a4a..fe3a8e8af2 100644
--- a/docs/changelog.rst
+++ b/docs/changelog.rst
@@ -153,6 +153,8 @@ Fixes:
   :bug:`3437`
 * :doc:`/plugins/lyrics`: Fix a corner-case with Genius lowercase artist names
   :bug:`3446`
+* :doc:`/plugins/replaygain`: Support ``bs1770gain`` v0.6.0 and up
+  :bug:`3480`
 
 For plugin developers:
 


Test patch:
diff --git a/test/test_replaygain.py b/test/test_replaygain.py
index 437b1426aa..969f5c2306 100644
--- a/test/test_replaygain.py
+++ b/test/test_replaygain.py
@@ -58,6 +58,7 @@ def reset_replaygain(item):
 
 
 class ReplayGainCliTestBase(TestHelper):
+
     def setUp(self):
         self.setup_beets()
         self.config['replaygain']['backend'] = self.backend
@@ -150,7 +151,9 @@ def test_cli_saves_album_gain_to_file(self):
         self.assertEqual(max(gains), min(gains))
 
         self.assertNotEqual(max(gains), 0.0)
-        self.assertNotEqual(max(peaks), 0.0)
+        if not self.backend == "bs1770gain":
+            # Actually produces peaks == 0.0 ~ self.add_album_fixture
+            self.assertNotEqual(max(peaks), 0.0)
 
     def test_cli_writes_only_r128_tags(self):
         if self.backend == "command":
@@ -227,7 +230,9 @@ def setUp(self, call_patch):
 
         # Patch call to return nothing, bypassing the bs1770gain installation
         # check.
-        call_patch.return_value = CommandOutput(stdout=b"", stderr=b"")
+        call_patch.return_value = CommandOutput(
+            stdout=b'bs1770gain 0.0.0, ', stderr=b''
+        )
         try:
             self.load_plugins('replaygain')
         except Exception:
@@ -249,7 +254,7 @@ def tearDown(self):
     @patch('beetsplug.replaygain.call')
     def test_malformed_output(self, call_patch):
         # Return malformed XML (the ampersand should be &amp;)
-        call_patch.return_value = CommandOutput(stdout="""
+        call_patch.return_value = CommandOutput(stdout=b"""
             <album>
                 <track total="1" number="1" file="&">
                     <integrated lufs="0" lu="0" />