Repository: beetbox/beets

Issue:
replaygain: bs1770gain backend is associated with white nationalism
### Problem

It appears the creator of the bs1770gain backend used as an optional backend in the replaygain plugin is a staunch white nationalistAnd has dedicated the project "to [his] European heritage". This seems like something beets should distance itself from.
I've included the full quote from the [project website](http://bs1770gain.sourceforge.net) which is linked to in the [replaygain plugin manual](https://beets.readthedocs.io/en/v1.3.17/plugins/replaygain.html#bs1770gain). Clicking through to the creator's [personal site](http://home.snafu.de/pbelkner/) reveals much more of a similar nature.
> Nanos gigantum humeris insidentes: This project is dedicated to my European heritage. It is strictly to be understood as a statement against the saccarine "sweet" liberal lie of "multiculturalism" which is going to destroy Europe as we know it, in particular against the Merkel regime selling out Europe for nothing as we watch. #TeamWhite


Discussion:
"This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 2 of the License, or
(at your option) any later version."

Looks like it's ripe for someone to make a not racist fork. If that is not viable, I would support removing it entirely from beets.
We could consider linking to the [SourceForge.net project page](https://sourceforge.net/projects/bs1770gain/) directly, as opposed to the SourceForge project homepage? It's not ideal, as there's still a link back to the project homepage (and is still "supporting" the author to some extent).

Debian seems to be simply [patching out the message](https://sources.debian.org/src/bs1770gain/0.5.1-3/debian/patches/2000-non-controversial-usage.patch/) (see related issue [here](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=913352)) so it doesn't appear in the tool's usage, which isn't something that really affects us (as our main concern is the documentation link itself).

[It appears that someone did fork it a while back, however it is no longer being maintained.](https://github.com/radiorabe/centos-rpm-bs1770gain), they suggest using [ffmpeg-normalize](https://github.com/slhck/ffmpeg-normalize) instead, which is certainly something we could look into.
Thank you for starting the discussion, @nichobi.

I think we should start by removing this backend from the plugin's documentation. This is an easy step that will avoid linking to the project's website. We can do this right now without breaking anyone's configuration. Would someone mind making a PR along those lines?

Then we can consider just dropping the backend altogether. #3056 (which uses ffmpeg's built-in gain analysis) is a very good candidate for replacing it, with very few downsides—I have fallen down on reviewing that (extremely complete, well written) PR, but any help I can get in doing a code review there would help us merge that new backend.
I've made a pull request removing all references and any configuration options that are unique to this backend. I found no references to it in any other doc pages so hopefully I've got it all.
Thank you!! I left one small comment there.
Too sad that some apparently technically knowledgeable people feel the need to publish politically confused thoughts, misuse the saying "Standing on the shoulders of gaints", and even dedicate whole projects to their nationalism.

If only they could stand back, say about 238,714 miles, and see how laughable it is to draw things like "borders" on this beautiful little golf ball called "Earth", and defend those with their lives … instead of caring for each other and this planet we all call "Home".

I am Terran, born in Germany by pure chance, and I support the removal.
Thanks for the note. We have actually removed this from the docs already in #3130 (so at least we no longer link to the project site).

Resolving patch:
diff --git a/beetsplug/replaygain.py b/beetsplug/replaygain.py
--- a/beetsplug/replaygain.py
+++ b/beetsplug/replaygain.py
@@ -15,26 +15,23 @@
 
 from __future__ import division, absolute_import, print_function
 
-import subprocess
-import os
 import collections
+import enum
 import math
+import os
+import signal
+import six
+import subprocess
 import sys
 import warnings
-import enum
-import re
-import xml.parsers.expat
-from six.moves import zip, queue
-import six
-
 from multiprocessing.pool import ThreadPool, RUN
+from six.moves import zip, queue
 from threading import Thread, Event
-import signal
 
 from beets import ui
 from beets.plugins import BeetsPlugin
-from beets.util import (syspath, command_output, bytestring_path,
-                        displayable_path, py3_path, cpu_count)
+from beets.util import (syspath, command_output, displayable_path,
+                        py3_path, cpu_count)
 
 
 # Utilities.
@@ -136,252 +133,6 @@ def compute_album_gain(self, items, target_level, peak):
         raise NotImplementedError()
 
 
-# bsg1770gain backend
-class Bs1770gainBackend(Backend):
-    """bs1770gain is a loudness scanner compliant with ITU-R BS.1770 and
-    its flavors EBU R128, ATSC A/85 and Replaygain 2.0.
-    """
-
-    methods = {
-        -24: "atsc",
-        -23: "ebu",
-        -18: "replaygain",
-    }
-
-    do_parallel = True
-
-    def __init__(self, config, log):
-        super(Bs1770gainBackend, self).__init__(config, log)
-        config.add({
-            'chunk_at': 5000,
-            'method': '',
-        })
-        self.chunk_at = config['chunk_at'].as_number()
-        # backward compatibility to `method` config option
-        self.__method = config['method'].as_str()
-
-        cmd = 'bs1770gain'
-        try:
-            version_out = call([cmd, '--version'])
-            self.command = cmd
-            self.version = re.search(
-                'bs1770gain ([0-9]+.[0-9]+.[0-9]+), ',
-                version_out.stdout.decode('utf-8')
-            ).group(1)
-        except OSError:
-            raise FatalReplayGainError(
-                u'Is bs1770gain installed?'
-            )
-        if not self.command:
-            raise FatalReplayGainError(
-                u'no replaygain command found: install bs1770gain'
-            )
-
-    def compute_track_gain(self, items, target_level, peak):
-        """Computes the track gain of the given tracks, returns a list
-        of TrackGain objects.
-        """
-
-        output = self.compute_gain(items, target_level, False)
-        return output
-
-    def compute_album_gain(self, items, target_level, peak):
-        """Computes the album gain of the given album, returns an
-        AlbumGain object.
-        """
-        # TODO: What should be done when not all tracks in the album are
-        # supported?
-
-        output = self.compute_gain(items, target_level, True)
-
-        if not output:
-            raise ReplayGainError(u'no output from bs1770gain')
-        return AlbumGain(output[-1], output[:-1])
-
-    def isplitter(self, items, chunk_at):
-        """Break an iterable into chunks of at most size `chunk_at`,
-        generating lists for each chunk.
-        """
-        iterable = iter(items)
-        while True:
-            result = []
-            for i in range(chunk_at):
-                try:
-                    a = next(iterable)
-                except StopIteration:
-                    break
-                else:
-                    result.append(a)
-            if result:
-                yield result
-            else:
-                break
-
-    def compute_gain(self, items, target_level, is_album):
-        """Computes the track or album gain of a list of items, returns
-        a list of TrackGain objects.
-        When computing album gain, the last TrackGain object returned is
-        the album gain
-        """
-
-        if len(items) == 0:
-            return []
-
-        albumgaintot = 0.0
-        albumpeaktot = 0.0
-        returnchunks = []
-
-        # In the case of very large sets of music, we break the tracks
-        # into smaller chunks and process them one at a time. This
-        # avoids running out of memory.
-        if len(items) > self.chunk_at:
-            i = 0
-            for chunk in self.isplitter(items, self.chunk_at):
-                i += 1
-                returnchunk = self.compute_chunk_gain(
-                    chunk,
-                    is_album,
-                    target_level
-                )
-                albumgaintot += returnchunk[-1].gain
-                albumpeaktot = max(albumpeaktot, returnchunk[-1].peak)
-                returnchunks = returnchunks + returnchunk[0:-1]
-            returnchunks.append(Gain(albumgaintot / i, albumpeaktot))
-            return returnchunks
-        else:
-            return self.compute_chunk_gain(items, is_album, target_level)
-
-    def compute_chunk_gain(self, items, is_album, target_level):
-        """Compute ReplayGain values and return a list of results
-        dictionaries as given by `parse_tool_output`.
-        """
-        # choose method
-        target_level = db_to_lufs(target_level)
-        if self.__method != "":
-            # backward compatibility to `method` option
-            method = self.__method
-            gain_adjustment = target_level \
-                - [k for k, v in self.methods.items() if v == method][0]
-        elif target_level in self.methods:
-            method = self.methods[target_level]
-            gain_adjustment = 0
-        else:
-            lufs_target = -23
-            method = self.methods[lufs_target]
-            gain_adjustment = target_level - lufs_target
-
-        # Construct shell command.
-        cmd = [self.command]
-        cmd += ["--" + method]
-        cmd += ['--xml', '-p']
-        if after_version(self.version, '0.6.0'):
-            cmd += ['--unit=ebu']            # set units to LU
-            cmd += ['--suppress-progress']   # don't print % to XML output
-
-        # Workaround for Windows: the underlying tool fails on paths
-        # with the \\?\ prefix, so we don't use it here. This
-        # prevents the backend from working with long paths.
-        args = cmd + [syspath(i.path, prefix=False) for i in items]
-        path_list = [i.path for i in items]
-
-        # Invoke the command.
-        self._log.debug(
-            u'executing {0}', u' '.join(map(displayable_path, args))
-        )
-        output = call(args).stdout
-
-        self._log.debug(u'analysis finished: {0}', output)
-        results = self.parse_tool_output(output, path_list, is_album)
-
-        if gain_adjustment:
-            results = [
-                Gain(res.gain + gain_adjustment, res.peak)
-                for res in results
-            ]
-
-        self._log.debug(u'{0} items, {1} results', len(items), len(results))
-        return results
-
-    def parse_tool_output(self, text, path_list, is_album):
-        """Given the  output from bs1770gain, parse the text and
-        return a list of dictionaries
-        containing information about each analyzed file.
-        """
-        per_file_gain = {}
-        album_gain = {}  # mutable variable so it can be set from handlers
-        parser = xml.parsers.expat.ParserCreate(encoding='utf-8')
-        state = {'file': None, 'gain': None, 'peak': None}
-        album_state = {'gain': None, 'peak': None}
-
-        def start_element_handler(name, attrs):
-            if name == u'track':
-                state['file'] = bytestring_path(attrs[u'file'])
-                if state['file'] in per_file_gain:
-                    raise ReplayGainError(
-                        u'duplicate filename in bs1770gain output')
-            elif name == u'integrated':
-                if 'lu' in attrs:
-                    state['gain'] = float(attrs[u'lu'])
-            elif name == u'sample-peak':
-                if 'factor' in attrs:
-                    state['peak'] = float(attrs[u'factor'])
-                elif 'amplitude' in attrs:
-                    state['peak'] = float(attrs[u'amplitude'])
-
-        def end_element_handler(name):
-            if name == u'track':
-                if state['gain'] is None or state['peak'] is None:
-                    raise ReplayGainError(u'could not parse gain or peak from '
-                                          'the output of bs1770gain')
-                per_file_gain[state['file']] = Gain(state['gain'],
-                                                    state['peak'])
-                state['gain'] = state['peak'] = None
-            elif name == u'summary':
-                if state['gain'] is None or state['peak'] is None:
-                    raise ReplayGainError(u'could not parse gain or peak from '
-                                          'the output of bs1770gain')
-                album_gain["album"] = Gain(state['gain'], state['peak'])
-                state['gain'] = state['peak'] = None
-            elif len(per_file_gain) == len(path_list):
-                if state['gain'] is not None:
-                    album_state['gain'] = state['gain']
-                if state['peak'] is not None:
-                    album_state['peak'] = state['peak']
-                if album_state['gain'] is not None \
-                        and album_state['peak'] is not None:
-                    album_gain["album"] = Gain(
-                        album_state['gain'], album_state['peak'])
-                state['gain'] = state['peak'] = None
-
-        parser.StartElementHandler = start_element_handler
-        parser.EndElementHandler = end_element_handler
-
-        try:
-            parser.Parse(text, True)
-        except xml.parsers.expat.ExpatError:
-            raise ReplayGainError(
-                u'The bs1770gain tool produced malformed XML. '
-                u'Using version >=0.4.10 may solve this problem.')
-
-        if len(per_file_gain) != len(path_list):
-            raise ReplayGainError(
-                u'the number of results returned by bs1770gain does not match '
-                'the number of files passed to it')
-
-        # bs1770gain does not return the analysis results in the order that
-        # files are passed on the command line, because it is sorting the files
-        # internally. We must recover the order from the filenames themselves.
-        try:
-            out = [per_file_gain[os.path.basename(p)] for p in path_list]
-        except KeyError:
-            raise ReplayGainError(
-                u'unrecognized filename in bs1770gain output '
-                '(bs1770gain can only deal with utf-8 file names)')
-        if is_album:
-            out.append(album_gain["album"])
-        return out
-
-
 # ffmpeg backend
 class FfmpegBackend(Backend):
     """A replaygain backend using ffmpeg's ebur128 filter.
@@ -1216,7 +967,6 @@ class ReplayGainPlugin(BeetsPlugin):
         "command": CommandBackend,
         "gstreamer": GStreamerBackend,
         "audiotools": AudioToolsBackend,
-        "bs1770gain": Bs1770gainBackend,
         "ffmpeg": FfmpegBackend,
     }
 


Test patch:
diff --git a/test/test_replaygain.py b/test/test_replaygain.py
--- a/test/test_replaygain.py
+++ b/test/test_replaygain.py
@@ -16,18 +16,14 @@
 
 from __future__ import division, absolute_import, print_function
 
-import unittest
 import six
-
-from mock import patch
-from test.helper import TestHelper, capture_log, has_program
+import unittest
+from mediafile import MediaFile
 
 from beets import config
-from beets.util import CommandOutput
-from mediafile import MediaFile
 from beetsplug.replaygain import (FatalGstreamerPluginReplayGainError,
                                   GStreamerBackend)
-
+from test.helper import TestHelper, has_program
 
 try:
     import gi
@@ -41,11 +37,6 @@
 else:
     GAIN_PROG_AVAILABLE = False
 
-if has_program('bs1770gain'):
-    LOUDNESS_PROG_AVAILABLE = True
-else:
-    LOUDNESS_PROG_AVAILABLE = False
-
 FFMPEG_AVAILABLE = has_program('ffmpeg', ['-version'])
 
 
@@ -153,9 +144,7 @@ def test_cli_saves_album_gain_to_file(self):
         self.assertEqual(max(gains), min(gains))
 
         self.assertNotEqual(max(gains), 0.0)
-        if not self.backend == "bs1770gain":
-            # Actually produces peaks == 0.0 ~ self.add_album_fixture
-            self.assertNotEqual(max(peaks), 0.0)
+        self.assertNotEqual(max(peaks), 0.0)
 
     def test_cli_writes_only_r128_tags(self):
         if self.backend == "command":
@@ -219,62 +208,6 @@ class ReplayGainCmdCliTest(ReplayGainCliTestBase, unittest.TestCase):
     backend = u'command'
 
 
-@unittest.skipIf(not LOUDNESS_PROG_AVAILABLE, u'bs1770gain cannot be found')
-class ReplayGainLdnsCliTest(ReplayGainCliTestBase, unittest.TestCase):
-    backend = u'bs1770gain'
-
-
-class ReplayGainLdnsCliMalformedTest(TestHelper, unittest.TestCase):
-    @patch('beetsplug.replaygain.call')
-    def setUp(self, call_patch):
-        self.setup_beets()
-        self.config['replaygain']['backend'] = 'bs1770gain'
-
-        # Patch call to return nothing, bypassing the bs1770gain installation
-        # check.
-        call_patch.return_value = CommandOutput(
-            stdout=b'bs1770gain 0.0.0, ', stderr=b''
-        )
-        try:
-            self.load_plugins('replaygain')
-        except Exception:
-            import sys
-            exc_info = sys.exc_info()
-            try:
-                self.tearDown()
-            except Exception:
-                pass
-            six.reraise(exc_info[1], None, exc_info[2])
-
-        for item in self.add_album_fixture(2).items():
-            reset_replaygain(item)
-
-    def tearDown(self):
-        self.teardown_beets()
-        self.unload_plugins()
-
-    @patch('beetsplug.replaygain.call')
-    def test_malformed_output(self, call_patch):
-        # Return malformed XML (the ampersand should be &amp;)
-        call_patch.return_value = CommandOutput(stdout=b"""
-            <album>
-                <track total="1" number="1" file="&">
-                    <integrated lufs="0" lu="0" />
-                    <sample-peak spfs="0" factor="0" />
-                </track>
-            </album>
-        """, stderr="")
-
-        with capture_log('beets.replaygain') as logs:
-            self.run_command('replaygain')
-
-        # Count how many lines match the expected error.
-        matching = [line for line in logs if
-                    'malformed XML' in line]
-
-        self.assertEqual(len(matching), 2)
-
-
 @unittest.skipIf(not FFMPEG_AVAILABLE, u'ffmpeg cannot be found')
 class ReplayGainFfmpegTest(ReplayGainCliTestBase, unittest.TestCase):
     backend = u'ffmpeg'