Repository: freedomofpress/securedrop

Issue:
Improve input validation in `manage.py`
# Feature request

## Description

* The [sh](https://github.com/freedomofpress/securedrop/blob/develop/securedrop/manage.py:L35) function as well as the translate function it [helps](https://github.com/freedomofpress/securedrop/blob/develop/securedrop/manage.py:L35) could benefit from further validation of the [messages.pot](https://github.com/freedomofpress/securedrop/blob/develop/securedrop/translations/messages.pot) file before invoking `sh`.

* Two commands appear to be invoked with an unneeded shell
https://github.com/freedomofpress/securedrop/blob/develop/securedrop/manage.py:L315
https://github.com/freedomofpress/securedrop/blob/develop/securedrop/manage.py:L362
However, removing the `shell=True` causes  `tests/test_i18n.py::TestI18N::test_i18n` to fail with `OSError: [Errno 2] No such file or directory.` 


While these features are only used by translation maintainers,  they require maintainers to be diligent in handling translation files while merging them into the codebase (see https://github.com/freedomofpress/securedrop/pull/3042).


## User Stories

As a translations maintainer, I was to ensure that malicious code within a translations file does not not execute unwanted code.


Discussion:
Marking this as `easy` since this is relatively contained to a single part of the code even though solving it might be a bit of a challenge.
I would like to work on this issue.
@usmanmuhd98 consider yourself assigned!
I was going through the code. It is that the shell is invoked for the git operations which do not run when shell=False. To avoid that problem, I think the git library in python may help.
If not, how else should I validate the input?
I would personally reject a PR that pulled in the whole git lib for this because that's too much for this simple thing. I think we can do some manual validation ourselves and (ideally) yank the `shell=True` line from the subprocess call.
@usmanmuhd98 what is the status of this issue? Let me know If you are not working then i would like to give it a shot.

@Anonymous26 I think you can give it a shot since there has not been activity in the past two weeks. @usmanmuhd98 if you're also working on it, please speak up now :-)
@Anonymous26 You can go ahead.
@dachary I am caught up with quite a lot of college work. Maybe I can take up a different issue once I am free.
Presently working on this during PyCon 2018 Sprints!
End of day update on this issue:

Basically, the `sh()` method is referenced in a number of places (including the tests), so swapping to `shell=False` isn't nearly as easy as it would seem on the surface. I've started re-writing a few commands at a time to some success (eg, the `pybabel` call in `translate_messages()`), but a better approach may be to refactor the commands to use `os` python functions for file manipulation and only call `subprocess.Popen` in specific instances.

An even more robust approach would involve directly calling the [`babel.messages` low-level extraction interface](http://babel.pocoo.org/en/latest/api/messages/extract.html) instead of the command-line interface for babel. The API to directly call babel's `extract` isn't as nice as the command-line interface, so it's not as big of a deal. 

I also noted that babel has some explicit documentation for adding it to a distutils setup: http://babel.pocoo.org/en/latest/setup.html#extract-messages which may be an alternative approach to importing translations.

I've only started looking at securedrop's code today, so I may be missing out on some important historic reasons for adding a separate util for handling translations.

One last note: In my humble opinion I don't think this should be labeled as `easy`. There are a few hidden moving parts which complicate this issue.

I'll take another crack at this tomorrow (EDT) and see what I can come up with, but I figured I'd get these observations down in case someone else starts down this issue.
@octaflop I agree this is not easy. The i18n_tool.py helper is meant to abstract the gory details of i18n for scripting. Back when @emkll created this issue, this script was part of manage.py which is a script deployed with SecureDrop and used in production. It deserves a lot of attention and my guess is that it is the primary reason why avoiding using shell snippets was suggested.

However i18n_tool.py was later extracted from manage.py and is never used in production, only for development. It may be worth reconsidering the benefits of removing the shell snippets in favor of more complicated and possibly more secure python equivalents.

Resolving patch:
diff --git a/securedrop/i18n_tool.py b/securedrop/i18n_tool.py
--- a/securedrop/i18n_tool.py
+++ b/securedrop/i18n_tool.py
@@ -5,6 +5,7 @@
 import io
 import logging
 import os
+import glob
 import re
 import signal
 import subprocess
@@ -14,52 +15,12 @@
 
 from os.path import dirname, join, realpath
 
+from sh import git, pybabel, sed, msgmerge, xgettext, msgfmt
+
 logging.basicConfig(format='%(asctime)s %(levelname)s %(message)s')
 log = logging.getLogger(__name__)
 
 
-def sh(command, input=None):
-    """Run the *command* which must be a shell snippet. The stdin is
-    either /dev/null or the *input* argument string.
-
-    The stderr/stdout of the snippet are captured and logged via
-    logging.debug(), one line at a time.
-    """
-    log.debug(":sh: " + command)
-    if input is None:
-        stdin = None
-    else:
-        stdin = subprocess.PIPE
-    proc = subprocess.Popen(
-        args=command,
-        stdin=stdin,
-        stdout=subprocess.PIPE,
-        stderr=subprocess.STDOUT,
-        shell=True,
-        bufsize=1)
-    if stdin is not None:
-        proc.stdin.write(input)
-        proc.stdin.close()
-    lines_of_command_output = []
-    loggable_line_list = []
-    with proc.stdout:
-        for line in iter(proc.stdout.readline, b''):
-            line = line.decode('utf-8')
-            lines_of_command_output.append(line)
-            loggable_line = line.strip().encode('ascii', 'ignore')
-            log.debug(loggable_line)
-            loggable_line_list.append(loggable_line)
-    if proc.wait() != 0:
-        if log.getEffectiveLevel() > logging.DEBUG:
-            for loggable_line in loggable_line_list:
-                log.error(loggable_line)
-        raise subprocess.CalledProcessError(
-            returncode=proc.returncode,
-            cmd=command
-        )
-    return "".join(lines_of_command_output)
-
-
 class I18NTool(object):
 
     #
@@ -92,114 +53,86 @@ def file_is_modified(self, path):
         return subprocess.call(['git', '-C', dir, 'diff', '--quiet', path])
 
     def ensure_i18n_remote(self, args):
-        sh("""
-        set -ex
-        cd {root}
-        if ! git remote | grep --quiet i18n ; then
-           git remote add i18n {url}
-        fi
-        git fetch i18n
-        """.format(root=args.root,
-                   url=args.url))
+        k = {'_cwd': args.root}
+        if 'i18n' not in git.remote(**k).stdout:
+            git.remote.add('i18n', args.url, **k)
+        git.fetch('i18n', **k)
 
     def translate_messages(self, args):
         messages_file = os.path.join(args.translations_dir, 'messages.pot')
 
         if args.extract_update:
-            sh("""
-            set -xe
-
-            mkdir -p {translations_dir}
-
-            pybabel extract \
-            --charset=utf-8 \
-            --mapping={mapping} \
-            --output={messages_file} \
-            --project=SecureDrop \
-            --version={version} \
-            --msgid-bugs-address='securedrop@freedom.press' \
-            --copyright-holder='Freedom of the Press Foundation' \
-            {sources}
-
-            # remove this line so the file does not change if no
-            # strings are modified
-            sed -i '/^"POT-Creation-Date/d' {messages_file}
-            """.format(translations_dir=args.translations_dir,
-                       mapping=args.mapping,
-                       messages_file=messages_file,
-                       version=args.version,
-                       sources=" ".join(args.sources.split(','))))
+            if not os.path.exists(args.translations_dir):
+                os.makedirs(args.translations_dir)
+            sources = args.sources.split(',')
+            pybabel.extract(
+                '--charset=utf-8',
+                '--mapping', args.mapping,
+                '--output', messages_file,
+                '--project=SecureDrop',
+                '--version', args.version,
+                "--msgid-bugs-address=securedrop@freedom.press",
+                "--copyright-holder=Freedom of the Press Foundation",
+                *sources)
+            sed('-i', '-e', '/^"POT-Creation-Date/d', messages_file)
 
             if (self.file_is_modified(messages_file) and
                     len(os.listdir(args.translations_dir)) > 1):
-                sh("""
-                set -xe
-                for translation in {translations_dir}/*/LC_MESSAGES/*.po ; do
-                  msgmerge --previous --update $translation {messages_file}
-                done
-                """.format(translations_dir=args.translations_dir,
-                           messages_file=messages_file))
-                log.warning("messages translations updated in " +
-                            messages_file)
+                tglob = '{}/*/LC_MESSAGES/*.po'.format(args.translations_dir)
+                for translation in glob.iglob(tglob):
+                    msgmerge('--previous', '--update', translation,
+                             messages_file)
+                log.warning("messages translations updated in {}".format(
+                            messages_file))
             else:
                 log.warning("messages translations are already up to date")
 
         if args.compile and len(os.listdir(args.translations_dir)) > 1:
-            sh("""
-            set -x
-            pybabel compile --directory {translations_dir}
-            """.format(translations_dir=args.translations_dir))
+            pybabel.compile('--directory', args.translations_dir)
 
     def translate_desktop(self, args):
         messages_file = os.path.join(args.translations_dir, 'desktop.pot')
 
         if args.extract_update:
-            sh("""
-            set -xe
-            cd {translations_dir}
-            xgettext \
-               --output=desktop.pot \
-               --language=Desktop \
-               --keyword \
-               --keyword=Name \
-               --package-version={version} \
-               --msgid-bugs-address='securedrop@freedom.press' \
-               --copyright-holder='Freedom of the Press Foundation' \
-               {sources}
-
-            # remove this line so the file does not change if no
-            # strings are modified
-            sed -i '/^"POT-Creation-Date/d' {messages_file}
-            """.format(translations_dir=args.translations_dir,
-                       messages_file=messages_file,
-                       version=args.version,
-                       sources=" ".join(args.sources.split(','))))
+            sources = args.sources.split(',')
+            k = {'_cwd': args.translations_dir}
+            xgettext(
+                "--output=desktop.pot",
+                "--language=Desktop",
+                "--keyword",
+                "--keyword=Name",
+                "--package-version", args.version,
+                "--msgid-bugs-address=securedrop@freedom.press",
+                "--copyright-holder=Freedom of the Press Foundation",
+                *sources,
+                **k)
+            sed('-i', '-e', '/^"POT-Creation-Date/d', messages_file, **k)
 
             if self.file_is_modified(messages_file):
                 for f in os.listdir(args.translations_dir):
                     if not f.endswith('.po'):
                         continue
                     po_file = os.path.join(args.translations_dir, f)
-                    sh("""
-                    msgmerge --update {po_file} {messages_file}
-                    """.format(po_file=po_file,
-                               messages_file=messages_file))
+                    msgmerge('--update', po_file, messages_file)
                 log.warning("messages translations updated in " +
                             messages_file)
             else:
                 log.warning("desktop translations are already up to date")
 
         if args.compile:
-            sh("""
-            set -ex
-            cd {translations_dir}
-            find *.po | sed -e 's/\.po$//' > LINGUAS
-            for source in {sources} ; do
-               target=$(basename $source .in)
-               msgfmt --desktop --template $source -o $target -d .
-            done
-            """.format(translations_dir=args.translations_dir,
-                       sources=" ".join(args.sources.split(','))))
+            pos = filter(lambda f: f.endswith('.po'),
+                         os.listdir(args.translations_dir))
+            linguas = map(lambda l: l.rstrip('.po'), pos)
+            content = "\n".join(linguas) + "\n"
+            open(join(args.translations_dir, 'LINGUAS'), 'w').write(content)
+
+            for source in args.sources.split(','):
+                target = source.rstrip('.in')
+                msgfmt('--desktop',
+                       '--template', source,
+                       '-o', target,
+                       '-d', '.',
+                       _cwd=args.translations_dir)
 
     def set_translate_parser(self,
                              subps,
@@ -280,16 +213,13 @@ def update_docs(self, args):
         io.open(l10n_txt, mode='w').write(l10n_content)
         self.require_git_email_name(includes)
         if self.file_is_modified(l10n_txt):
-            sh("""
-            set -ex
-            cd {includes}
-            git add l10n.txt
-            git commit \
-              -m 'docs: update the list of supported languages' \
-              l10n.txt
-            """.format(includes=includes))
+            k = {'_cwd': includes}
+            git.add('l10n.txt', **k)
+            msg = 'docs: update the list of supported languages'
+            git.commit('-m', msg, 'l10n.txt', **k)
             log.warning(l10n_txt + " updated")
-            log.warning(sh("cd " + includes + "; git show"))
+            git_show_out = git.show(**k)
+            log.warning(git_show_out)
         else:
             log.warning(l10n_txt + " already up to date")
 
@@ -314,19 +244,17 @@ def update_from_weblate(self, args):
 
             def need_update(p):
                 exists = os.path.exists(join(args.root, p))
-                sh("""
-                set -ex
-                cd {r}
-                git checkout i18n/i18n -- {p}
-                git reset HEAD -- {p}
-                """.format(r=args.root, p=p))
+                k = {'_cwd': args.root}
+                git.checkout('i18n/i18n', '--', p, **k)
+                git.reset('HEAD', '--', p, **k)
                 if not exists:
                     return True
                 else:
                     return self.file_is_modified(join(args.root, p))
 
             def add(p):
-                sh("git -C {r} add {p}".format(r=args.root, p=p))
+                git('-C', args.root, 'add', p)
+
             updated = False
             #
             # Update messages
@@ -353,20 +281,24 @@ def add(p):
     def upstream_commit(self, args, code):
         self.require_git_email_name(args.root)
         authors = set()
-        for path in sh("git -C {r} diff --name-only --cached".format(
-                r=args.root)).split():
-            previous_message = sh("git -C {r} log -n 1 {p}".format(
-                r=args.root, p=path))
-            m = re.search('copied from (\w+)', previous_message)
+        diffs = git('--no-pager', '-C', args.root,
+                    'diff', '--name-only', '--cached').stdout
+        for path in diffs.strip().split('\n'):
+            previous_message = unicode(git(
+                '--no-pager', '-C', args.root, 'log', '-n', '1', path,
+                _encoding='utf-8'))
+            m = re.search(u'copied from (\w+)', previous_message)
             if m:
                 origin = m.group(1)
             else:
                 origin = ''
-            authors |= set(sh("""
-            git -C {r} log --format=%aN {o}..i18n/i18n -- {p}
-            """.format(r=args.root, o=origin, p=path)).strip().split('\n'))
-        current = sh("git -C {r} rev-parse i18n/i18n".format(
-            r=args.root)).strip()
+            git_authors = unicode(git(
+                '--no-pager', '-C', args.root, 'log', '--format=%aN',
+                '{}..i18n/i18n'.format(origin), '--',
+                path, _encoding='utf-8'))
+            git_authors = git_authors.strip().split(u'\n')
+            authors |= set(git_authors)
+        current = git('-C', args.root, 'rev-parse', 'i18n/i18n').stdout
         info = I18NTool.SUPPORTED_LANGUAGES[code]
         message = textwrap.dedent(u"""
         l10n: updated {code} {name}
@@ -377,11 +309,10 @@ def upstream_commit(self, args, code):
         copied from {current}
         """.format(remote=args.url,
                    name=info['name'],
-                   authors=", ".join(authors),
+                   authors=u", ".join(authors),
                    code=code,
                    current=current))
-        sh(u'git -C {r} commit -m "{message}"'.format(
-            r=args.root, message=message.replace('"', '\"')).encode('utf-8'))
+        git('-C', args.root, 'commit', '-m', message)
 
     def set_update_from_weblate_parser(self, subps):
         parser = subps.add_parser('update-from-weblate',
@@ -419,6 +350,7 @@ def get_args(self):
 
     def setup_verbosity(self, args):
         if args.verbose:
+            logging.getLogger('sh.command').setLevel(logging.INFO)
             log.setLevel(logging.DEBUG)
         else:
             log.setLevel(logging.INFO)


Test patch:
diff --git a/securedrop/tests/test_i18n.py b/securedrop/tests/test_i18n.py
--- a/securedrop/tests/test_i18n.py
+++ b/securedrop/tests/test_i18n.py
@@ -31,6 +31,8 @@
 import pytest
 import source_app
 
+from sh import sed, pybabel
+
 
 def verify_i18n(app):
     not_translated = 'code hello i18n'
@@ -185,29 +187,19 @@ def test_i18n(journalist_app, config):
         '--extract-update',
     ])
 
-    i18n_tool.sh("""
-    pybabel init -i {d}/messages.pot -d {d} -l en_US
-
-    pybabel init -i {d}/messages.pot -d {d} -l fr_FR
-    sed -i -e '/code hello i18n/,+1s/msgstr ""/msgstr "code bonjour"/' \
-          {d}/fr_FR/LC_MESSAGES/messages.po
-
-    pybabel init -i {d}/messages.pot -d {d} -l zh_Hans_CN
-    sed -i -e '/code hello i18n/,+1s/msgstr ""/msgstr "code chinese"/' \
-          {d}/zh_Hans_CN/LC_MESSAGES/messages.po
-
-    pybabel init -i {d}/messages.pot -d {d} -l ar
-    sed -i -e '/code hello i18n/,+1s/msgstr ""/msgstr "code arabic"/' \
-          {d}/ar/LC_MESSAGES/messages.po
-
-    pybabel init -i {d}/messages.pot -d {d} -l nb_NO
-    sed -i -e '/code hello i18n/,+1s/msgstr ""/msgstr "code norwegian"/' \
-          {d}/nb_NO/LC_MESSAGES/messages.po
-
-    pybabel init -i {d}/messages.pot -d {d} -l es_ES
-    sed -i -e '/code hello i18n/,+1s/msgstr ""/msgstr "code spanish"/' \
-          {d}/es_ES/LC_MESSAGES/messages.po
-    """.format(d=config.TEMP_DIR))
+    pot = os.path.join(config.TEMP_DIR, 'messages.pot')
+    pybabel('init', '-i', pot, '-d', config.TEMP_DIR, '-l', 'en_US')
+
+    for (l, s) in (('fr_FR', 'code bonjour'),
+                   ('zh_Hans_CN', 'code chinese'),
+                   ('ar', 'code arabic'),
+                   ('nb_NO', 'code norwegian'),
+                   ('es_ES', 'code spanish')):
+        pybabel('init', '-i', pot, '-d', config.TEMP_DIR, '-l', l)
+        po = os.path.join(config.TEMP_DIR, l, 'LC_MESSAGES/messages.po')
+        sed('-i', '-e',
+            '/code hello i18n/,+1s/msgstr ""/msgstr "{}"/'.format(s),
+            po)
 
     i18n_tool.I18NTool().main([
         '--verbose',
diff --git a/securedrop/tests/test_i18n_tool.py b/securedrop/tests/test_i18n_tool.py
--- a/securedrop/tests/test_i18n_tool.py
+++ b/securedrop/tests/test_i18n_tool.py
@@ -4,15 +4,15 @@
 import os
 from os.path import abspath, dirname, exists, getmtime, join, realpath
 os.environ['SECUREDROP_ENV'] = 'test'  # noqa
-import logging
 import i18n_tool
 from mock import patch
 import pytest
 import shutil
 import signal
-import subprocess
 import time
 
+from sh import sed, msginit, pybabel, git, touch
+
 
 class TestI18NTool(object):
 
@@ -32,21 +32,6 @@ def test_main(self, tmpdir, caplog):
                 '--translations-dir', str(tmpdir)
             ]) == signal.SIGINT
 
-        assert tool.main([
-            'translate-messages',
-            '--translations-dir', str(tmpdir),
-            '--extract-update'
-        ]) is None
-        assert 'pybabel extract' not in caplog.text
-
-        assert tool.main([
-            '--verbose',
-            'translate-messages',
-            '--translations-dir', str(tmpdir),
-            '--extract-update'
-        ]) is None
-        assert 'pybabel extract' in caplog.text
-
     def test_translate_desktop_l10n(self, tmpdir):
         in_files = {}
         for what in ('source', 'journalist'):
@@ -89,17 +74,15 @@ def test_translate_desktop_l10n(self, tmpdir):
 
         locale = 'fr_FR'
         po_file = join(str(tmpdir), locale + ".po")
-        i18n_tool.sh("""
-        msginit  --no-translator \
-                 --locale {locale} \
-                 --output {po_file} \
-                 --input {messages_file}
-        sed -i -e '/{source}/,+1s/msgstr ""/msgstr "SOURCE FR"/' \
-                 {po_file}
-        """.format(source='SecureDrop Source Interfaces',
-                   messages_file=messages_file,
-                   po_file=po_file,
-                   locale=locale))
+        msginit(
+            '--no-translator',
+            '--locale', locale,
+            '--output', po_file,
+            '--input', messages_file)
+        source = 'SecureDrop Source Interfaces'
+        sed('-i', '-e',
+            '/{}/,+1s/msgstr ""/msgstr "SOURCE FR"/'.format(source),
+            po_file)
         assert exists(po_file)
 
         #
@@ -146,11 +129,7 @@ def test_translate_messages_l10n(self, tmpdir):
 
         locale = 'en_US'
         locale_dir = join(str(tmpdir), locale)
-        i18n_tool.sh("pybabel init -i {} -d {} -l {}".format(
-            messages_file,
-            str(tmpdir),
-            locale,
-        ))
+        pybabel('init', '-i', messages_file, '-d', str(tmpdir), '-l', locale)
         mo_file = join(locale_dir, 'LC_MESSAGES/messages.mo')
         assert not exists(mo_file)
         i18n_tool.I18NTool().main(args)
@@ -180,11 +159,7 @@ def test_translate_messages_compile_arg(self, tmpdir):
         locale = 'en_US'
         locale_dir = join(str(tmpdir), locale)
         po_file = join(locale_dir, 'LC_MESSAGES/messages.po')
-        i18n_tool.sh("pybabel init -i {} -d {} -l {}".format(
-            messages_file,
-            str(tmpdir),
-            locale,
-        ))
+        pybabel(['init', '-i', messages_file, '-d', str(tmpdir), '-l', locale])
         assert exists(po_file)
         # pretend this happened a few seconds ago
         few_seconds_ago = time.time() - 60
@@ -224,32 +199,26 @@ def test_translate_messages_compile_arg(self, tmpdir):
             assert 'template hello i18n' not in mo
 
     def test_require_git_email_name(self, tmpdir):
-        i18n_tool.sh("""
-        cd {dir}
-        git init
-        """.format(dir=str(tmpdir)))
+        k = {'_cwd': str(tmpdir)}
+        git('init', **k)
         with pytest.raises(Exception) as excinfo:
             i18n_tool.I18NTool.require_git_email_name(str(tmpdir))
         assert 'please set name' in excinfo.value.message
-        i18n_tool.sh("""
-        cd {dir}
-        git config user.email "you@example.com"
-        git config user.name "Your Name"
-        """.format(dir=str(tmpdir)))
+
+        git.config('user.email', "you@example.com", **k)
+        git.config('user.name', "Your Name", **k)
         assert i18n_tool.I18NTool.require_git_email_name(str(tmpdir))
 
     def test_update_docs(self, tmpdir, caplog):
-        os.makedirs(join(str(tmpdir), 'includes'))
-        i18n_tool.sh("""
-        cd {dir}
-        git init
-        git config user.email "you@example.com"
-        git config user.name "Your Name"
-        mkdir includes
-        touch includes/l10n.txt
-        git add includes/l10n.txt
-        git commit -m 'init'
-        """.format(dir=str(tmpdir)))
+        k = {'_cwd': str(tmpdir)}
+        git.init(**k)
+        git.config('user.email', "you@example.com", **k)
+        git.config('user.name', "Your Name", **k)
+        os.mkdir(join(str(tmpdir), 'includes'))
+        touch('includes/l10n.txt', **k)
+        git.add('includes/l10n.txt', **k)
+        git.commit('-m', 'init', **k)
+
         i18n_tool.I18NTool().main([
             '--verbose',
             'update-docs',
@@ -264,25 +233,25 @@ def test_update_docs(self, tmpdir, caplog):
 
     def test_update_from_weblate(self, tmpdir, caplog):
         d = str(tmpdir)
-        i18n_tool.sh("""
-        set -ex
-        for r in i18n securedrop ; do
-           mkdir {d}/$r
-           cd {d}/$r
-           git init
-           git config user.email "you@example.com"
-           git config user.name "Loïc Nordhøy"
-           touch README.md
-           git add README.md
-           git commit -m 'README' README.md
-        done
-        cp -a {o}/i18n/* {d}/i18n
-        cd {d}/i18n
-        git add securedrop install_files
-        git commit -m 'init' -a
-        git checkout -b i18n master
-        """.format(o=self.dir,
-                   d=d))
+        for repo in ('i18n', 'securedrop'):
+            os.mkdir(join(d, repo))
+            k = {'_cwd': join(d, repo)}
+            git.init(**k)
+            git.config('user.email', 'you@example.com', **k)
+            git.config('user.name',  u'Loïc Nordhøy', **k)
+            touch('README.md', **k)
+            git.add('README.md', **k)
+            git.commit('-m', 'README', 'README.md', **k)
+        for o in os.listdir(join(self.dir, 'i18n')):
+            f = join(self.dir, 'i18n', o)
+            if os.path.isfile(f):
+                shutil.copyfile(f, join(d, 'i18n', o))
+            else:
+                shutil.copytree(f, join(d, 'i18n', o))
+        k = {'_cwd': join(d, 'i18n')}
+        git.add('securedrop', 'install_files', **k)
+        git.commit('-m', 'init', '-a', **k)
+        git.checkout('-b', 'i18n', 'master', **k)
 
         def r():
             return "".join([str(l) for l in caplog.records])
@@ -330,26 +299,24 @@ def r():
         ])
         assert 'l10n: updated nl' not in r()
         assert 'l10n: updated de_DE' not in r()
-        message = i18n_tool.sh("git -C {d}/securedrop show".format(d=d))
+        message = unicode(git('--no-pager', '-C', 'securedrop', 'show',
+                              _cwd=d, _encoding='utf-8'))
         assert u"Loïc" in message
 
         #
         # an update is done to nl in weblate
         #
-        i18n_tool.sh("""
-        set -ex
-        cd {d}/i18n
-        f=securedrop/translations/nl/LC_MESSAGES/messages.po
-        sed -i -e 's/inactiviteit/INACTIVITEIT/' $f
-        git add $f
-        git config user.email "somone@else.com"
-        git config user.name "Someone Else"
-        git commit -m 'translation change' $f
-
-        cd {d}/securedrop
-        git config user.email "somone@else.com"
-        git config user.name "Someone Else"
-        """.format(d=d))
+        k = {'_cwd': join(d, 'i18n')}
+        f = 'securedrop/translations/nl/LC_MESSAGES/messages.po'
+        sed('-i', '-e', 's/inactiviteit/INACTIVITEIT/', f, **k)
+        git.add(f, **k)
+        git.config('user.email', 'somone@else.com', **k)
+        git.config('user.name', 'Someone Else', **k)
+        git.commit('-m', 'translation change', f, **k)
+
+        k = {'_cwd': join(d, 'securedrop')}
+        git.config('user.email', 'somone@else.com', **k)
+        git.config('user.name', 'Someone Else', **k)
 
         #
         # the nl translation update from weblate is copied
@@ -365,42 +332,7 @@ def r():
         ])
         assert 'l10n: updated nl' in r()
         assert 'l10n: updated de_DE' not in r()
-        message = i18n_tool.sh("git -C {d}/securedrop show".format(d=d))
+        message = unicode(git('--no-pager', '-C', 'securedrop', 'show',
+                              _cwd=d))
         assert "Someone Else" in message
         assert u"Loïc" not in message
-
-
-class TestSh(object):
-
-    def test_sh(self):
-        assert 'A' == i18n_tool.sh("echo -n A")
-        with pytest.raises(Exception) as excinfo:
-            i18n_tool.sh("exit 123")
-        assert excinfo.value.returncode == 123
-
-    def test_sh_progress(self, caplog):
-        i18n_tool.sh("echo AB ; sleep 5 ; echo C")
-        records = caplog.records
-        assert ':sh: ' in records[0].message
-        assert records[0].levelname == 'DEBUG'
-        assert 'AB' == records[1].message
-        assert records[1].levelname == 'DEBUG'
-        assert 'C' == records[2].message
-        assert records[2].levelname == 'DEBUG'
-
-    def test_sh_input(self, caplog):
-        assert 'abc' == i18n_tool.sh("cat", 'abc')
-
-    def test_sh_fail(self, caplog):
-        level = i18n_tool.log.getEffectiveLevel()
-        i18n_tool.log.setLevel(logging.INFO)
-        assert i18n_tool.log.getEffectiveLevel() == logging.INFO
-        with pytest.raises(subprocess.CalledProcessError) as excinfo:
-            i18n_tool.sh("echo AB ; echo C ; exit 111")
-        i18n_tool.log.setLevel(level)
-        assert excinfo.value.returncode == 111
-        records = caplog.records
-        assert 'AB' == records[0].message
-        assert records[0].levelname == 'ERROR'
-        assert 'C' == records[1].message
-        assert records[1].levelname == 'ERROR'
diff --git a/securedrop/tests/test_template_filters.py b/securedrop/tests/test_template_filters.py
--- a/securedrop/tests/test_template_filters.py
+++ b/securedrop/tests/test_template_filters.py
@@ -11,6 +11,8 @@
 import source_app
 import template_filters
 
+from sh import pybabel
+
 
 def verify_rel_datetime_format(app):
     with app.test_client() as c:
@@ -103,10 +105,9 @@ def do_test(config, create_app):
         '--compile',
     ])
 
-    i18n_tool.sh("""
-    pybabel init -i {d}/messages.pot -d {d} -l en_US
-    pybabel init -i {d}/messages.pot -d {d} -l fr_FR
-    """.format(d=config.TEMP_DIR))
+    for l in ('en_US', 'fr_FR'):
+        pot = os.path.join(config.TEMP_DIR, 'messages.pot')
+        pybabel('init', '-i', pot, '-d', config.TEMP_DIR, '-l', l)
 
     app = create_app(config)