internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: freedomofpress/securedrop

Issue:
Installation does not like having no sasl_domain
# Bug

## Description

Our mail server auths with just a username, not a username and a domain.

## Steps to Reproduce

set sasl_domain to "" in the site-specific vars
run `./securedrop-admin install`

## Expected Behavior

 - pass validation
 - installs a config at /etc/postfix/sasl_password that looks like

```
[$smtp_relay]:$smtp_port      $sasl_username:$sasl_password
```

## Actual Behavior

 - Fails validation by trying to nslookup empty string
 - writes out a config with an `@` after the sasl_username

## Comments

I think the `@` is ok, it just looks super weird. It seems to be working though.


Discussion:
Thanks, @icco! During QA we typically only test with 1 or 2 mail setups, and that's clearly not enough. When we have a patch to address the problem you're describing, we'll ping you for review to make sure it works for your setup, as well as what we're already testing against. 

Resolving patch:
diff --git a/admin/securedrop_admin/__init__.py b/admin/securedrop_admin/__init__.py
--- a/admin/securedrop_admin/__init__.py
+++ b/admin/securedrop_admin/__init__.py
@@ -266,7 +266,7 @@ def __init__(self, args):
              int],
             ['sasl_domain', "gmail.com", str,
              u'SASL domain for sending OSSEC alerts',
-             SiteConfig.ValidateNotEmpty(),
+             None,
              None],
             ['sasl_username', '', str,
              u'SASL username for sending OSSEC alerts',


Test patch:
diff --git a/admin/tests/test_securedrop-admin.py b/admin/tests/test_securedrop-admin.py
--- a/admin/tests/test_securedrop-admin.py
+++ b/admin/tests/test_securedrop-admin.py
@@ -367,13 +367,19 @@ def verify_prompt_fingerprint(self, site_config, desc):
         clean_fpr = site_config.sanitize_fingerprint(fpr)
         assert site_config.user_prompt_config_one(desc, fpr) == clean_fpr
 
+    def verify_desc_consistency_allow_empty(self, site_config, desc):
+        (var, default, etype, prompt, validator, transform) = desc
+        # verify the default passes validation
+        assert site_config.user_prompt_config_one(desc, None) == default
+        assert type(default) == etype
+
     verify_prompt_securedrop_app_gpg_fingerprint = verify_prompt_fingerprint
     verify_prompt_ossec_alert_gpg_public_key = verify_desc_consistency
     verify_prompt_ossec_gpg_fpr = verify_prompt_fingerprint
     verify_prompt_ossec_alert_email = verify_prompt_not_empty
     verify_prompt_smtp_relay = verify_prompt_not_empty
     verify_prompt_smtp_relay_port = verify_desc_consistency
-    verify_prompt_sasl_domain = verify_desc_consistency
+    verify_prompt_sasl_domain = verify_desc_consistency_allow_empty
     verify_prompt_sasl_username = verify_prompt_not_empty
     verify_prompt_sasl_password = verify_prompt_not_empty
Repository: freedomofpress/securedrop

Issue:
[RFE] Add type annotation
# Feature request

## Description

We need to add the initial type annotation in the Python code for #2795 





Discussion:

Resolving patch:
diff --git a/securedrop/crypto_util.py b/securedrop/crypto_util.py
--- a/securedrop/crypto_util.py
+++ b/securedrop/crypto_util.py
@@ -9,6 +9,7 @@
 import gnupg
 from gnupg._util import _is_stream, _make_binary_stream
 import scrypt
+from typing import Dict, List, Text  # noqa: F401
 
 import config
 import store
@@ -49,7 +50,7 @@ def do_runtime_tests():
 gpg = gnupg.GPG(binary='gpg2', homedir=config.GPG_KEY_DIR)
 
 # map code for a given language to a localized wordlist
-language2words = {}
+language2words = {}  # type: Dict[Text,List[str]]
 nouns = open(config.NOUNS).read().rstrip('\n').split('\n')
 adjectives = open(config.ADJECTIVES).read().rstrip('\n').split('\n')
 
@@ -80,6 +81,7 @@ def clean(s, also=''):
 
 
 def _get_wordlist(locale):
+    # type: (Text) -> List[str]
     """" Ensure the wordlist for the desired locale is read and available
     in the words global variable. If there is no wordlist for the
     desired local, fallback to the default english wordlist.
diff --git a/securedrop/journalist_app/utils.py b/securedrop/journalist_app/utils.py
--- a/securedrop/journalist_app/utils.py
+++ b/securedrop/journalist_app/utils.py
@@ -20,6 +20,7 @@
 
 
 def logged_in():
+    # type: () -> bool
     # When a user is logged in, we push their user ID (database primary key)
     # into the session. setup_g checks for this value, and if it finds it,
     # stores a reference to the user's Journalist object in g.
diff --git a/securedrop/models.py b/securedrop/models.py
--- a/securedrop/models.py
+++ b/securedrop/models.py
@@ -7,7 +7,7 @@
 try:
     from cStringIO import StringIO
 except ImportError:
-    from StringIO import StringIO
+    from StringIO import StringIO  # type: ignore
 
 from sqlalchemy import ForeignKey
 from sqlalchemy.orm import relationship, backref


Test patch:
Repository: freedomofpress/securedrop

Issue:
Ensure every call to `open` uses a context manager
# Feature request

## Description

There are places in the code where we call `foo = open(some_file).read()`. This could lead dangling resources if an exception is raised. This should be:

```python
with open(some_file) as f:
    foo = f.read()
```


Discussion:

Resolving patch:
diff --git a/install_files/ansible-base/roles/tails-config/files/securedrop_init.py b/install_files/ansible-base/roles/tails-config/files/securedrop_init.py
--- a/install_files/ansible-base/roles/tails-config/files/securedrop_init.py
+++ b/install_files/ansible-base/roles/tails-config/files/securedrop_init.py
@@ -20,24 +20,29 @@
 
 # load torrc_additions
 if os.path.isfile(path_torrc_additions):
-    torrc_additions = open(path_torrc_additions).read()
+    with open(path_torrc_additions) as f:
+        torrc_additions = f.read()
 else:
     sys.exit('Error opening {0} for reading'.format(path_torrc_additions))
 
 # load torrc
 if os.path.isfile(path_torrc_backup):
-    torrc = open(path_torrc_backup).read()
+    with open(path_torrc_backup) as f:
+        torrc = f.read()
 else:
     if os.path.isfile(path_torrc):
-        torrc = open(path_torrc).read()
+        with open(path_torrc) as f:
+            torrc = f.read()
     else:
         sys.exit('Error opening {0} for reading'.format(path_torrc))
 
     # save a backup
-    open(path_torrc_backup, 'w').write(torrc)
+    with open(path_torrc_backup, 'w') as f:
+        f.write(torrc)
 
 # append the additions
-open(path_torrc, 'w').write(torrc + torrc_additions)
+with open(path_torrc, 'w') as f:
+    f.write(torrc + torrc_additions)
 
 # reload tor
 try:
diff --git a/securedrop/source_app/main.py b/securedrop/source_app/main.py
--- a/securedrop/source_app/main.py
+++ b/securedrop/source_app/main.py
@@ -73,9 +73,11 @@ def lookup():
                 reply.filename,
             )
             try:
+                with open(reply_path) as f:
+                    contents = f.read()
                 reply.decrypted = current_app.crypto_util.decrypt(
                     g.codename,
-                    open(reply_path).read()).decode('utf-8')
+                    contents).decode('utf-8')
             except UnicodeDecodeError:
                 current_app.logger.error("Could not decode reply %s" %
                                          reply.filename)
diff --git a/securedrop/source_app/utils.py b/securedrop/source_app/utils.py
--- a/securedrop/source_app/utils.py
+++ b/securedrop/source_app/utils.py
@@ -60,7 +60,8 @@ def generate_unique_codename(config):
 
 
 def get_entropy_estimate():
-    return int(open('/proc/sys/kernel/random/entropy_avail').read())
+    with open('/proc/sys/kernel/random/entropy_avail') as f:
+        return int(f.read())
 
 
 def async(f):


Test patch:
Repository: freedomofpress/securedrop

Issue:
[0.6.0] duplicated help string for update and check_for_updates
## Description

Help strings are accidentally duplicated for two subcommands of `securedrop-admin`

## Steps to Reproduce

`./securedrop-admin -h`

## Expected Behavior

You get a unique string describing each possible command

## Actual Behavior

```
...
    update              Verify, and apply latest SecureDrop workstation update
    check_for_updates   Verify, and apply latest SecureDrop workstation update
...
```




Discussion:

Resolving patch:
diff --git a/admin/securedrop_admin/__init__.py b/admin/securedrop_admin/__init__.py
--- a/admin/securedrop_admin/__init__.py
+++ b/admin/securedrop_admin/__init__.py
@@ -590,7 +590,7 @@ class ArgParseFormatterCombo(argparse.ArgumentDefaultsHelpFormatter,
     parse_update.set_defaults(func=update)
 
     parse_check_updates = subparsers.add_parser('check_for_updates',
-                                                help=update.__doc__)
+                                                help=check_for_updates.__doc__)
     parse_check_updates.set_defaults(func=check_for_updates)
 
     parse_logs = subparsers.add_parser('logs',


Test patch:
Repository: freedomofpress/securedrop

Issue:
[RFE] Do not show any ANSI escape sequence in the GUI outout
# Feature request

## Description

Do not show any ANSI escape sequence in the journalist updater GUI. Based on the comment [here](https://github.com/freedomofpress/securedrop/pull/3300#discussion_r184053702)



Discussion:

Resolving patch:
diff --git a/journalist_gui/journalist_gui/SecureDropUpdater.py b/journalist_gui/journalist_gui/SecureDropUpdater.py
--- a/journalist_gui/journalist_gui/SecureDropUpdater.py
+++ b/journalist_gui/journalist_gui/SecureDropUpdater.py
@@ -3,12 +3,14 @@
 from PyQt5.QtCore import QThread, pyqtSignal
 import subprocess
 import os
+import re
 import pexpect
 
 from journalist_gui import updaterUI, strings, resources_rc  # noqa
 
 
 FLAG_LOCATION = "/home/amnesia/Persistent/.securedrop/securedrop_update.flag"  # noqa
+ESCAPE_POD = re.compile(r'\x1B\[[0-?]*[ -/]*[@-~]')
 
 
 class SetupThread(QThread):
@@ -121,7 +123,7 @@ def run(self):
             self.update_success = False
             self.failure_reason = strings.tailsconfig_failed_generic_reason
         result = {'status': self.update_success,
-                  'output': self.output,
+                  'output': ESCAPE_POD.sub('', self.output),
                   'failure_reason': self.failure_reason}
         self.signal.emit(result)
 


Test patch:
Repository: freedomofpress/securedrop

Issue:
[RFE] Remove the sizegrip from the statusbar of the GUI tool
# Feature request

## Description

There should not be any sizegrip available in the statusbar of the GUI tool as suggested in https://github.com/freedomofpress/securedrop/pull/3300#discussion_r184066922


Discussion:

Resolving patch:
diff --git a/journalist_gui/journalist_gui/SecureDropUpdater.py b/journalist_gui/journalist_gui/SecureDropUpdater.py
--- a/journalist_gui/journalist_gui/SecureDropUpdater.py
+++ b/journalist_gui/journalist_gui/SecureDropUpdater.py
@@ -130,6 +130,7 @@ class UpdaterApp(QtWidgets.QMainWindow, updaterUI.Ui_MainWindow):
     def __init__(self, parent=None):
         super(UpdaterApp, self).__init__(parent)
         self.setupUi(self)
+        self.statusbar.setSizeGripEnabled(False)
         self.output = strings.initial_text_box
         self.plainTextEdit.setPlainText(self.output)
         self.update_success = False


Test patch:
Repository: freedomofpress/securedrop

Issue:
securedrop-admin sdconfig is allowing journalist notification email address without gpg key
# Bug

## Description

`./securedrop-admin sdconfig` allows to put a journalist notification email address without mentioning any journalist GPG key file, or the fingerprint.

## Steps to Reproduce

`./securedrop-admin sdconfig`

## Expected Behavior

If the journalist notification GPG key is not given, then it should skip next two questions.

## Actual Behavior

It allows the admin to add a journalist notification email address.

```
Local filepath to journalist alerts GPG public key (optional): 
Full fingerprint for the journalist alerts GPG public key (optional): 
Email address for receiving journalist alerts (optional): *******@gmail.com
```



Discussion:
The sdconfig implementation unfortunately does not allow for contextual behavior. It should be improved to go beyond asking a set of questions and select which questions should be asked depending on the previous answers.

I tend to agree this is a usability bug although it could be seen as a feature request.

Resolving patch:
diff --git a/admin/securedrop_admin/__init__.py b/admin/securedrop_admin/__init__.py
--- a/admin/securedrop_admin/__init__.py
+++ b/admin/securedrop_admin/__init__.py
@@ -247,6 +247,7 @@ def validate(self, document):
 
     def __init__(self, args):
         self.args = args
+        self.config = {}
         translations = SiteConfig.Locales(
             self.args.app_path).get_translations()
         translations = " ".join(translations)
@@ -352,8 +353,7 @@ def __init__(self, args):
     def load_and_update_config(self):
         if self.exists():
             self.config = self.load()
-        else:
-            self.config = {}
+
         return self.update_config()
 
     def update_config(self):
@@ -367,6 +367,16 @@ def user_prompt_config(self):
         config = {}
         for desc in self.desc:
             (var, default, type, prompt, validator, transform) = desc
+            if var == 'journalist_gpg_fpr':
+                if not config.get('journalist_alert_gpg_public_key',
+                                  None):
+                    config[var] = ''
+                    continue
+            if var == 'journalist_alert_email':
+                if not config.get('journalist_alert_gpg_public_key',
+                                  None):
+                    config[var] = ''
+                    continue
             config[var] = self.user_prompt_config_one(desc,
                                                       self.config.get(var))
         return config


Test patch:
Repository: freedomofpress/securedrop

Issue:
Docker dev env still uses the term "Document Interface"
# Bug

## Description

We changed the term to "Journalist Interface" a while ago, but the Docker dev env still lists it as "Document Interface":

```
 => Document Interface <= 

 * Running on http://0.0.0.0:8081/ (Press CTRL+C to quit)
 * Restarting with stat
```

# User Stories

As a developer, I don't want to be confused by outdated terminology, so I can be more productive.


Discussion:

Resolving patch:
diff --git a/securedrop/management/run.py b/securedrop/management/run.py
--- a/securedrop/management/run.py
+++ b/securedrop/management/run.py
@@ -170,7 +170,7 @@ def run(args):  # pragma: no cover
         lambda: DevServerProcess('Source Interface',
                                  ['python', 'source.py'],
                                  'blue'),
-        lambda: DevServerProcess('Document Interface',
+        lambda: DevServerProcess('Journalist Interface',
                                  ['python', 'journalist.py'],
                                  'cyan'),
         lambda: DevServerProcess('SASS Compiler',


Test patch:
Repository: freedomofpress/securedrop

Issue:
[QA] Tails GUI updater reporting new versions
# Bug

## Description

The 0.7.0 GUI updater reports new versions even when it's running the latest. Just ran a pre-flight check with the 0.7.0 tag, checked out inside a Tails VM. The install portion completed fine. So did `./securedrop-admin tailsconfig`. However, after finishing, it popped up the GUI declaring there were new updates. Which there definitely should not be, given that 0.7.0 is the latest release.

After a reboot of the Tails VM, the GUI updater displayed again, prompting to install updates. I accepted. Here's the detailed output: https://gist.github.com/conorsch/2e2da8fb909df067b693949474ef945c



## Steps to Reproduce

See above.

## Expected Behavior

0.7.0 is determined to be latest release; no further prompting. 

## Actual Behavior

Prompts for updates even though 0.7.0 is latest release.

## Comments



Discussion:
The GUI only appears when `./securedrop-admin check_for_updates` reports that updates are needed, so there is something wrong with that logic..

Resolving patch:
diff --git a/admin/securedrop_admin/__init__.py b/admin/securedrop_admin/__init__.py
--- a/admin/securedrop_admin/__init__.py
+++ b/admin/securedrop_admin/__init__.py
@@ -588,7 +588,8 @@ def check_for_updates(args):
     sdlog.info("Checking for SecureDrop updates...")
 
     # Determine what branch we are on
-    current_tag = subprocess.check_output(['git', 'describe'], cwd=args.root)
+    current_tag = subprocess.check_output(['git', 'describe'],
+                                          cwd=args.root).rstrip('\n')
 
     # Fetch all branches
     git_fetch_cmd = ['git', 'fetch', '--all']


Test patch:
diff --git a/admin/tests/test_securedrop-admin.py b/admin/tests/test_securedrop-admin.py
--- a/admin/tests/test_securedrop-admin.py
+++ b/admin/tests/test_securedrop-admin.py
@@ -66,6 +66,21 @@ def test_check_for_updates_update_needed(self, tmpdir, caplog):
                 assert update_status is True
                 assert tag == '0.6.1'
 
+    def test_check_for_updates_ensure_newline_stripped(self, tmpdir, caplog):
+        """Regression test for #3426"""
+        git_repo_path = str(tmpdir)
+        args = argparse.Namespace(root=git_repo_path)
+        current_tag = "0.6.1\n"
+        tags_available = "0.6\n0.6-rc1\n0.6.1\n"
+
+        with mock.patch('subprocess.check_call'):
+            with mock.patch('subprocess.check_output',
+                            side_effect=[current_tag, tags_available]):
+                update_status, tag = securedrop_admin.check_for_updates(args)
+                assert "All updates applied" in caplog.text
+                assert update_status is False
+                assert tag == '0.6.1'
+
     def test_check_for_updates_update_not_needed(self, tmpdir, caplog):
         git_repo_path = str(tmpdir)
         args = argparse.Namespace(root=git_repo_path)
Repository: freedomofpress/securedrop

Issue:
Fix check_for_updates
## Status

Ready for review

## Description of Changes

Fixes #3426.

Changes proposed in this pull request:
 * Strips newline from `current_tag` in `securedrop-admin check_for_updates` 

## Test plan

All of the following should be done in Tails.

### No updates needed case

0. Check out 0.6
1. Apply the diff in `admin/securedrop_admin/__init__.py` to 0.6
2. Run `./securedrop-admin check_for_updates`

You should see that updates are not needed: `INFO: All updates applied`

### Updates needed case

0. Check out a random branch that is not this one
1. Add a new tag locally that has a higher version number than the current prod version of SecureDrop, e.g. 0.7.0
2. Switch back to this branch
3. Run `./securedrop-admin check_for_updates`

You should see that updates are needed: `INFO: Update needed`

## Deployment

Will go out in workstation update

## Checklist

### If you made changes to `securedrop-admin`:

- [x] Linting and tests (`make -C admin test`) pass in the admin development container



Discussion:

Resolving patch:
diff --git a/admin/securedrop_admin/__init__.py b/admin/securedrop_admin/__init__.py
--- a/admin/securedrop_admin/__init__.py
+++ b/admin/securedrop_admin/__init__.py
@@ -588,7 +588,8 @@ def check_for_updates(args):
     sdlog.info("Checking for SecureDrop updates...")
 
     # Determine what branch we are on
-    current_tag = subprocess.check_output(['git', 'describe'], cwd=args.root)
+    current_tag = subprocess.check_output(['git', 'describe'],
+                                          cwd=args.root).rstrip('\n')
 
     # Fetch all branches
     git_fetch_cmd = ['git', 'fetch', '--all']


Test patch:
diff --git a/admin/tests/test_securedrop-admin.py b/admin/tests/test_securedrop-admin.py
--- a/admin/tests/test_securedrop-admin.py
+++ b/admin/tests/test_securedrop-admin.py
@@ -66,6 +66,21 @@ def test_check_for_updates_update_needed(self, tmpdir, caplog):
                 assert update_status is True
                 assert tag == '0.6.1'
 
+    def test_check_for_updates_ensure_newline_stripped(self, tmpdir, caplog):
+        """Regression test for #3426"""
+        git_repo_path = str(tmpdir)
+        args = argparse.Namespace(root=git_repo_path)
+        current_tag = "0.6.1\n"
+        tags_available = "0.6\n0.6-rc1\n0.6.1\n"
+
+        with mock.patch('subprocess.check_call'):
+            with mock.patch('subprocess.check_output',
+                            side_effect=[current_tag, tags_available]):
+                update_status, tag = securedrop_admin.check_for_updates(args)
+                assert "All updates applied" in caplog.text
+                assert update_status is False
+                assert tag == '0.6.1'
+
     def test_check_for_updates_update_not_needed(self, tmpdir, caplog):
         git_repo_path = str(tmpdir)
         args = argparse.Namespace(root=git_repo_path)
Repository: freedomofpress/securedrop

Issue:
Formatting of i18 codes in includes/l10.txt and install.rst
## Description

The readability of the list of supported languages in `install.rst` could be improved.

## Suggestion

If the output in `includes/l10.txt` could include backticks around the language codes, it would make the section [here](https://docs.securedrop.org/en/stable/install.html#localization-of-the-source-and-journalist-interfaces) easier to digest. There would be a visual cue that the strings in parentheses are the "codes" mentioned in the doc:

```
.. GENERATED BY i18n_tool.py DO NOT EDIT:

* Arabic (``ar``)
* German (``de_DE``)
* Spanish (``es_ES``)
* French (``fr_FR``)
* Hindi (``hi``)
* Italian (``it_IT``)
* Norwegian (``nb_NO``)
* Dutch (``nl``)
* Portuguese, Brasil (``pt_BR``)
* Russian (``ru``)
* Turkish (``tr``)
* Chinese, Traditional (``zh_Hant``)
```




Discussion:

Resolving patch:
diff --git a/securedrop/i18n_tool.py b/securedrop/i18n_tool.py
--- a/securedrop/i18n_tool.py
+++ b/securedrop/i18n_tool.py
@@ -274,7 +274,7 @@ def require_git_email_name(git_dir):
     def update_docs(self, args):
         l10n_content = u'.. GENERATED BY i18n_tool.py DO NOT EDIT:\n\n'
         for (code, info) in sorted(I18NTool.SUPPORTED_LANGUAGES.items()):
-            l10n_content += '* ' + info['name'] + ' (' + code + ')\n'
+            l10n_content += '* ' + info['name'] + ' (``' + code + '``)\n'
         includes = join(args.documentation_dir, 'includes')
         l10n_txt = join(includes, 'l10n.txt')
         io.open(l10n_txt, mode='w').write(l10n_content)


Test patch: