internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: freedomofpress/securedrop

Issue:
Install securedrop-keyring package
This is the change to the playbook required to install the [SecureDrop keyring package](https://github.com/freedomofpress/securedrop-keyring) as part of the FPF apt repo, which will allow us to update the public key which is used to sign the [apt.freedom.press](https://apt.freedom.press) Release file and its expiry in the future without requiring manual action by SecureDrop instance admins. For context, our master signing key expired without us being aware on October 26, 2015, necessitating the 0.3.6 hotfix.

Per #1196 , the `install_fpf_repo` role is mostly the same, it just has slashes instead of underscores.

The actual package and its build process is simple and described in the linked GitHub repository. Merging this PR should occur concurrent with adding the package to our repository, which is a whole process.



Discussion:
This is blocked on us creating the securedrop-keyring package (code for that is here: https://github.com/freedomofpress/securedrop-keyring) and pushing it to the apt package server, so it needs to be part of a release :weary:.

The underlying task lists have moved around a bit, so these changes will likely need to be rebased. We'll also need to update the documentation to change mention of the `securedrop-keyring` package from a note to an item in the definition list. As @garrettr mentioned, we'll block on this until we have a new release. 

Rebased on latest develop. Looks like this is being targeted for 0.4. We have 5 months before the key expires again, so might be worth considering for a point release.

Latest commit makes `securedrop-keyring` a dependency of `securedrop-app-code`. This is intended to avoid requiring manual action by admins in order to install the keyring package before the next key expiration date. Instead, it can be pulled in automatically via the app code dependencies when the app code package is updated.

See also, Recommends or Suggests: https://www.debian.org/doc/debian-policy/ch-relationships.html#s-binarydeps


Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -59,9 +59,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.3.9'
+version = '0.3.10'
 # The full version, including alpha/beta/rc tags.
-release = '0.3.9'
+release = '0.3.10'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
diff --git a/securedrop/version.py b/securedrop/version.py
--- a/securedrop/version.py
+++ b/securedrop/version.py
@@ -1 +1 @@
-__version__ = '0.3.9'
+__version__ = '0.3.10'


Test patch:
Repository: freedomofpress/securedrop

Issue:
Merge changes from 0.3.10 into develop 
The 0.3.10 release has one major change, rotation of the signing key, plus a few docs changes. Backport those changes into the develop branch so staging/prod VMs still work past key expiry.



Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -59,9 +59,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.3.9'
+version = '0.3.10'
 # The full version, including alpha/beta/rc tags.
-release = '0.3.9'
+release = '0.3.10'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
diff --git a/securedrop/version.py b/securedrop/version.py
--- a/securedrop/version.py
+++ b/securedrop/version.py
@@ -1 +1 @@
-__version__ = '0.3.9'
+__version__ = '0.3.10'


Test patch:
Repository: freedomofpress/securedrop

Issue:
Add linting for docs changes to CI
When building the project documentation locally via `make clean && sphinx-autobuild . _build/html`, as specified in the [developer docs](https://docs.securedrop.org/en/stable/development/documentation.html), nearly two dozen warnings are displayed, generating more than a full screen of red text in the terminal on first run. 

The warnings are mostly related to style, e.g. `WARNING: Title underline too short`, but some seem of greater concern: `WARNING: duplicate label troubleshooting`. The warnings still permit the docs to be built, but we should clean up those warnings and add linting to the CI config to guard against regressions. Particularly when overhauling the docs as described in #1544, we'll want to make sure the docs stay clean and consistent.

Haven't done much reading on best practice for linting Sphinx docs, but a quick and dirty method for detecting mistakes is: `make clean && sphinx-build -Wn . _build/html`. When run inside the `docs/` folder of the project, any warnings will cause the build to exit non-zero. 


Discussion:
We don't currently provide any developer-specific requirements file for installing the tooling necessary to test and configure the SecureDrop VMs. As of #1616 we're quite close with `testinfra/requirements.txt`, since that installs pytest and the like on the developer machine (preferably in a virtualenv), but we still only recommend installing sphinx for docs manipulation via a copy/pasted command from the developer docs. Will fold that logic into a `dev-requirements.txt` file, so it's installable under Travis as well, then update the dev docs to point to that file where appropriate (#1208).

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -156,7 +156,7 @@
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+# html_static_path = ['_static']
 
 # Add any extra paths that contain custom files (such as robots.txt or
 # .htaccess) here, relative to this directory. These files are copied


Test patch:
Repository: freedomofpress/securedrop

Issue:
Fix hard-coded local IPv4 addresses in staging tests
During merge of #1709, there was a regression in the testinfra tests, so the hardcoded IPv4 addresses used in Vagrant VMs were overwritten with ephemeral IPv4 addresses for CI. Specifically the commt in question is 48db3f57528e69a3e7cbe2e337cec114192d54ef.

We'll need to modify the testinfra/vars/ files to use the old, local IPv4 addresses again.
Fix hard-coded local IPv4 addresses in staging tests
During merge of #1709, there was a regression in the testinfra tests, so the hardcoded IPv4 addresses used in Vagrant VMs were overwritten with ephemeral IPv4 addresses for CI. Specifically the commt in question is 48db3f57528e69a3e7cbe2e337cec114192d54ef.

We'll need to modify the testinfra/vars/ files to use the old, local IPv4 addresses again.


Discussion:

Resolving patch:
diff --git a/install_files/ansible-base/action_plugins/synchronize.py b/install_files/ansible-base/action_plugins/synchronize.py
--- a/install_files/ansible-base/action_plugins/synchronize.py
+++ b/install_files/ansible-base/action_plugins/synchronize.py
@@ -364,7 +364,7 @@ def run(self, tmp=None, task_vars=None):
                 # If no rsync_path is set, become was originally set, and dest is
                 # remote then add privilege escalation here.
                 if self._play_context.become_method == 'sudo':
-                    rsync_path = 'sudo rsync'
+                    rsync_path = "'sudo rsync'"
                 # TODO: have to add in the rest of the become methods here
 
             # We cannot use privilege escalation on the machine running the


Test patch:
Repository: freedomofpress/securedrop

Issue:
Port Tails setup to Ansible logic
# Feature request

## Description

The existing setup logic for provisioning Admin and Journalist Workstations is to run the `tails_files/install.sh` script. That script contains about 500 lines of Bash, and contains some tricky grep and sed logic that is difficult to test reliably (which has caused problems in the past, e.g. #1633). The sole reason we're using Bash to configure the Tails environment is that Ansible is only available on the Admin Workstation, but not the Journalist Workstation. Give the tooling presented in #1781 to wrap the Ansible bootstrap logic, this is a fine opportunity to port the Bash logic over to stable Ansible playbooks, so we can make use of DRY vars.

Additionally, much of the logic contained in `tails_files/install.sh` contains legacy cleanup tasks. Due to the Tails 2 -> 3 migration for 0.4 (#1551), Admins must configure new Tails volumes, so much of the legacy cleanup tasks can be removed.

## User Stories

As an Admin, I want to run a single script and configure the Tails environment post-install. The script should understand target workstation roles such as "Admin" and "Journalist" and configure them accordingly.

Relevant issues: #1533, #1176, #1177.



Discussion:

Resolving patch:
diff --git a/tails_files/securedrop_init.py b/install_files/ansible-base/roles/tails-config/files/securedrop_init.py
similarity index 100%
rename from tails_files/securedrop_init.py
rename to install_files/ansible-base/roles/tails-config/files/securedrop_init.py


Test patch:
Repository: freedomofpress/securedrop

Issue:
Wordlists are not being parsed correctly
# Bug

## Description

`crypo_util.{words,nouns,adjectives}` all contain an empty string as their last element.


Discussion:

Resolving patch:
diff --git a/securedrop/crypto_util.py b/securedrop/crypto_util.py
--- a/securedrop/crypto_util.py
+++ b/securedrop/crypto_util.py
@@ -46,9 +46,9 @@ def do_runtime_tests():
 
 gpg = gnupg.GPG(binary='gpg2', homedir=config.GPG_KEY_DIR)
 
-words = open(config.WORD_LIST).read().split('\n')
-nouns = open(config.NOUNS).read().split('\n')
-adjectives = open(config.ADJECTIVES).read().split('\n')
+words = open(config.WORD_LIST).read().rstrip('\n').split('\n')
+nouns = open(config.NOUNS).read().rstrip('\n').split('\n')
+adjectives = open(config.ADJECTIVES).read().rstrip('\n').split('\n')
 
 
 class CryptoException(Exception):


Test patch:
diff --git a/securedrop/tests/test_crypto_util.py b/securedrop/tests/test_crypto_util.py
--- a/securedrop/tests/test_crypto_util.py
+++ b/securedrop/tests/test_crypto_util.py
@@ -20,6 +20,11 @@ def setUp(self):
     def tearDown(self):
         utils.env.teardown()
 
+    def test_word_list_does_not_contain_empty_strings(self):
+        self.assertNotIn('', (crypto_util.words
+                              + crypto_util.nouns
+                              + crypto_util.adjectives))
+
     def test_clean(self):
         ok = (' !#%$&)(+*-1032547698;:=?@acbedgfihkjmlonqpsrutwvyxzABCDEFGHIJ'
               'KLMNOPQRSTUVWXYZ')
Repository: freedomofpress/securedrop

Issue:
inconsistent display when exiting /use-tor
# Bug

## Description

The display shows both "you are not using tor" and "Please select "New Identity" from the green Onion button in the Tor browser".

## Steps to Reproduce

* navigate to the source interface with firefox
* click on the "Learn how to install it" link
* click on the exit button

## Expected Behavior

![expected](https://user-images.githubusercontent.com/433594/28218975-98345dec-68ba-11e7-895f-a2f96e049557.png)

## Actual Behavior

![inconsistent](https://user-images.githubusercontent.com/433594/28218983-a340c504-68ba-11e7-88fd-d65b98a58742.png)



Discussion:
Great catch @dachary, thanks! This has irked me before, but since I've switched to use Tor Browser for all SD testing, I haven't seen this lately. Going to toss this on the 0.4.1 milestone, so it'll get patched soon, but not block ongoing QA for 0.4. 

Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -399,7 +399,7 @@ def logout():
     if logged_in():
         session.clear()
         msg = render_template('logout_flashed_message.html')
-        flash(Markup(msg), "important")
+        flash(Markup(msg), "important hide-if-not-tor-browser")
     return redirect(url_for('index'))
 
 


Test patch:
diff --git a/securedrop/tests/functional/source_navigation_steps.py b/securedrop/tests/functional/source_navigation_steps.py
--- a/securedrop/tests/functional/source_navigation_steps.py
+++ b/securedrop/tests/functional/source_navigation_steps.py
@@ -161,5 +161,4 @@ def _source_deletes_a_journalist_reply(self):
 
     def _source_logs_out(self):
         self.driver.find_element_by_id('logout').click()
-        notification = self.driver.find_element_by_css_selector('.important')
-        assert 'Thank you for exiting your session!' in notification.text
+        assert self.driver.find_element_by_css_selector('.important')
Repository: freedomofpress/securedrop

Issue:
Add backup and restore options to securedrop-admin
## Description

Instead of admins directly running Ansible playbooks for backup and restore, which can be a bit confusing for the uninitiated, add command line options to the `securedrop-admin` script to perform the backup and restore to make this task a bit more user friendly, e.g. `securedrop-admin backup` and `securedrop-admin restore <restore_file>`

## User Stories

As an administrator, I want to have a straightforward tool for backups so I will frequently backup my SecureDrop instance. 

As an administrator, I want to have a straightforward tool for restoring backups so I can easily restore my SecureDrop instance in case of failure. 


Discussion:

Resolving patch:
diff --git a/install_files/ansible-base/roles/backup/files/0.3_restore.py b/install_files/ansible-base/roles/restore/files/0.3_restore.py
similarity index 100%
rename from install_files/ansible-base/roles/backup/files/0.3_restore.py
rename to install_files/ansible-base/roles/restore/files/0.3_restore.py
diff --git a/install_files/ansible-base/roles/backup/files/restore.py b/install_files/ansible-base/roles/restore/files/restore.py
similarity index 100%
rename from install_files/ansible-base/roles/backup/files/restore.py
rename to install_files/ansible-base/roles/restore/files/restore.py


Test patch:
Repository: freedomofpress/securedrop

Issue:
Release SecureDrop 0.4.1
This is a tracking issue for this week's release of SecureDrop 0.4.1, a bugfix release for #2049. Tasks may get added or modified. 

# Prepare release
- [x] Create `release/0.4.1` branch - @redshiftzero
- [x] PR, test and merge the hotfix for #2049 in [this comment](https://github.com/freedomofpress/securedrop/issues/2049#issuecomment-318251010)
- [x] Bump version numbers and build test debian packages. I'm suggesting that we bump the version numbers even though the application code will be unchanged (the fix is in Ansible used in the Tails workstation environment) for consistency. i.e. if admins check out the SecureDrop blog or tags, they should be able to look at the "Powered by SecureDrop x.x" and not be confused about what happened. 

# QA
 
- [x] Run through Tails 2 -> Tails 3 upgrade one last time - @redshiftzero

# Release

- [x] Release 0.4.1
- [x] Publish blog post about 0.4.1 bugfix - @redshiftzero

# Post-release

- [x] Merge release changes into master branch - @redshiftzero
- [x] Merge release changes into development branch - @redshiftzero


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -59,9 +59,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.4'
+version = '0.4.1'
 # The full version, including alpha/beta/rc tags.
-release = '0.4'
+release = '0.4.1'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
diff --git a/securedrop/version.py b/securedrop/version.py
--- a/securedrop/version.py
+++ b/securedrop/version.py
@@ -1 +1 @@
-__version__ = '0.4'
+__version__ = '0.4.1'


Test patch:
Repository: freedomofpress/securedrop

Issue:
Release SecureDrop 0.4.2
This is a tracking issue for this week's release of SecureDrop 0.4.2, a bugfix release for #2105. Tasks may get added or modified. 

# Prepare release
- [x] Create `release/0.4.2` branch - @conorsch
- [x] PR, test and merge the hotfix for #2105 - @conorsch
- [x] Bump version numbers and build test debian packages. - @conorsch

# QA
 
- [x] Confirm clean installs in staging (using the 0.4.2 rc deb packages) succeed - @conorsch
- [x] Test updates from 0.4.1 -> 0.4.2 rc and confirm upgrade restores web interfaces - @conorsch

# Release

- [x] Release 0.4.2 - @conorsch
- [x] Publish blog post about 0.4.2 bugfix - @conorsch

# Post-release

- [x] Merge release changes into master branch - @redshiftzero
- [x] Merge release changes into development branch - @redshiftzero


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -59,9 +59,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.4.1'
+version = '0.4.2'
 # The full version, including alpha/beta/rc tags.
-release = '0.4.1'
+release = '0.4.2'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
diff --git a/securedrop/version.py b/securedrop/version.py
--- a/securedrop/version.py
+++ b/securedrop/version.py
@@ -1 +1 @@
-__version__ = '0.4.1'
+__version__ = '0.4.2'


Test patch:
diff --git a/testinfra/app/test_apparmor.py b/testinfra/app/test_apparmor.py
--- a/testinfra/app/test_apparmor.py
+++ b/testinfra/app/test_apparmor.py
@@ -16,6 +16,7 @@ def test_apparmor_enabled(Command, Sudo):
         assert Command("aa-status --enabled").rc == 0
 
 apache2_capabilities = [
+        'dac_override',
         'kill',
         'net_bind_service',
         'sys_ptrace'
Repository: freedomofpress/securedrop

Issue:
Remove usage of `safe` from Jinja templates
# Feature request

## Description

Jinja templates should not use the `safe` function for strings-that-are-HTML. In `{source,journalist}.py`, the `Markdown` class should be used when a value is known to valid HTML.

### Example (bad)
```html
<p>{{ message|safe }}</p>
```

## User Stories

Kids, just say no to XSS.


Discussion:

Resolving patch:
diff --git a/securedrop/db.py b/securedrop/db.py
--- a/securedrop/db.py
+++ b/securedrop/db.py
@@ -14,6 +14,7 @@
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy import Column, Integer, String, Boolean, DateTime, Binary
 from sqlalchemy.orm.exc import MultipleResultsFound, NoResultFound
+from jinja2 import Markup
 
 import scrypt
 import pyotp
@@ -369,7 +370,7 @@ def shared_secret_qrcode(self):
 
         svg_out = StringIO()
         img.save(svg_out)
-        return svg_out.getvalue()
+        return Markup(svg_out.getvalue())
 
     @property
     def formatted_otp_secret(self):
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -118,9 +118,10 @@ def check_tor2web():
     # ignore_static here so we only flash a single message warning
     # about Tor2Web, corresponding to the initial page load.
     if 'X-tor2web' in request.headers:
-        flash('<strong>WARNING:</strong> You appear to be using Tor2Web. '
-              'This <strong>does not</strong> provide anonymity. '
-              '<a href="/tor2web-warning">Why is this dangerous?</a>',
+        flash(Markup('<strong>WARNING:</strong> You appear to be using '
+                     'Tor2Web. This <strong>does not</strong> provide '
+                     'anonymity. <a href="/tor2web-warning">Why is this '
+                     'dangerous?</a>'),
               "banner-warning")
 
 


Test patch:
Repository: freedomofpress/securedrop

Issue:
i18n Doesn't respect user locale preferences
# Bug

I accept several languages, and send a header with a different priority. I prefer for example Dutch to German and a typical Accept-Language header here could look like: 
`Accept-Language:en-US,en;q=0.8,nl;q=0.6,fr;q=0.4,de;q=0.2
`
This will *not* display the Dutch interface but the French one. If I send one like:
`Accept-Language: en-US,en;q=0.8,nl;q=0.6,de;q=0.4
`
I would get the German one.

## Steps to Reproduce

Browse to http://source.i18n.securedrop.club/ and send such headers:

GET / HTTP/1.1
Host: source.i18n.securedrop.club
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chromium/63.0.XXXXX Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: http://i18n.securedrop.club/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8,nl;q=0.6,de;q=0.4



Discussion:

Resolving patch:
diff --git a/securedrop/i18n.py b/securedrop/i18n.py
--- a/securedrop/i18n.py
+++ b/securedrop/i18n.py
@@ -66,14 +66,14 @@ def get_locale():
     - 'en_US'
     """
     locale = None
-    accept_languages = set()
+    accept_languages = []
     for l in request.accept_languages.values():
         if '-' in l:
             sep = '-'
         else:
             sep = '_'
         try:
-            accept_languages.add(str(core.Locale.parse(l, sep)))
+            accept_languages.append(str(core.Locale.parse(l, sep)))
         except:
             pass
     if 'l' in request.args:


Test patch:
diff --git a/securedrop/tests/test_i18n.py b/securedrop/tests/test_i18n.py
--- a/securedrop/tests/test_i18n.py
+++ b/securedrop/tests/test_i18n.py
@@ -80,6 +80,13 @@ def verify_i18n(self, app):
                 {{ gettext('code hello i18n') }}
                 ''').strip() == translated_fr
 
+        # https://github.com/freedomofpress/securedrop/issues/2379
+        headers = Headers([('Accept-Language',
+                            'en-US;q=0.6,fr_FR;q=0.4,nb_NO;q=0.2')])
+        with app.test_request_context(headers=headers):
+            assert not hasattr(request, 'babel_locale')
+            assert not_translated == gettext(not_translated)
+
         translated_cn = 'code chinese'
 
         for lang in ('zh-CN', 'zh-Hans-CN'):
Repository: freedomofpress/securedrop

Issue:
ngettext not used for journalist login throttling
# Bug

## Description

We don't use ngettext for the login throttling and this makes translating into other languages grammatically incorrect (Russian in this case).

Reported by @xella

## Steps to Reproduce

Grep for the following string.

## Expected Behavior
```
ngettext('Please wait at least {seconds} seconds before logging in again.'.format(seconds=seconds), seconds)
```

## Actual Behavior
```
gettext('Please wait at least {seconds} seconds before logging in again.'.format(seconds=seconds))
```


Discussion:
The number of seconds will always be > 1, reason why I did not ngettext this. I'm curious about why it creates a problem in Russian ?
Plurality is non-trivial in Russian. [Link](http://www.russianlessons.net/lessons/lesson11_main.php). In short, there are different ways to write things if the number ends in:

- 1
- 2, 3, 4
- 5, 6, 7, 8, 9, 0

So the standard English/French/etc. rule of "more than one" is insufficient. I'm also fairly sure Russian isn't the only language with such a pattern.
Oh .... ok. Thanks for explaining, I'll fix this.

Resolving patch:
diff --git a/securedrop/journalist.py b/securedrop/journalist.py
--- a/securedrop/journalist.py
+++ b/securedrop/journalist.py
@@ -134,10 +134,14 @@ def login():
 
             if isinstance(e, LoginThrottledException):
                 login_flashed_msg += " "
-                login_flashed_msg += gettext(
+                period = Journalist._LOGIN_ATTEMPT_PERIOD
+                # ngettext is needed although we always have period > 1
+                # see https://github.com/freedomofpress/securedrop/issues/2422
+                login_flashed_msg += ngettext(
+                    "Please wait at least {seconds} second "
+                    "before logging in again.",
                     "Please wait at least {seconds} seconds "
-                    "before logging in again.").format(
-                        seconds=Journalist._LOGIN_ATTEMPT_PERIOD)
+                    "before logging in again.", period).format(seconds=period)
             else:
                 try:
                     user = Journalist.query.filter_by(


Test patch: