internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: freedomofpress/securedrop

Issue:
Add v2 deprecation warning to securedrop-admin
As we're implementing an official deprecation timeline for v2 Onion Services (see #5302), as a standard practice for deprecation warnings, we should display appropriate warnings in CLI tooling as well. This includes `securedrop-admin sdconfig`, and possibly `securedrop-admin install`. We don't need to go overboard, but we should make the removal date clear in the CLI output for any command run that includes the enabling or provisioning of v2 Onion Services.


Discussion:

Resolving patch:
diff --git a/admin/securedrop_admin/__init__.py b/admin/securedrop_admin/__init__.py
--- a/admin/securedrop_admin/__init__.py
+++ b/admin/securedrop_admin/__init__.py
@@ -410,7 +410,9 @@ def __init__(self, args):
              str.split,
              lambda config: True],
             ['v2_onion_services', self.check_for_v2_onion(), bool,
-             'Do you want to enable v2 onion services (recommended only for SecureDrop instances installed before 1.0.0)?',  # noqa: E501
+             'WARNING: For security reasons, support for v2 onion services ' +
+             'will be removed in February 2021. ' +
+             'Do you want to enable v2 onion services?',
              SiteConfig.ValidateYesNo(),
              lambda x: x.lower() == 'yes',
              lambda config: True],


Test patch:
diff --git a/admin/tests/test_integration.py b/admin/tests/test_integration.py
--- a/admin/tests/test_integration.py
+++ b/admin/tests/test_integration.py
@@ -340,7 +340,7 @@ def verify_locales_prompt(child):
 
 
 def verify_v2_onion_for_first_time(child):
-    child.expect(rb' installed before 1.0.0\)\?\:', timeout=2)  # noqa: E501
+    child.expect(rb'Do you want to enable v2 onion services\?\:', timeout=2)  # noqa: E501
     assert ANSI_ESCAPE.sub('', child.buffer.decode("utf-8")).strip() == 'no'  # noqa: E501
Repository: freedomofpress/securedrop

Issue:
`new-password` route is defined twice
It looks to me like this route in the admin UI is defined redundantly:
https://github.com/freedomofpress/securedrop/blob/develop/securedrop/journalist_app/admin.py#L219
https://github.com/freedomofpress/securedrop/blob/develop/securedrop/journalist_app/admin.py#L259

If there is no reason for this, we should remove one of the redundant definitions.


Discussion:

Resolving patch:
diff --git a/securedrop/journalist_app/admin.py b/securedrop/journalist_app/admin.py
--- a/securedrop/journalist_app/admin.py
+++ b/securedrop/journalist_app/admin.py
@@ -216,23 +216,6 @@ def edit_user(user_id):
         return render_template("edit_account.html", user=user,
                                password=password)
 
-    @view.route('/edit/<int:user_id>/new-password', methods=('POST',))
-    @admin_required
-    def set_password(user_id):
-        try:
-            user = Journalist.query.get(user_id)
-        except NoResultFound:
-            abort(404)
-
-        password = request.form.get('password')
-        if set_diceware_password(user, password) is not False:
-            if user.last_token is not None:
-                revoke_token(user, user.last_token)
-            user.session_nonce += 1
-            db.session.commit()
-
-        return redirect(url_for('admin.edit_user', user_id=user_id))
-
     @view.route('/delete/<int:user_id>', methods=('POST',))
     @admin_required
     def delete_user(user_id):


Test patch:
Repository: freedomofpress/securedrop

Issue:
Update removed `platform.linux_distribution` funtion call in Python 3.8
## Description

We are using [platform.linux_distribution](https://github.com/freedomofpress/securedrop/blob/4c73102ca9151a86a08396de40163b48a5a21768/securedrop/source_app/api.py#L20) function in our metadata endpoint. But, this function was deprecated from Python3.5 and totally removed from Python 3.8. 

## Solution

We can directly read the `/etc/lsb-release` and `/etc/os-release` file as required.



Discussion:
 #4768 is dependent on this. 
@kushaldas and @eloquence, I can take care of this if it's OK with you.

Resolving patch:
diff --git a/securedrop/source_app/api.py b/securedrop/source_app/api.py
--- a/securedrop/source_app/api.py
+++ b/securedrop/source_app/api.py
@@ -1,5 +1,4 @@
 import json
-import platform
 
 from flask import Blueprint, current_app, make_response
 
@@ -8,6 +7,10 @@
 import version
 
 
+with open("/etc/lsb-release", "r") as f:
+    server_os = f.readlines()[1].split("=")[1].strip("\n")
+
+
 def make_blueprint(config):
     view = Blueprint('api', __name__)
 
@@ -17,7 +20,7 @@ def metadata():
             'allow_document_uploads': current_app.instance_config.allow_document_uploads,
             'gpg_fpr': config.JOURNALIST_KEY,
             'sd_version': version.__version__,
-            'server_os': platform.linux_distribution()[1],
+            'server_os': server_os,
             'supported_languages': config.SUPPORTED_LOCALES,
             'v2_source_url': get_sourcev2_url(),
             'v3_source_url': get_sourcev3_url()


Test patch:
diff --git a/securedrop/tests/test_source.py b/securedrop/tests/test_source.py
--- a/securedrop/tests/test_source.py
+++ b/securedrop/tests/test_source.py
@@ -583,8 +583,7 @@ def test_why_journalist_key(source_app):
 
 
 def test_metadata_route(config, source_app):
-    with patch.object(source_app_api.platform, "linux_distribution") as mocked_platform:
-        mocked_platform.return_value = ("Ubuntu", "16.04", "xenial")
+    with patch.object(source_app_api, "server_os", new="16.04"):
         with source_app.test_client() as app:
             resp = app.get(url_for('api.metadata'))
             assert resp.status_code == 200
Repository: freedomofpress/securedrop

Issue:
Test result combination scripts can be removed
## Description

We have a couple of scripts for combining test results in CI that are no longer being used and can be deleted:
```
devops/scripts/combine-junit.py
devops/scripts/combine-junit-test-results.sh
```



Discussion:

Resolving patch:
diff --git a/devops/scripts/combine-junit.py b/devops/scripts/combine-junit.py
deleted file mode 100755
--- a/devops/scripts/combine-junit.py
+++ /dev/null
@@ -1,68 +0,0 @@
-#!/usr/bin/env python
-#
-#  Corey Goldberg, Dec 2012
-#  Original source from gist.github.com/cgoldberg/4320815
-
-import os
-import sys
-import xml.etree.ElementTree as ET
-
-
-"""Merge multiple JUnit XML files into a single results file.
-
-Output dumps to sdtdout.
-
-example usage:
-    $ python merge_junit_results.py results1.xml results2.xml > results.xml
-"""
-
-
-def main():
-    args = sys.argv[1:]
-    if not args:
-        usage()
-        sys.exit(2)
-    if '-h' in args or '--help' in args:
-        usage()
-        sys.exit(2)
-    merge_results(args[:])
-
-
-def merge_results(xml_files):
-    failures = 0
-    tests = 0
-    errors = 0
-    time = 0.0
-    cases = []
-
-    for file_name in xml_files:
-        # We disable bandit checking to permit B314, which recommends use
-        # of defusedxml to protect against malicious XML files. This code
-        # path only runs in CI, not on developer workstations, and the XML
-        # output is generated by testinfra on staging machines.
-        tree = ET.parse(file_name)  # nosec
-        test_suite = tree.getroot()
-        failures += int(test_suite.attrib['failures'])
-        tests += int(test_suite.attrib['tests'])
-        errors += int(test_suite.attrib['errors'])
-        time += float(test_suite.attrib['time'])
-        cases.append(test_suite.getchildren())
-
-    new_root = ET.Element('testsuite')
-    new_root.attrib['failures'] = '%s' % failures
-    new_root.attrib['tests'] = '%s' % tests
-    new_root.attrib['errors'] = '%s' % errors
-    new_root.attrib['time'] = '%s' % time
-    for case in cases:
-        new_root.extend(case)
-    new_tree = ET.ElementTree(new_root)
-    ET.dump(new_tree)
-
-
-def usage():
-    this_file = os.path.basename(__file__)
-    print('Usage:  %s results1.xml results2.xml' % this_file)
-
-
-if __name__ == '__main__':
-    main()


Test patch:
Repository: freedomofpress/securedrop

Issue:
Can not delete source keys in Focal
## Description

In Focal container if we delete a source, the related `gpg` keys should also be deleted, but they are not getting deleted.

## Steps to Reproduce

- [ ] Start a Focal dev container from `dev_focal` branch: `make dev-focal`
- [ ] Login as journalist, delete a source. 
- [ ] `docker exec -it containerid /bin/bash`
- [ ] root@7701c5345245:~/code/securedrop/securedrop# gpg2 --homedir=/var/lib/securedrop/keys --list-keys 

## Expected Behavior

- [ ] Only 2 keys, 1 for journalist, and 1 remaining source key should be there.


## Actual Behavior

- [ ] All 3 keys are there.

```
root@7701c5345245:~/code/securedrop/securedrop# gpg2 --homedir=/var/lib/securedrop/keys --list-keys                                                                                               
/var/lib/securedrop/keys/pubring.gpg
------------------------------------
pub   rsa4096 2013-10-12 [SC]
      65A1B5FF195B56353CC63DFFCC40EF1228271441
uid           [ unknown] SecureDrop Test/Development (DO NOT USE IN PRODUCTION)
sub   rsa4096 2013-10-12 [E]

pub   rsa1024 2013-05-14 [SCEA]
      A7501454825B423CF894E7CDEDE3DF05DE24277C
uid           [ultimate] Source Key <EVAXJEWVIHP4SSBOEF4N5KVFHBKN6FOGFBIEJQIIBGDE3R3LCUBSUHRCV72X6IS6SWSKRBIPVL7QJV2PD6UFBPL6XZY2VVCPY67BZEI=>                                                    

pub   rsa1024 2013-05-14 [SCEA]
      787F6132468D0A1E125C070F14530E2B67230F7D
uid           [ultimate] Source Key <OD4QTM56LDGKN3FUTCTN66GFB27D5STUAZP6LTMDS467YPE52HSXW4WJSNEPGIGQT3B75UMVRWO3HFNY4KUEJQFRY3L36A7FRKTFM4Q=>                                                    

```

## Comments

Suggestions to fix, any other relevant information.



Discussion:

Resolving patch:
diff --git a/securedrop/crypto_util.py b/securedrop/crypto_util.py
--- a/securedrop/crypto_util.py
+++ b/securedrop/crypto_util.py
@@ -267,7 +267,7 @@ def delete_reply_keypair(self, source_filesystem_id: str) -> None:
 
         # Always delete keys without invoking pinentry-mode = loopback
         # see: https://lists.gnupg.org/pipermail/gnupg-users/2016-May/055965.html
-        temp_gpg = gnupg.GPG(binary='gpg2', homedir=self.gpg_key_dir)
+        temp_gpg = gnupg.GPG(binary='gpg2', homedir=self.gpg_key_dir, options=["--yes"])
 
         # The subkeys keyword argument deletes both secret and public keys.
         temp_gpg.delete_keys(fingerprint, secret=True, subkeys=True)


Test patch:
Repository: freedomofpress/securedrop

Issue:
dev server hot reload has stopped working
## Description

In #5532 the `env` attribute was dropped from `SDConfig` in `sdconfig.py`. That value is checked in [`source.py`](https://github.com/freedomofpress/securedrop/blob/6246482157e31d0655a91c5e7284cc8550f2c289/securedrop/source.py#L11) and [`journalist.py`](https://github.com/freedomofpress/securedrop/blob/6246482157e31d0655a91c5e7284cc8550f2c289/securedrop/journalist.py#L26) to determine whether the Flask app will run in [debug](https://flask.palletsprojects.com/en/1.1.x/config/#DEBUG) mode. By default it will not, so the dev server has stopped responding to code changes.

Given the Flask documentation warnings about setting debug mode via code and not the `FLASK_DEBUG` environment variable, we may want to reevaluate all of this, but right now let's just get back to a properly functioning dev server.

## Steps to Reproduce

- Check out `develop` at a commit before the `sdconfig.py` change (eff931fa8a0e74d5c3be87e46c5d0f004f02e289).
- Run `make dev`.
- Change `securedrop/journalist_app/main.py` to trigger a reload, and confirm that the change is detected.
- Stop the dev server.
- Check out `develop` and run `make dev`.
- Change `securedrop/journalist_app/main.py` again, and observe that the change is not detected.

## Expected Behavior

That the dev server would notice code changes and reload to pick them up.

## Actual Behavior

It does not care one whit about your useless flailings. We are all `prod` now.

## Comments

Just need to restore `SDConfig.env`.



Discussion:

Resolving patch:
diff --git a/securedrop/sdconfig.py b/securedrop/sdconfig.py
--- a/securedrop/sdconfig.py
+++ b/securedrop/sdconfig.py
@@ -53,7 +53,8 @@ def __init__(self) -> None:
 
         self.WORKER_PIDFILE = _config.WORKER_PIDFILE  # type: str
 
-        if _config.env == 'test':
+        self.env = getattr(_config, 'env', 'prod')  # type: str
+        if self.env == 'test':
             self.RQ_WORKER_NAME = 'test'  # type: str
         else:
             self.RQ_WORKER_NAME = 'default'


Test patch:
Repository: freedomofpress/securedrop

Issue:
Resetting journalist password more than once can raise IntegrityError
## Description

If a journalist's password is reset twice in quick succession, the second attempt to revoke the two-factor token will trigger an exception:
```
sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: revoked_tokens.token
[SQL: INSERT INTO revoked_tokens (journalist_id, token) VALUES (?, ?)]
```

## Steps to Reproduce

- Run `make dev`
- Log in as `dellsberg`.
- Log in as `journalist`, visit http://localhost:8081/admin/ and edit the `dellsberg` account. 
- Reset the password once, then again. You should see the `IntegrityError`.

## Expected Behavior

That you could reset the password as many times as you like.

## Actual Behavior

You cannot reset the password as many times as you like.



Discussion:

Resolving patch:
diff --git a/securedrop/journalist_app/utils.py b/securedrop/journalist_app/utils.py
--- a/securedrop/journalist_app/utils.py
+++ b/securedrop/journalist_app/utils.py
@@ -472,6 +472,11 @@ def cleanup_expired_revoked_tokens() -> None:
 
 
 def revoke_token(user: Journalist, auth_token: str) -> None:
-    revoked_token = RevokedToken(token=auth_token, journalist_id=user.id)
-    db.session.add(revoked_token)
-    db.session.commit()
+    try:
+        revoked_token = RevokedToken(token=auth_token, journalist_id=user.id)
+        db.session.add(revoked_token)
+        db.session.commit()
+    except IntegrityError as e:
+        db.session.rollback()
+        if "UNIQUE constraint failed: revoked_tokens.token" not in str(e):
+            raise e


Test patch:
Repository: freedomofpress/securedrop

Issue:
Dual distro support broke "securedrop-admin verify"
## Description

When adding support for Focal to the configuration tests in #5529, a check of the `MOLECULE_SCENARIO_NAME` environment variable broke `securedrop-admin verify`, where it's not set.

## Steps to Reproduce

On an admin workstation:
- Run `securedrop-admin setup -t`
- Run `securedrop-admin verify`

## Expected Behavior

That the configuration tests would run.

## Actual Behavior

You get an error [here](https://github.com/freedomofpress/securedrop/blob/76d133a7e5962f8d904e507d93e6a61575358eeb/molecule/testinfra/conftest.py#L31) saying `'NoneType' object has no attribute 'endswith'`. 

## Comments

Should probably check if it's `None` or just add `""` as the default in the `os.environ.get` call.


Discussion:
(Added to 1.7.0 milestone, seems like we want a fix in place before QA begins.)

Resolving patch:
diff --git a/molecule/testinfra/conftest.py b/molecule/testinfra/conftest.py
--- a/molecule/testinfra/conftest.py
+++ b/molecule/testinfra/conftest.py
@@ -28,7 +28,16 @@ def securedrop_import_testinfra_vars(hostname, with_header=False):
     with io.open(filepath, 'r') as f:
         hostvars = yaml.safe_load(f)
 
-    if os.environ.get("MOLECULE_SCENARIO_NAME").endswith("focal"):
+    # Testing against both Focal and Xenial must be supported for now in both
+    # staging scenarios, and in prod via `USE_FOCAL=1 ./securedrop-admin verify`
+    testing_focal = False
+    scenario_env = "MOLECULE_SCENARIO_NAME"
+    if scenario_env in os.environ and os.environ.get(scenario_env).endswith("focal"):
+        testing_focal = True
+    if "USE_FOCAL" in os.environ:
+        testing_focal = True
+
+    if testing_focal:
         hostvars['securedrop_venv_site_packages'] = hostvars["securedrop_venv_site_packages"].format("3.8")  # noqa: E501
         hostvars['python_version'] = "3.8"
     else:


Test patch:
Repository: freedomofpress/securedrop

Issue:
1.8.0 translation feedback: Focal upgrade warning
## Description

AO [suggested](https://weblate.securedrop.org/translate/securedrop/securedrop/en/?checksum=c775f74150ac3872) changing `A manual update is urgently required to remain safe.` to `A manual upgrade is urgently required to remain safe.` in:

```
<strong>Critical Security:</strong>&nbsp;&nbsp;The operating system used by your SecureDrop servers 
will reach its end-of-life on April 30, 2021. A manual update is urgently required to remain safe. 
Please contact your adminstrator. <a href="//securedrop.org/xenial-eol" rel="noreferrer">Learn More</a>
```

as well as [here](https://weblate.securedrop.org/translate/securedrop/securedrop/en/?checksum=60e0284e103e2c51):
```
<strong>Critical Security:</strong>&nbsp;&nbsp;The operating system used by your SecureDrop servers 
has reached its end-of-life. A manual update is required to re-enable the Source Interface and remain safe. 
Please contact your administrator. <a href="//securedrop.org/xenial-eol" rel="noreferrer">Learn More</a>
```

There's also a typo in the first string: "adminstrator" needs another "i".




Discussion:
:+1: to  update->upgrade and typo fix (good catch!)

Resolving patch:
diff --git a/securedrop/journalist_app/col.py b/securedrop/journalist_app/col.py
--- a/securedrop/journalist_app/col.py
+++ b/securedrop/journalist_app/col.py
@@ -70,7 +70,7 @@ def delete_single(filesystem_id: str) -> werkzeug.Response:
                     # confirming the success of an operation.
                     escape(gettext("Success!")),
                     escape(gettext(
-                        "The account and data for the source {} has been deleted.").format(
+                        "The account and data for the source {} have been deleted.").format(
                             source.journalist_designation))
                 )
             ), 'success')


Test patch:
diff --git a/securedrop/tests/test_integration.py b/securedrop/tests/test_integration.py
--- a/securedrop/tests/test_integration.py
+++ b/securedrop/tests/test_integration.py
@@ -507,7 +507,7 @@ def test_delete_collection(mocker, source_app, journalist_app, test_journo):
 
         text = resp.data.decode('utf-8')
         assert escape(
-            "The account and data for the source {} has been deleted.".format(col_name)) in text
+            "The account and data for the source {} have been deleted.".format(col_name)) in text
 
         assert "No documents have been submitted!" in text
         assert async_genkey.called
Repository: freedomofpress/securedrop

Issue:
Backup script does not include custom logo
## Description

The backup script at https://github.com/freedomofpress/securedrop/blob/develop/install_files/ansible-base/roles/backup/files/backup.py currently does not add the custom logo, because it was not updated after #3057 changed the behavior to stop overwriting `logo.png` with the custom logo.

## Steps to Reproduce

(Requires prod or staging instance.)

1. Customize your logo
2. Run `./securedrop-admin backup`
3. Inspect backup contents

## Expected Behavior

Backup contains custom logo image.

## Actual Behavior

Backup only contains original default logo.


Discussion:
https://github.com/freedomofpress/securedrop-docs/pull/181 adds backup/restore docs for now, but I would suggest that we fix this properly with the next regular release at the latest.

Resolving patch:
diff --git a/install_files/ansible-base/roles/backup/files/backup.py b/install_files/ansible-base/roles/backup/files/backup.py
--- a/install_files/ansible-base/roles/backup/files/backup.py
+++ b/install_files/ansible-base/roles/backup/files/backup.py
@@ -1,8 +1,11 @@
 #!/opt/venvs/securedrop-app-code/bin/python
 """
-This script is copied to the App server and run by the Ansible playbook. When
-run (as root), it collects all of the necessary information to backup the 0.3
-system and stores it in /tmp/sd-backup-0.3-TIME_STAMP.tar.gz.
+This script is copied to the App server (to /tmp) and run by the Ansible playbook,
+typically via `securedrop-admin`.
+
+The backup file in the format sd-backup-$TIMESTAMP.tar.gz is then copied to the
+Admin Workstation by the playbook, and removed on the server. For further
+information and limitations, see https://docs.securedrop.org/en/stable/backup_and_restore.html
 """
 
 from datetime import datetime
@@ -19,14 +22,17 @@ def main():
 
     sd_code = '/var/www/securedrop'
     sd_config = os.path.join(sd_code, "config.py")
-    sd_custom_logo = os.path.join(sd_code, "static/i/logo.png")
+    sd_custom_logo = os.path.join(sd_code, "static/i/custom_logo.png")
 
     tor_hidden_services = "/var/lib/tor/services"
     torrc = "/etc/tor/torrc"
 
     with tarfile.open(backup_filename, 'w:gz') as backup:
         backup.add(sd_config)
-        backup.add(sd_custom_logo)
+
+        # If no custom logo has been configured, the file will not exist
+        if os.path.exists(sd_custom_logo):
+            backup.add(sd_custom_logo)
         backup.add(sd_data)
         backup.add(tor_hidden_services)
         backup.add(torrc)


Test patch:
Repository: freedomofpress/securedrop

Issue:
Be nicer to admins when an uploaded logo doesn't work.
## Description

If an uploaded logo can't be processed, the [error message shown to admins](https://github.com/freedomofpress/securedrop/blob/427675858c49ef67573bb002b37fce06c029a84d/securedrop/journalist_app/admin.py#L56) is curt: "Unable to process the image file. Try another one."

Let's say "Please try another one." 

Thanks to AO on Weblate for [pointing it out](https://weblate.securedrop.org/translate/securedrop/securedrop/en/?checksum=c2a5a60aaa227c25#comments).



Discussion:

Resolving patch:
diff --git a/securedrop/journalist_app/admin.py b/securedrop/journalist_app/admin.py
--- a/securedrop/journalist_app/admin.py
+++ b/securedrop/journalist_app/admin.py
@@ -53,7 +53,10 @@ def manage_config() -> Union[str, werkzeug.Response]:
                 flash(gettext("Image updated."), "logo-success")
             except Exception:
                 # Translators: This error is shown when an uploaded image cannot be used.
-                flash(gettext("Unable to process the image file. Try another one."), "logo-error")
+                flash(
+                    gettext("Unable to process the image file. Please try another one."),
+                    "logo-error"
+                )
             finally:
                 return redirect(url_for("admin.manage_config") + "#config-logoimage")
         else:


Test patch:
diff --git a/securedrop/tests/test_journalist.py b/securedrop/tests/test_journalist.py
--- a/securedrop/tests/test_journalist.py
+++ b/securedrop/tests/test_journalist.py
@@ -2025,7 +2025,7 @@ def test_logo_upload_save_fails(config, journalist_app, test_admin, locale):
                     )
 
                     assert page_language(resp.data) == language_tag(locale)
-                    msgids = ["Unable to process the image file. Try another one."]
+                    msgids = ["Unable to process the image file. Please try another one."]
                     with xfail_untranslated_messages(config, locale, msgids):
                         ins.assert_message_flashed(gettext(msgids[0]), "logo-error")
     finally: