internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: freedomofpress/securedrop

Issue:
Return current journalist ID when creating a token
## Description

The endpoint that generates auth tokens needs to return the current user's UUID so the client knows which user it is without having to hit the user data endpoint.


Discussion:

Resolving patch:
diff --git a/securedrop/journalist_app/api.py b/securedrop/journalist_app/api.py
--- a/securedrop/journalist_app/api.py
+++ b/securedrop/journalist_app/api.py
@@ -105,9 +105,12 @@ def get_token():
             journalist = Journalist.login(username, passphrase, one_time_code)
             token_expiry = datetime.utcnow() + timedelta(
                 seconds=TOKEN_EXPIRATION_MINS * 60)
-            response = jsonify({'token': journalist.generate_api_token(
-                 expiration=TOKEN_EXPIRATION_MINS * 60),
-                 'expiration': token_expiry.isoformat() + 'Z'})
+
+            response = jsonify({
+                'token': journalist.generate_api_token(expiration=TOKEN_EXPIRATION_MINS * 60),
+                'expiration': token_expiry.isoformat() + 'Z',
+                'journalist_uuid': journalist.uuid,
+            })
 
             # Update access metadata
             journalist.last_access = datetime.utcnow()


Test patch:
diff --git a/securedrop/tests/conftest.py b/securedrop/tests/conftest.py
--- a/securedrop/tests/conftest.py
+++ b/securedrop/tests/conftest.py
@@ -155,7 +155,8 @@ def test_journo(journalist_app):
                 'username': username,
                 'password': password,
                 'otp_secret': otp_secret,
-                'id': user.id}
+                'id': user.id,
+                'uuid': user.uuid}
 
 
 @pytest.fixture(scope='function')
diff --git a/securedrop/tests/test_journalist_api.py b/securedrop/tests/test_journalist_api.py
--- a/securedrop/tests/test_journalist_api.py
+++ b/securedrop/tests/test_journalist_api.py
@@ -41,6 +41,7 @@ def test_valid_user_can_get_an_api_token(journalist_app, test_journo):
                             headers=get_api_headers())
         observed_response = json.loads(response.data)
 
+        assert observed_response['journalist_uuid'] == test_journo['uuid']
         assert isinstance(Journalist.validate_api_token_and_get_user(
             observed_response['token']), Journalist) is True
         assert response.status_code == 200
Repository: freedomofpress/securedrop

Issue:
[0.12.0] Followup tasks for backup and restore
## Description

This is a followup from #4133 to:

- [ ] Update the docs to indicate that `securedrop-admin install` [here](https://github.com/freedomofpress/securedrop/pull/4133/files#diff-ed8769ff4c99e7e369511dcbe0ee78f9R159) will only fetch back the *THS files for SSH over LAN, not SSH over Tor. We should indicate in the docs for admins that have the default SSH over Tor configured, that they need to manually extract the *THS files from the restore tarball.
- [ ] Remove or document this stanza: https://github.com/freedomofpress/securedrop/pull/4133/files#diff-ff45d239e10e0aa98937629a72d83336R41 


Discussion:

Resolving patch:
diff --git a/install_files/ansible-base/roles/restore/files/restore.py b/install_files/ansible-base/roles/restore/files/restore.py
--- a/install_files/ansible-base/roles/restore/files/restore.py
+++ b/install_files/ansible-base/roles/restore/files/restore.py
@@ -8,7 +8,6 @@
 """
 
 import os
-import shutil
 import subprocess
 import sys
 import tarfile
@@ -36,13 +35,6 @@ def verify_args():
 def main():
     verify_args()
 
-    # Remove the /var/lib/tor/services directories to purge values that may have been
-    # generated by running the ansible playbooks
-    for d in ['journalist', 'source']:
-        full_path = os.path.join('/var/lib/tor/services', d)
-        if os.path.exists(full_path):
-            shutil.rmtree(full_path)
-
     with tarfile.open(sys.argv[1], 'r:*') as backup:
         # This assumes that both the old installation (source of the backup)
         # and the new installation (destination of the restore) used the


Test patch:
Repository: freedomofpress/securedrop

Issue:
Port support for creating dev data in staging VMs from `tbb-0.9.0` to `develop`
In the `tbb-0.9.0` branch for functional testing via the Tor browser we added support for user creation in staging VMs, using the `create-dev-data.py` script, see #3672 and https://github.com/freedomofpress/securedrop/commit/249dd16ae9ae861c801ce282302ef07b6d38a16b.

As part of incrementally merging the key changes in this branch, let's port this specific functionality to `develop`.


Discussion:

Resolving patch:
diff --git a/securedrop/create-dev-data.py b/securedrop/create-dev-data.py
--- a/securedrop/create-dev-data.py
+++ b/securedrop/create-dev-data.py
@@ -3,6 +3,7 @@
 
 import datetime
 import os
+import argparse
 
 from flask import current_app
 from sqlalchemy.exc import IntegrityError
@@ -15,7 +16,7 @@
 from models import Journalist, Reply, Source, Submission
 
 
-def main():
+def main(staging=False):
     app = journalist_app.create_app(config)
     with app.app_context():
         # Add two test users
@@ -26,6 +27,10 @@ def main():
                       test_password,
                       test_otp_secret,
                       is_admin=True)
+
+        if staging:
+            return
+
         add_test_user("dellsberg",
                       test_password,
                       test_otp_secret,
@@ -104,4 +109,9 @@ def create_source_and_submissions(num_submissions=2, num_replies=2):
 
 
 if __name__ == "__main__":  # pragma: no cover
-    main()
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--staging", help="Adding user for staging tests.",
+                        action="store_true")
+    args = parser.parse_args()
+
+    main(args.staging)


Test patch:
diff --git a/securedrop/tests/functional/README.md b/securedrop/tests/functional/README.md
new file mode 100644
--- /dev/null
+++ b/securedrop/tests/functional/README.md
@@ -0,0 +1,33 @@
+### To test in prod vms
+
+- `sudo -u www-data bash`
+- `cd /var/www/securedrop/`
+- `./manage.py reset`    # This will clean the DB for testing
+- `./create-dev-data.py --staging`
+
+Update this information to the `tests/functional/instance_information.json` file.
+
+The content of the file looks like below.
+
+```
+{
+    "hidserv_token": "asfjsdfag",
+    "journalist_location": "http://thejournalistfqb.onion",
+    "source_location": "http://thesourceadsfa.onion",
+    "sleep_time": 10,
+    "user": {
+        "name": "journalist",
+        "password": "WEjwn8ZyczDhQSK24YKM8C9a",
+        "secret": "JHCOGO7VCER3EJ4L"
+    }
+}
+```
+
+### Run the tests
+
+```
+cd securedrop
+./bin/dev-shell ./bin/run-test -v tests/functional/
+```
+You may wish to append a pipe to less (i.e. `| less`), as a failure may generate
+many pages of output, making it difficult to scroll back.
Repository: freedomofpress/securedrop

Issue:
Update SQLAlchemy to 1.3.0
## Description

Multiple vulnerabilities were reported in SQLAlchemy <1.3.0:
- CVE-2019-7164 (moderate) : SQLAlchemy through 1.2.17 and 1.3.x through 1.3.0b2 allows SQL Injection via the order_by parameter.
- CVE-2019-7548 (moderate) : SQLAlchemy 1.2.17 has SQL Injection when the group_by parameter can be controlled.

We should update this dependency ASAP for the SecureDrop application.



Discussion:

Resolving patch:
diff --git a/install_files/ansible-base/callback_plugins/ansible_version_check.py b/install_files/ansible-base/callback_plugins/ansible_version_check.py
--- a/install_files/ansible-base/callback_plugins/ansible_version_check.py
+++ b/install_files/ansible-base/callback_plugins/ansible_version_check.py
@@ -21,7 +21,7 @@ class CallbackModule(CallbackBase):
     def __init__(self):
         # Can't use `on_X` because this isn't forwards compatible
         # with Ansible 2.0+
-        required_version = '2.6.8'  # Keep synchronized with requirements files
+        required_version = '2.6.14'  # Keep synchronized with requirements files
         if not ansible.__version__.startswith(required_version):
             print_red_bold(
                 "SecureDrop restriction: only Ansible {version}.*"


Test patch:
diff --git a/securedrop/tests/test_journalist.py b/securedrop/tests/test_journalist.py
--- a/securedrop/tests/test_journalist.py
+++ b/securedrop/tests/test_journalist.py
@@ -1148,10 +1148,10 @@ def test_admin_add_user_integrity_error(journalist_app, test_admin, mocker):
     log_event = mocked_error_logger.call_args[0][0]
     if six.PY2:
         assert ("Adding user 'username' failed: (__builtin__.NoneType) "
-                "None [SQL: 'STATEMENT'] [parameters: 'PARAMETERS']") in log_event
+                "None\n[SQL: STATEMENT]\n[parameters: 'PARAMETERS']") in log_event
     else:
         assert ("Adding user 'username' failed: (builtins.NoneType) "
-                "None [SQL: 'STATEMENT'] [parameters: 'PARAMETERS']") in log_event
+                "None\n[SQL: STATEMENT]\n[parameters: 'PARAMETERS']") in log_event
 
 
 def test_logo_upload_with_valid_image_succeeds(journalist_app, test_admin):
Repository: freedomofpress/securedrop

Issue:
Include `fingerprint` in the `key` object in a `Source` response
## Description

To easily manipulate and reference PGP keys using GPG or GPG-wrappers, we need to know the key's fingerprint. It may make sense for the server to calculate this for us and send it as part of the response.


Discussion:
we should implement this as part of resolving https://github.com/freedomofpress/securedrop-client/issues/363. The reason why is because the existing logic on the client side parsing stdout to get the key fingerprint needs modification when we use the qubes RPC service for importing keys. This is a symptom of the fact that parsing stdout to get key fingerprints is pretty brittle. We should just expose the fingerprint via the API. 

Resolving patch:
diff --git a/securedrop/models.py b/securedrop/models.py
--- a/securedrop/models.py
+++ b/securedrop/models.py
@@ -101,6 +101,18 @@ def collection(self):
         collection.sort(key=lambda x: int(x.filename.split('-')[0]))
         return collection
 
+    @property
+    def fingerprint(self):
+        return current_app.crypto_util.getkey(self.filesystem_id)
+
+    @fingerprint.setter
+    def fingerprint(self, value):
+        raise NotImplementedError
+
+    @fingerprint.deleter
+    def fingerprint(self):
+        raise NotImplementedError
+
     @property
     def public_key(self):
         return current_app.crypto_util.export_pubkey(self.filesystem_id)
@@ -136,7 +148,8 @@ def to_json(self):
             'interaction_count': self.interaction_count,
             'key': {
               'type': 'PGP',
-              'public': self.public_key
+              'public': self.public_key,
+              'fingerprint': self.fingerprint
             },
             'number_of_documents': docs_msg_count['documents'],
             'number_of_messages': docs_msg_count['messages'],


Test patch:
diff --git a/securedrop/tests/test_journalist_api.py b/securedrop/tests/test_journalist_api.py
--- a/securedrop/tests/test_journalist_api.py
+++ b/securedrop/tests/test_journalist_api.py
@@ -277,6 +277,8 @@ def test_authorized_user_gets_single_source(journalist_app, test_source,
         assert response.status_code == 200
 
         assert response.json['uuid'] == test_source['source'].uuid
+        assert response.json['key']['fingerprint'] == \
+            test_source['source'].fingerprint
         assert 'BEGIN PGP PUBLIC KEY' in response.json['key']['public']
Repository: freedomofpress/securedrop

Issue:
Check documentation links as part of docs linting
## Description

Sphinx [linkcheck](https://www.sphinx-doc.org/en/master/usage/builders/index.html#sphinx.builders.linkcheck.CheckExternalLinksBuilder) allows the verification of links with the `requests` library to ensure that the links are still valid and active. It might be useful to run this regularly or as part of CI to catch dead or broken links.
## User Stories
As a user, clicking on a link and getting a 404 can be a a frustrating experience.




Discussion:
Great idea! I'll take a look into this—

Could it be run as part of `make docs-lint`?
possibly, how long does this take to run on the current docs? 
Not sure. Haven't been able to check with my current environment problems (#4472).

Will try to test this using a previous docker image.
Time to run sphinx's linkcheck feature is at least a few minutes. I'm at 5 minutes right now on an iffy connection, 70% done.

Any objections to making this its own make target, `make docs-linkcheck`? Would be useful for those regularly working on docs as a whole.
yeah making it a separate target from `docs-lint` is a good call given the time to run :+1: 

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -299,3 +299,14 @@
 
 # If true, do not generate a @detailmenu in the "Top" node's menu.
 # texinfo_no_detailmenu = False
+
+# -- Options for linkcheck --
+
+linkcheck_retries = 3
+
+linkcheck_ignore = [
+  r'http://127.0.0.1(:\d+)?/?',
+  r'http://localhost(:\d+)?/?',
+  'https://forum.securedrop.org/admin/users/list/active',
+  'https://weblate.securedrop.org/projects/securedrop/securedrop/#repository',
+]


Test patch:
diff --git a/docs/development/testing_continuous_integration.rst b/docs/development/testing_continuous_integration.rst
--- a/docs/development/testing_continuous_integration.rst
+++ b/docs/development/testing_continuous_integration.rst
@@ -62,7 +62,7 @@ output as shown below:
     This message shows that your installation appears to be working correctly.
     ...
 
-.. _Docker installation: https://www.docker.com/community-edition#/download
+.. _Docker installation: https://docs.docker.com/install/
 
 Setup Environment Parameters
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Repository: freedomofpress/securedrop

Issue:
securedrop-admin logs command should clean up logs from admin workstation and server after task completion
## Description

When running `./securedrop-admin logs`, the unencrypted log tarball is stored in the user's home directory, root owned with a+r permissions. Both encrypted and non-encrypted log tarballs are also stored in install_files/ansible-base.

The `./securedrop-admin logs` command should clean up the unencrypted tarballs from both the admin workstation as well as both app and mon servers' home directories.

## User Stories

As an administrator, I do not want old logs to use up space nor to be available in the clear to a non-privileged user



Discussion:
Related if we wanted to delete files securely without relying on shell commands: https://github.com/ansible/ansible/issues/20958 

Resolving patch:
diff --git a/admin/securedrop_admin/__init__.py b/admin/securedrop_admin/__init__.py
--- a/admin/securedrop_admin/__init__.py
+++ b/admin/securedrop_admin/__init__.py
@@ -41,6 +41,8 @@
 sdlog = logging.getLogger(__name__)
 RELEASE_KEY = '22245C81E3BAEB4138B36061310F561200F4AD77'
 DEFAULT_KEYSERVER = 'hkps://keys.openpgp.org'
+SUPPORT_ONION_URL = 'http://support6kv2242qx.onion'
+SUPPORT_URL = 'https://support.freedom.press'
 EXIT_SUCCESS = 0
 EXIT_SUBPROCESS_ERROR = 1
 EXIT_INTERRUPT = 2
@@ -741,8 +743,8 @@ def get_logs(args):
         os.path.join(args.ansible_path, 'securedrop-logs.yml'),
     ]
     subprocess.check_call(ansible_cmd, cwd=args.ansible_path)
-    sdlog.info("Encrypt logs and send to securedrop@freedom.press or upload "
-               "to the SecureDrop support portal.")
+    sdlog.info("Please send the encrypted logs to securedrop@freedom.press or "
+               "upload them to the SecureDrop support portal: " + SUPPORT_URL)
     return 0
 
 


Test patch:
Repository: freedomofpress/securedrop

Issue:
Adding Catalán (ca) as a translated language
https://forum.securedrop.org/t/adding-catalan-to-securedrop-0-14-0/1228

The localization team requested to add Catalán (ca) as a translated language to the 0.14.0 release.

https://forum.securedrop.org/t/adding-catalan-to-securedrop-0-14-0/1228


Discussion:

Resolving patch:
diff --git a/securedrop/i18n_tool.py b/securedrop/i18n_tool.py
--- a/securedrop/i18n_tool.py
+++ b/securedrop/i18n_tool.py
@@ -37,6 +37,7 @@ class I18NTool(object):
     #
     SUPPORTED_LANGUAGES = {
         'ar': {'name': 'Arabic', 'desktop': 'ar', },
+        'ca': {'name': 'Catalan', 'desktop': 'ca', },
         'de_DE': {'name': 'German', 'desktop': 'de_DE', },
         'es_ES': {'name': 'Spanish', 'desktop': 'es_ES', },
         'fr_FR': {'name': 'French', 'desktop': 'fr', },


Test patch:
Repository: freedomofpress/securedrop

Issue:
securedrop-admin setup fails on Tails 4
## Description

We need to add a torify call to the `pip install` commands in `securedrop-admin setup`

## Steps to Reproduce

1. Install the Tails 4 beta: https://tails.boum.org/news/test_4.0-beta2/
2. Set up persistence
3. Clone the SecureDrop repo
4. From `~/Persistent/securedrop`, run `./securedrop-admin setup`

## Expected Behavior

apt and python dependencies are installed

## Actual Behavior

apt dependencies are installed, but then:

```
ERROR: Failed to install pip dependencies. Check network connection and try again.
```

## Comments

Rerunning with `-v` indicates that the problem is a connection failure. Prefixing the `./admin/.venv/bin/pip install -r admin/requirements.txt [...more options snipped....]` command with `torify` resolves. 

Related to #4669


Discussion:
@pradyunsg should we also file a bug against `pip` for this? 

If someone is willing to post the options passed to pip and more complete output from pip (redacting any sensitive information appropriately), it'll be much easier for me to tell if it's a pip thing or a tor/tails thing or an invocation issue. The arguments/output would help me confirm that.

---

I feel that this is more of a situation of needing to push pip's connections through Tor, when on Tails.

A quick look at the corresponding code makes me think that the issue is either in pip, or `admin/bootstrap.py:is_tails()` isn't working right. There's already `maybe_torify()` at that call site (I think the OP is suggesting that's what is needed). I'm not familiar with the code here, so I'm guessing here. :)

https://github.com/freedomofpress/securedrop/blob/7a9750d93b0d8be77fceca1a1bdb6446b5433038/admin/bootstrap.py#L185

Tails 4.0 is scheduled to be released on October 22 (per https://tails.boum.org/contribute/calendar/)
yep looks like the issue is `is_tails()` (`lsb_release` is not Tails), meaning this ticket has the same root cause as #4767
Filed a bug against Tails instead of pip: https://redmine.tails.boum.org/code/issues/17074

Adding an unreliable check (is the system Debian? is the node name `amnesia`? - probably Tails then) to unblock Tails 4.0 testing until bug above is fixed.

Resolving patch:
diff --git a/admin/bootstrap.py b/admin/bootstrap.py
--- a/admin/bootstrap.py
+++ b/admin/bootstrap.py
@@ -66,6 +66,12 @@ def is_tails():
                                      shell=True).strip()
     except subprocess.CalledProcessError:
         id = None
+
+    # dirty hack to unreliably detect Tails 4.0~beta2
+    if id == 'Debian':
+        if os.uname()[1] == 'amnesia':
+            id = 'Tails'
+
     return id == 'Tails'
 
 


Test patch:
Repository: freedomofpress/securedrop

Issue:
Tails 4 desktop shortcuts not showing up
## Description

After tailsconfig on Tails 4 beta, .desktop files are not working on the Desktop

## Steps to Reproduce

1. Install the Tails 4 beta: https://tails.boum.org/news/test_4.0-beta2/
2. Set up persistence
3. Clone the SecureDrop repo, install SD (or copy over your keys/creds/`site-specific`)
4. From `~/Persistent/securedrop`, run `./securedrop-admin tailsconfig`

## Expected Behavior

Desktop shortcuts appear on desktop to the source and journalist interfaces, when clicked they open Tor Browser

## Actual Behavior

Desktop shortcuts appear, but they open in Text Editor when clicked

## Comments

Needs investigation, note we had a similar issue back when we updated to Tails 3.3: #2586

Related to #4669


Discussion:
Some notes: the situation has changed a little in Gnome 3.3 (buster) vs 3.2 (stretch). Desktop icons are disabled by default and reenabled via an extension: 
https://gitlab.gnome.org/World/ShellExtensions/desktop-icons

setting `metadata::trusted` via `gio` directly isn't working for me, but there is a right-click menu item **Allow Launching** that makes an icon launchable. The code behind it is here:

https://gitlab.gnome.org/World/ShellExtensions/desktop-icons/blob/master/fileItem.js#L528

probably the best bet is to replicate any new functionality there  in `securedrop_init.py` along with what's already being set for Tails 3.*.

Resolving patch:
diff --git a/install_files/ansible-base/roles/tails-config/files/securedrop_init.py b/install_files/ansible-base/roles/tails-config/files/securedrop_init.py
--- a/install_files/ansible-base/roles/tails-config/files/securedrop_init.py
+++ b/install_files/ansible-base/roles/tails-config/files/securedrop_init.py
@@ -126,7 +126,14 @@
     subprocess.call(['ln', '-s', path_persistent_desktop + shortcut,
                      path_desktop + shortcut], env=env)
     subprocess.call(['gio', 'set', path_desktop + shortcut,
-                     'metadata::trusted', 'yes'], env=env)
+                     'metadata::trusted', 'true'], env=env)
+
+# in Tails 4, reload gnome-shell desktop icons extension to update with changes above
+cmd = ["lsb_release", "--id", "--short"]
+p = subprocess.check_output(cmd)
+distro_id = p.rstrip()
+if distro_id == 'Debian' and os.uname()[1] == 'amnesia':
+    subprocess.call(['gnome-shell-extension-tool', '-r', 'desktop-icons@csoriano'], env=env)
 
 # reacquire uid0 and notify the user
 os.setresuid(0, 0, -1)


Test patch:
Repository: freedomofpress/securedrop

Issue:
Upgrade Ansble to 2.7 series
## Description

SecureDrop currently uses the Ansible 2.6 series which is approaching end-of-life. Ansible major versions are supported (receive security patches) for [three major releases](https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html).

We should move to Ansible 2.7 to ensure a smooth transition, which would allow us to rapidly patch should there be a need to quickly patch.

## User Stories

As a developer, I want to make sure my dependencies are being maintained for security issues.



Discussion:

Resolving patch:
diff --git a/install_files/ansible-base/callback_plugins/ansible_version_check.py b/install_files/ansible-base/callback_plugins/ansible_version_check.py
--- a/install_files/ansible-base/callback_plugins/ansible_version_check.py
+++ b/install_files/ansible-base/callback_plugins/ansible_version_check.py
@@ -21,7 +21,7 @@ class CallbackModule(CallbackBase):
     def __init__(self):
         # Can't use `on_X` because this isn't forwards compatible
         # with Ansible 2.0+
-        required_version = '2.6.19'  # Keep synchronized with requirements files
+        required_version = '2.7.13'  # Keep synchronized with requirements files
         if not ansible.__version__.startswith(required_version):
             print_red_bold(
                 "SecureDrop restriction: only Ansible {version}.*"


Test patch:
Repository: freedomofpress/securedrop

Issue:
Add Czech to list of supported languages
## Description

SecureDrop has been translated into Czech. Let's make it official.



Discussion:

Resolving patch:
diff --git a/securedrop/i18n_tool.py b/securedrop/i18n_tool.py
--- a/securedrop/i18n_tool.py
+++ b/securedrop/i18n_tool.py
@@ -37,6 +37,7 @@ class I18NTool(object):
     SUPPORTED_LANGUAGES = {
         'ar': {'name': 'Arabic', 'desktop': 'ar', },
         'ca': {'name': 'Catalan', 'desktop': 'ca', },
+        'cs': {'name': 'Czech', 'desktop': 'cs', },
         'de_DE': {'name': 'German', 'desktop': 'de_DE', },
         'el': {'name': 'Greek', 'desktop': 'el', },
         'es_ES': {'name': 'Spanish', 'desktop': 'es_ES', },


Test patch: