internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: freedomofpress/securedrop

Issue:
[1.1.0-rc4] "Unable to create virtualenv. Check network settings and try again"
(Tested on a Tails 3.16 Admin Workstation by checking out 1.1.0-rc4 tag, without updating my servers.)

As expected, running `securedrop-admin` commands triggered the "run setup" step. However, the `securedrop-admin setup` step itself did not complete successfully; it went pretty far along but finally failed with this error:

"Unable to create virtualenv. Check network settings and try again"

Tor seems to be working fine. Possibly intermittent issues but good to warn users about and have mitigation instructions if it is likely to arise during updates.
[1.1.0-rc4] "Unable to create virtualenv. Check network settings and try again"
(Tested on a Tails 3.16 Admin Workstation by checking out 1.1.0-rc4 tag, without updating my servers.)

As expected, running `securedrop-admin` commands triggered the "run setup" step. However, the `securedrop-admin setup` step itself did not complete successfully; it went pretty far along but finally failed with this error:

"Unable to create virtualenv. Check network settings and try again"

Tor seems to be working fine. Possibly intermittent issues but good to warn users about and have mitigation instructions if it is likely to arise during updates.


Discussion:

Resolving patch:
diff --git a/admin/bootstrap.py b/admin/bootstrap.py
--- a/admin/bootstrap.py
+++ b/admin/bootstrap.py
@@ -64,12 +64,12 @@ def run_command(command):
 def is_tails():
     try:
         id = subprocess.check_output('lsb_release --id --short',
-                                     shell=True).strip()
+                                     shell=True).decode('utf-8').strip()
     except subprocess.CalledProcessError:
         id = None
 
     # dirty hack to unreliably detect Tails 4.0~beta2
-    if id == b'Debian':
+    if id == 'Debian':
         if os.uname()[1] == 'amnesia':
             id = 'Tails'
 


Test patch:
Repository: freedomofpress/securedrop

Issue:
[1.1.0-rc4] "Unable to create virtualenv. Check network settings and try again"
(Tested on a Tails 3.16 Admin Workstation by checking out 1.1.0-rc4 tag, without updating my servers.)

As expected, running `securedrop-admin` commands triggered the "run setup" step. However, the `securedrop-admin setup` step itself did not complete successfully; it went pretty far along but finally failed with this error:

"Unable to create virtualenv. Check network settings and try again"

Tor seems to be working fine. Possibly intermittent issues but good to warn users about and have mitigation instructions if it is likely to arise during updates.


Discussion:

Resolving patch:
diff --git a/admin/bootstrap.py b/admin/bootstrap.py
--- a/admin/bootstrap.py
+++ b/admin/bootstrap.py
@@ -64,12 +64,12 @@ def run_command(command):
 def is_tails():
     try:
         id = subprocess.check_output('lsb_release --id --short',
-                                     shell=True).strip()
+                                     shell=True).decode('utf-8').strip()
     except subprocess.CalledProcessError:
         id = None
 
     # dirty hack to unreliably detect Tails 4.0~beta2
-    if id == b'Debian':
+    if id == 'Debian':
         if os.uname()[1] == 'amnesia':
             id = 'Tails'
 


Test patch:
Repository: freedomofpress/securedrop

Issue:
Add Slovak to list of supported languages
## Description

SecureDrop has been translated into Slovak, so let's add it to the supported list.


Discussion:
We're waiting for a potential reviewer to start the nomination process.
Slovak language has just been fully reviewed.

Resolving patch:
diff --git a/securedrop/i18n_tool.py b/securedrop/i18n_tool.py
--- a/securedrop/i18n_tool.py
+++ b/securedrop/i18n_tool.py
@@ -50,6 +50,7 @@ class I18NTool(object):
         'pt_BR': {'name': 'Portuguese, Brasil', 'desktop': 'pt_BR', },
         'ro': {'name': 'Romanian', 'desktop': 'ro', },
         'ru': {'name': 'Russian', 'desktop': 'ru', },
+        'sk': {'name': 'Slovak', 'desktop': 'sk', },
         'sv': {'name': 'Swedish', 'desktop': 'sv', },
         'tr': {'name': 'Turkish', 'desktop': 'tr', },
         'zh_Hant': {'name': 'Chinese, Traditional', 'desktop': 'zh_Hant', },


Test patch:
Repository: freedomofpress/securedrop

Issue:
Please make the rqrequeue service quieter
## Description

The rqrequeue service feels compelled to report that it has nothing to do, resulting in an endless stream of "No interrupted jobs found in started job registry." messages. This is not helpful during normal operations, and annoying during development.


Discussion:

Resolving patch:
diff --git a/securedrop/worker.py b/securedrop/worker.py
--- a/securedrop/worker.py
+++ b/securedrop/worker.py
@@ -88,7 +88,7 @@ def requeue_interrupted_jobs(queue_name=None):
     logging.debug("candidate job ids: {}".format(job_ids))
 
     if not job_ids:
-        logging.info("No interrupted jobs found in started job registry.")
+        logging.debug("No interrupted jobs found in started job registry.")
 
     for job_id in job_ids:
         logging.debug("Considering job %s", job_id)


Test patch:
Repository: freedomofpress/securedrop

Issue:
app code attempting to delete /var/lib/securedrop/store
## Description

When I'm deleting files I'm seeing an OSSEC alert indicating that the app code is trying to rm `/var/lib/securedrop/store` (which contains all submissions). It's being stopped thanks to an AppArmor denial but this means there's a potentially very destructive bug and we need to carefully investigate why this is happening.

## Expected behavior

No OSSEC alert

## Actual behavior

```
OSSEC HIDS Notification.
2019 Nov 25 21:07:04

Received From: (app) 10.20.2.2->/var/log/syslog
Rule: 100012 fired (level 7) -> "Apparmor denied event"
Portion of the log(s):

Nov 25 21:07:04 app kernel: [ 2160.102070] audit: type=1400 audit(1574734024.249:26): apparmor="DENIED" operation="rmdir" profile="/usr/sbin/apache2" name="/var/lib/securedrop/store/" pid=3067 comm="apache2" requested_mask="d" denied_mask="d" fsuid=33 ouid=33



 --END OF NOTIFICATION
```

## Comments

I would first investigate this method as it was recently merged and is performing deletions: https://github.com/freedomofpress/securedrop/blob/1d8484e3d42f63b9cec68be14e587175fa01adfc/securedrop/store.py#L235



Discussion:

Resolving patch:
diff --git a/securedrop/store.py b/securedrop/store.py
--- a/securedrop/store.py
+++ b/securedrop/store.py
@@ -68,6 +68,25 @@ class NotEncrypted(Exception):
     pass
 
 
+def safe_renames(old, new):
+    """safe_renames(old, new)
+
+    This is a modified version of Python's os.renames that does not
+    prune directories.
+    Super-rename; create directories as necessary without deleting any
+    left empty.  Works like rename, except creation of any intermediate
+    directories needed to make the new pathname good is attempted
+    first.
+    Note: this function can fail with the new directory structure made
+    if you lack permissions needed to unlink the leaf directory or
+    file.
+    """
+    head, tail = os.path.split(new)
+    if head and tail and not os.path.exists(head):
+        os.makedirs(head)
+    os.rename(old, new)
+
+
 class Storage:
 
     def __init__(self, storage_path, temp_dir, gpg_key):
@@ -207,7 +226,7 @@ def move_to_shredder(self, path: str):
         """
         Moves content from the store to the shredder for secure deletion.
 
-        Python's os.renames (and the underlying rename(2) calls) will
+        Python's safe_renames (and the underlying rename(2) calls) will
         silently overwrite content, which could bypass secure
         deletion, so we create a temporary directory under the
         shredder directory and move the specified content there.
@@ -230,7 +249,7 @@ def move_to_shredder(self, path: str):
         relpath = os.path.relpath(path, start=self.storage_path)
         dest = os.path.join(tempfile.mkdtemp(dir=self.__shredder_path), relpath)
         current_app.logger.info("Moving {} to shredder: {}".format(path, dest))
-        os.renames(path, dest)
+        safe_renames(path, dest)
 
     def clear_shredder(self):
         current_app.logger.info("Clearing shredder")


Test patch:
Repository: freedomofpress/securedrop

Issue:
Tails NetworkManager hook still running under Python2
## Description
Despite the shebang line in the `securedrop_init.py` pointing to Python3, the script is still run via Python2 due to the calling script hardcoding `/usr/bin/python` in the NetworkManager hook:

https://github.com/freedomofpress/securedrop/blob/1d8484e3d42f63b9cec68be14e587175fa01adfc/install_files/ansible-base/roles/tails-config/files/65-configure-tor-for-securedrop.sh#L17

## Steps to Reproduce

Review diffs and relevant files above.

## Expected Behavior

All Tails 4 admin tooling is Python 3. 

## Actual Behavior
We're still calling the nm hook via Python 2. 

## Comments
The PR to resolve is simple, but we'll have to factor in the QA time to verify we haven't introduced side-effects.



Discussion:

Resolving patch:
diff --git a/install_files/ansible-base/roles/tails-config/files/securedrop_init.py b/install_files/ansible-base/roles/tails-config/files/securedrop_init.py
--- a/install_files/ansible-base/roles/tails-config/files/securedrop_init.py
+++ b/install_files/ansible-base/roles/tails-config/files/securedrop_init.py
@@ -138,10 +138,10 @@
 # reacquire uid0 and notify the user
 os.setresuid(0, 0, -1)
 os.setresgid(0, 0, -1)
+success_message = 'You can now access the Journalist Interface.\nIf you are an admin, you can now SSH to the servers.'  # noqa: E501
 subprocess.call(['tails-notify-user',
                  'SecureDrop successfully auto-configured!',
-                 'You can now access the Journalist Interface.\n',
-                 'If you are an admin, you can now SSH to the servers.'])
+                 success_message])
 
 # As the amnesia user, check for SecureDrop workstation updates.
 os.setresgid(amnesia_gid, amnesia_gid, -1)
@@ -152,6 +152,6 @@
                                   'check_for_updates'], env=env)
 
 flag_location = "/home/amnesia/Persistent/.securedrop/securedrop_update.flag"
-if 'Update needed' in output or os.path.exists(flag_location):
+if b'Update needed' in output or os.path.exists(flag_location):
     # Start the SecureDrop updater GUI.
     subprocess.Popen(['python3', path_gui_updater], env=env)


Test patch:
Repository: freedomofpress/securedrop

Issue:
Graphical updater "is already running" dialog during late stages of the update process
## Description

The graphical updater briefly shows an "is already running" dialog during the last stages of the process. The dialog can be dismissed and has no negative consequences.

Tested on Tails 3.16.

## Steps to Reproduce

Perform a full graphical update using the graphical updater, with the Tails admin password set.

## Expected Behavior

The graphical update completes without any unusual messages.

## Actual Behavior

An "already running" dialog briefly appears towards the end of the process.


Discussion:
This happens because the network hook is now always triggered by tailsconfig, rather than only on a change in the network hook script (consequence of v3 services.)

If it's a problem, probably the easiest thing to do would be to have the second copy of securedropupdater just exit silently.

Resolving patch:
diff --git a/journalist_gui/journalist_gui/SecureDropUpdater.py b/journalist_gui/journalist_gui/SecureDropUpdater.py
--- a/journalist_gui/journalist_gui/SecureDropUpdater.py
+++ b/journalist_gui/journalist_gui/SecureDropUpdater.py
@@ -7,6 +7,7 @@
 import pexpect
 import socket
 import sys
+import syslog as log
 
 from journalist_gui import updaterUI, strings, resources_rc  # noqa
 
@@ -35,9 +36,7 @@ def prevent_second_instance(app: QtWidgets.QApplication, name: str) -> None:  #
         app.instance_binding.bind(IDENTIFIER)
     except OSError as e:
         if e.errno == ALREADY_BOUND_ERRNO:
-            err_dialog = QtWidgets.QMessageBox()
-            err_dialog.setText(name + strings.app_is_already_running)
-            err_dialog.exec()
+            log.syslog(log.LOG_NOTICE, name + strings.app_is_already_running)
             sys.exit()
         else:
             raise


Test patch:
diff --git a/journalist_gui/test_gui.py b/journalist_gui/test_gui.py
--- a/journalist_gui/test_gui.py
+++ b/journalist_gui/test_gui.py
@@ -13,7 +13,7 @@
 
 
 @mock.patch('journalist_gui.SecureDropUpdater.sys.exit')
-@mock.patch('journalist_gui.SecureDropUpdater.QtWidgets.QMessageBox')
+@mock.patch('syslog.syslog')
 class TestSecondInstancePrevention(unittest.TestCase):
     def setUp(self):
         self.mock_app = mock.MagicMock()
Repository: freedomofpress/securedrop

Issue:
Add an option to "securedrop-admin restore" to skip restoration of the instance's tor config
## Description
The current restore procedure restores both SecureDrop data and Tor configurations, overwriting the existing data and config. In situations where it's not desirable to overwrite the Tor configuration, it should be possible to to do a data-only restore. Such situations might include:
- When an instance is recreated with a new address (such as a v3 address) 
- When one or more service addresses have been rotated for security reasons
- When an existing dataset is being replicated on a staging or prod instance (for QA/integration testing)

## User Research Evidence
Personal experience only.

## User Stories

As a tester, I want to set up SecureDrop servers with a predefined dataset, without hidden service name collisions with other testers.
As an administrator, I want to be able to restore a backup to a new or updated SecureDrop instance, preserving the new service addresses.



Discussion:

Resolving patch:
diff --git a/admin/securedrop_admin/__init__.py b/admin/securedrop_admin/__init__.py
--- a/admin/securedrop_admin/__init__.py
+++ b/admin/securedrop_admin/__init__.py
@@ -698,12 +698,25 @@ def restore_securedrop(args):
     # Would like readable output if there's a problem
     os.environ["ANSIBLE_STDOUT_CALLBACK"] = "debug"
 
-    ansible_cmd = [
+    ansible_cmd_full_restore = [
         'ansible-playbook',
         os.path.join(args.ansible_path, 'securedrop-restore.yml'),
         '-e',
         "restore_file='{}'".format(restore_file_basename),
     ]
+
+    ansible_cmd_skip_tor = [
+        'ansible-playbook',
+        os.path.join(args.ansible_path, 'securedrop-restore.yml'),
+        '-e',
+        "restore_file='{}' restore_skip_tor='True'".format(restore_file_basename),
+    ]
+
+    if args.restore_skip_tor:
+        ansible_cmd = ansible_cmd_skip_tor
+    else:
+        ansible_cmd = ansible_cmd_full_restore
+
     return subprocess.check_call(ansible_cmd, cwd=args.ansible_path)
 
 
@@ -935,6 +948,10 @@ class ArgParseFormatterCombo(argparse.ArgumentDefaultsHelpFormatter,
                                           help=restore_securedrop.__doc__)
     parse_restore.set_defaults(func=restore_securedrop)
     parse_restore.add_argument("restore_file")
+    parse_restore.add_argument("--preserve-tor-config", default=False,
+                               action='store_true',
+                               dest='restore_skip_tor',
+                               help="Preserve the server's current Tor config")
 
     parse_update = subparsers.add_parser('update', help=update.__doc__)
     parse_update.set_defaults(func=update)


Test patch:
Repository: freedomofpress/securedrop

Issue:
update Ansible due to CVE-2019-14864 
## Description

[CVE-2019-14864](https://nvd.nist.gov/vuln/detail/CVE-2019-14864) is a vulnerability in Ansible's `no_log` flag for the splunk and sumologic plugins (sensitive data is incorrectly logged) but neither of which we're using. Regardless, we should update Ansible to a version that does not have this vulnerability in the next release.

@emkll also pointed out to me that this is a [good time](https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html) to get onto the 2.8 series since the 2.7 series will become unmaintained when 2.10 is released (it's in development). 


Discussion:
Seems like we might as well jump to 2.9 for the same reason -- barring other security patches, it'll be longer before we're forced to update it again.

Resolving patch:
diff --git a/install_files/ansible-base/callback_plugins/ansible_version_check.py b/install_files/ansible-base/callback_plugins/ansible_version_check.py
--- a/install_files/ansible-base/callback_plugins/ansible_version_check.py
+++ b/install_files/ansible-base/callback_plugins/ansible_version_check.py
@@ -19,13 +19,18 @@ def print_red_bold(text):
 
 class CallbackModule(CallbackBase):
     def __init__(self):
-        # Can't use `on_X` because this isn't forwards compatible
-        # with Ansible 2.0+
-        required_version = '2.7.13'  # Keep synchronized with requirements files
-        if not ansible.__version__.startswith(required_version):
+        # The acceptable version range needs to be synchronized with
+        # requirements files.
+        viable_start = [2, 9, 7]
+        viable_end = [2, 10, 0]
+        ansible_version = [int(v) for v in ansible.__version__.split('.')]
+        if not (viable_start <= ansible_version < viable_end):
             print_red_bold(
-                "SecureDrop restriction: only Ansible {version}.*"
-                "is supported."
-                .format(version=required_version)
+                "SecureDrop restriction: Ansible version must be at least {viable_start} "
+                "and less than {viable_end}."
+                .format(
+                    viable_start='.'.join(str(v) for v in viable_start),
+                    viable_end='.'.join(str(v) for v in viable_end),
+                )
             )
             sys.exit(1)


Test patch:
Repository: freedomofpress/securedrop

Issue:
When a logged-in source logs in again, our informational message points the wrong way
## Description

In the source interface, when a source who is already logged in logs in again, they see a message saying "You are already logged in. Please verify your codename below as it may differ from the one displayed on the previous page." But the "show codename" tool is now located _above_ the message.

## Steps to Reproduce

(Cribbed from the [1.3.0 test plan](https://github.com/freedomofpress/securedrop/wiki/1.3.0-Test-plan#5075).)

- Open the Source Interface in two different tabs (tab A, tab B) and navigate to `/generate` in both:
  - Verify that two different source codenames are displayed, and note them
- Click **Submit Documents** in tab A:
  - The `/lookup` page is displayed, and when **Show** is clicked in the codename hint, the tab A codename is displayed
- Click **Submit Documents** in tab B:
  - The `/lookup` page is displayed with a blue flash message saying "You are already logged in. Please verify your codename below as it may differ from the one displayed on the previous page."

## Expected Behavior

That we would direct the source to the right place to verify their codename.

## Actual Behavior

We misdirect them, grinning maniacally and chortling to ourselves at our ingenious trickery:

![codename-misdirection](https://user-images.githubusercontent.com/47430598/80763162-f0424f00-8b0b-11ea-95f1-f3e26bf35679.png)

## Comments

We should probably replace "below" with "above" in the message. Or move the codename widget. Something. I don't know, it's hard.



Discussion:

Resolving patch:
diff --git a/securedrop/source_app/main.py b/securedrop/source_app/main.py
--- a/securedrop/source_app/main.py
+++ b/securedrop/source_app/main.py
@@ -61,7 +61,7 @@ def select_logo():
     @view.route('/create', methods=['POST'])
     def create():
         if session.get('logged_in', False):
-            flash(gettext("You are already logged in. Please verify your codename below as it " +
+            flash(gettext("You are already logged in. Please verify your codename above as it " +
                           "may differ from the one displayed on the previous page."),
                   'notification')
         else:


Test patch:
Repository: freedomofpress/securedrop

Issue:
qa_loader.py uses a fixed random seed every run
## Description

Always using the same seed makes it impossible to run `qa_loader.py` multiple times with the same database, as supposedly random values aren't, causing unique constraint violations.

## Steps to Reproduce

- Run the dev server with `make dev`
- Start a shell in the container with `docker exec -it securedrop-dev-0 bash`
- In that shell, run `./qa_loader --journalist-count 1 --source-count 1`
- Run the same command a second time.

## Expected Behavior

That you could keep adding random journalists and sources to the database.

## Actual Behavior

You get `sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) UNIQUE constraint failed: journalists.username` because [`random.seed` is always called with the same value](https://github.com/freedomofpress/securedrop/blob/ec2220c3c2b9120d029b616d3a07647b175bc6ab/securedrop/qa_loader.py#L22).


Discussion:

Resolving patch:
diff --git a/securedrop/qa_loader.py b/securedrop/qa_loader.py
--- a/securedrop/qa_loader.py
+++ b/securedrop/qa_loader.py
@@ -19,9 +19,6 @@
 from sdconfig import config as sdconfig
 
 
-random.seed("~(=^–^)")  # mrow?
-
-
 def random_bool():
     return bool(random.getrandbits(1))
 


Test patch:
Repository: freedomofpress/securedrop

Issue:
Upgrade sphinx for develop requirements
## Description

This is caught by `safety` tool.

```
Checking file ./securedrop/requirements/python3/develop-requirements.txt
safety report
checked 123 packages, using default DB
---
-> sphinx, installed 1.6.3, affected <3.0.4, id 38330
Sphinx 3.0.4 updates jQuery version from 3.4.1 to 3.5.1 for security reasons.
--
Makefile:139: recipe for target 'safety' failed
make: *** [safety] Error 1
make: Leaving directory '/home/circleci/project'
```



Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -49,7 +49,7 @@
 
 # General information about the project.
 project = u'SecureDrop'
-copyright = u'2015-2019, Freedom of the Press Foundation'
+copyright = u'2015-2020, Freedom of the Press Foundation'
 author = u'SecureDrop Team and Contributors'
 
 # The version info for the project you're documenting, acts as replacement for


Test patch:
Repository: freedomofpress/securedrop

Issue:
doc-linkcheck needs some appeasement
## Description

We have a [CI failure because of a link to a private repo](https://app.circleci.com/jobs/github/freedomofpress/securedrop/42146). 

## Steps to Reproduce

Run `make docs-linkcheck`.

## Expected Behavior

That it would complete with no error.

## Actual Behavior

The link to the private repo causes a 404. 

## Comments

That private URL should be added to `linkcheck_ignore` in `docs/conf.py`. 



Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -309,4 +309,5 @@
   r'http://localhost(:\d+)?/?',
   'https://forum.securedrop.org/admin/users/list/active',
   'https://weblate.securedrop.org/projects/securedrop/securedrop/#repository',
+  'https://github.com/freedomofpress/securedrop-debian-packages-lfs',
 ]


Test patch: