internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: freedomofpress/securedrop

Issue:
[functional testing] create-dev-data.py should get one user for stage based testing
## Description

`create-dev-data.py` should get a `--staging` flag to create only one user which can be used
by Tor based functional tests.

Part of #3488

## User Stories

I am a SecureDrop developer and I want an easy way to add one user for functional tests in my staging or production instance. 


Discussion:

Resolving patch:
diff --git a/securedrop/create-dev-data.py b/securedrop/create-dev-data.py
--- a/securedrop/create-dev-data.py
+++ b/securedrop/create-dev-data.py
@@ -3,6 +3,8 @@
 
 import datetime
 import os
+import sys
+import argparse
 from sqlalchemy.exc import IntegrityError
 
 os.environ["SECUREDROP_ENV"] = "dev"  # noqa
@@ -74,10 +76,19 @@ def create_source_and_submissions(num_submissions=2):
     test_password = "correct horse battery staple profanity oil chewy"
     test_otp_secret = "JHCOGO7VCER3EJ4L"
 
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--staging", help="Adding user for staging tests.",
+                    action="store_true")
+    args = parser.parse_args()
     add_test_user("journalist",
                   test_password,
                   test_otp_secret,
                   is_admin=True)
+
+    # If staging, we only need the journalist user (admin)
+    if args.staging:
+        sys.exit(0)
+
     add_test_user("dellsberg",
                   test_password,
                   test_otp_secret,


Test patch:
diff --git a/securedrop/tests/functional/README.md b/securedrop/tests/functional/README.md
--- a/securedrop/tests/functional/README.md
+++ b/securedrop/tests/functional/README.md
@@ -3,7 +3,7 @@
 - `sudo -u www-data bash`
 - `cd /var/wwww/securedrop/`
 - `./manage.py reset`    # This will clean the DB for testing
-- `./create-demo-user.py`  
+- `./create-dev-data.py --staging`
 
 Update this information to the `tests/functional/instance_information.json file.
Repository: freedomofpress/securedrop

Issue:
Error 500 when accessing journalist interface in dev container
## Description

## Steps to Reproduce

1. checkout develop
2. make dev
3. log into the journalist interface
4. observe error 500

## Expected Behavior

Upon successful login, the list of sources in the Journalist interface should be displayed.

## Actual Behavior

An error 500 occurs: 
```
  File "/home/user/src/securedrop/securedrop/journalist_templates/_source_row.html", line 4, in top-level template code
    <time class="date" title="{{ source.last_updated|rel_datetime_format }}" datetime="{{ source.last_updated|rel_datetime_format(fmt="%Y-%m-%d %H:%M:%S%Z") }}">{{ source.last_updated|rel_datetime_format(relative=True) }}</time>
TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'
```
## Comments
source.last_updated must be manually set in `create-dev-data.py` to avoid this exception, due to changes in the model where the `last_updated` is not populated upon source creation.



Discussion:

Resolving patch:
diff --git a/securedrop/create-dev-data.py b/securedrop/create-dev-data.py
--- a/securedrop/create-dev-data.py
+++ b/securedrop/create-dev-data.py
@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
+import datetime
 import os
 from sqlalchemy.exc import IntegrityError
 
@@ -58,6 +59,7 @@ def create_source_and_submissions(num_submissions=2):
                 source.journalist_filename,
                 'test submission!'
             )
+            source.last_updated = datetime.datetime.utcnow()
             submission = Submission(source, fpath)
             db.session.add(submission)
 


Test patch:
Repository: freedomofpress/securedrop

Issue:
[functional testing] Fix staging CI job on tbb-0.9.0
We removed the application/functional test run from the staging environment in #3697. We should also update the testinfra test references and remove the application test run from CI, otherwise we get a few testinfra test failures due to pip deps, and an error when we attempt to run the application tests in CI: 

```
TASK [Run application tests] ***************************************************
    Friday 10 August 2018  19:28:17 +0000 (0:00:00.037)       0:01:08.223 *********
    fatal: [app-staging]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 127, "stderr": "Shared connection to 52.36.194.59 closed.\r\n", "stdout": "/home/sdrop/.ansible/tmp/ansible-tmp-1533929297.62-93522333058246/app-tests.sh: line 13: pytest: command not found\r\n", "stdout_lines": ["/home/sdrop/.ansible/tmp/ansible-tmp-1533929297.62-93522333058246/app-tests.sh: line 13: pytest: command not found"]}
    ...ignoring
```


Discussion:

Resolving patch:
diff --git a/securedrop/create-dev-data.py b/securedrop/create-dev-data.py
--- a/securedrop/create-dev-data.py
+++ b/securedrop/create-dev-data.py
@@ -78,7 +78,7 @@ def create_source_and_submissions(num_submissions=2):
 
     parser = argparse.ArgumentParser()
     parser.add_argument("--staging", help="Adding user for staging tests.",
-                    action="store_true")
+                        action="store_true")
     args = parser.parse_args()
     add_test_user("journalist",
                   test_password,


Test patch:
diff --git a/securedrop/tests/pages-layout/functional_test.py b/securedrop/tests/pages-layout/functional_test.py
--- a/securedrop/tests/pages-layout/functional_test.py
+++ b/securedrop/tests/pages-layout/functional_test.py
@@ -44,7 +44,8 @@ def webdriver_fixture(self, request):
         if not os.path.exists(self.log_dir):
             os.makedirs(self.log_dir)
         self.new_profile = webdriver.FirefoxProfile()
-        self.new_profile.set_preference("intl.accept_languages", self.accept_languages)
+        self.new_profile.set_preference("intl.accept_languages",
+                                        self.accept_languages)
         self.override_driver = True
         yield None
Repository: freedomofpress/securedrop

Issue:
SecureDrop backups from previous versions don't work if database migration has occurred
## Description

Backup restore (https://github.com/freedomofpress/securedrop/blob/develop/install_files/ansible-base/roles/restore/files/restore.py) script does not apply database migrations, and as such breaks the application upon backup restore. Manual workaround that appears to have no side-effects is to run `sudo dpkg-reconfigure securedrop-app-code` on the app server.

## Steps to Reproduce

1. Install SecureDrop 0.8.0
2. `securedrop-admin backup`
3. Upgrade to 0.9.0 (or higher)
4. `securedrop-admin restore`
5. Observe source and journalist interface return error 500

## Expected Behavior

The application should be operational.

## Actual Behavior

The source and journalist interfaces return error 500s.

## Comments

Running `sudo dpkg-reconfigure securedrop-app-code` calls the postinst script which will apply migration. Based on my testing, this seems to work reliably.


Discussion:
Using `dpkg-reconfigure` is the best choice because it means we keep the upgrade/migration logic in one place.

Resolving patch:
diff --git a/install_files/ansible-base/roles/restore/files/restore.py b/install_files/ansible-base/roles/restore/files/restore.py
--- a/install_files/ansible-base/roles/restore/files/restore.py
+++ b/install_files/ansible-base/roles/restore/files/restore.py
@@ -45,6 +45,8 @@ def main():
     # If the process exits with a non-zero return code, raises an exception.
     subprocess.check_call(['service', 'apache2', 'restart'])
     subprocess.check_call(['service', 'tor', 'reload'])
+    # Apply database migrations (if backed-up version < version to restore)
+    subprocess.check_call(['dpkg-reconfigure', 'securedrop-app-code'])
 
 
 if __name__ == "__main__":


Test patch:
Repository: freedomofpress/securedrop

Issue:
Builder image needs updates
## Description

sd-builder image needs to be updated.

## Steps to Reproduce

`make build-debs` and observe error.

## Expected Behavior

`make build-debs` should exit without error.

## Actual Behavior

`make-build-debs` returns an error, security updates are needed for the container.
## Comments
Instructions are available here:
https://docs.securedrop.org/en/latest/development/dockerbuildmaint.html


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -57,9 +57,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.9.0~rc1'
+version = '0.9.0~rc2'
 # The full version, including alpha/beta/rc tags.
-release = '0.9.0~rc1'
+release = '0.9.0~rc2'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
diff --git a/securedrop/version.py b/securedrop/version.py
--- a/securedrop/version.py
+++ b/securedrop/version.py
@@ -1 +1 @@
-__version__ = '0.9.0~rc1'
+__version__ = '0.9.0~rc2'


Test patch:
diff --git a/molecule/builder/tests/vars.yml b/molecule/builder/tests/vars.yml
--- a/molecule/builder/tests/vars.yml
+++ b/molecule/builder/tests/vars.yml
@@ -1,5 +1,5 @@
 ---
-securedrop_version: "0.9.0~rc1"
+securedrop_version: "0.9.0~rc2"
 ossec_version: "2.8.2"
 keyring_version: "0.1.2"
 config_version: "0.1.1"
Repository: freedomofpress/securedrop

Issue:
0.9.0-rc2 does not allow text only submissions
## Description

I believe I found a bug in upgrading from `0.8` to `0.9rc2` in that sources can only submit documents or documents and messages. If I try to send _only_ text or a blank form field I get a "Bad Request, the browser or proxy sent a request that this server could not understand" error.


## Steps to Reproduce

First I installed 0.8.0 on hardware and create a journalist. I then ran ./qa-loader.py -m 25. I logged in and and see submissions. I then added the apt-test key and updated sources.list to `apt.freedom.press` to `apt-test.freedom.press`. Finally I ran `sudo cron-apt -i -s` to update to `0.9rc2`

## Expected Behavior

A source can send text to journalists.

## Actual Behavior

Error in the webapp.

## Comments

I also enabled apache debug logging and attempted to patch the source_app/ code to log anything related to CSRF violations and I was was not able to trigger a debug log. 



Discussion:
Thanks for this report, I was unable to reproduce this on `rc2`, I will debug with the instance having this issue tomorrow with you if that works @b-meson 
I can confirm the error. I see the same error message.

```
"Bad Request, the browser or proxy sent a request that this server could not understand"
```
Found the cause will update the ticket + a PR after some coffee. 

Resolving patch:
diff --git a/securedrop/source_app/main.py b/securedrop/source_app/main.py
--- a/securedrop/source_app/main.py
+++ b/securedrop/source_app/main.py
@@ -124,7 +124,9 @@ def lookup():
     @login_required
     def submit():
         msg = request.form['msg']
-        fh = request.files['fh']
+        fh = None
+        if 'fh' in request.files:
+            fh = request.files['fh']
 
         # Don't submit anything if it was an "empty" submission. #878
         if not (msg or fh):


Test patch:
Repository: freedomofpress/securedrop

Issue:
Update builder hash
## Status

Ready for review

## Description of Changes

Fixes #3765 , updates securedrop buillder image with latest packages
## Testing

`make build-debs` should complete without error.

## Deployment

Dev only



Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -57,9 +57,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.9.0~rc2'
+version = '0.9.0~rc3'
 # The full version, including alpha/beta/rc tags.
-release = '0.9.0~rc2'
+release = '0.9.0~rc3'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
diff --git a/securedrop/version.py b/securedrop/version.py
--- a/securedrop/version.py
+++ b/securedrop/version.py
@@ -1 +1 @@
-__version__ = '0.9.0~rc2'
+__version__ = '0.9.0~rc3'


Test patch:
diff --git a/molecule/builder/tests/vars.yml b/molecule/builder/tests/vars.yml
--- a/molecule/builder/tests/vars.yml
+++ b/molecule/builder/tests/vars.yml
@@ -1,5 +1,5 @@
 ---
-securedrop_version: "0.9.0~rc2"
+securedrop_version: "0.9.0~rc3"
 ossec_version: "2.8.2"
 keyring_version: "0.1.2"
 config_version: "0.1.1"
Repository: freedomofpress/securedrop

Issue:
Updater GUI returns error when updating to 0.9.0
## Description

When updating a Tails workstation with the SecureDrop Admin Updater, an error box appears:
```
Error Updating SecureDrop Workstation

WARNING: Signature verification failed. Contact your SecureDrop administrator or securedrop@freedom.press immediately.
```

Note that invoking the functionality by the command line: `./securedrop-admin update` works as expected.

## Steps to Reproduce
On a Tails machine:
- git checkout 0.8.0
- click on update via updater gui (disable/re-enable network to get it to appear again)
- observe error

## Expected Behavior
Updater GUI should update without error.

## Actual Behavior

A signature verification error appears on screen when attempting to update.

## Comments



Discussion:

Resolving patch:
diff --git a/admin/securedrop_admin/__init__.py b/admin/securedrop_admin/__init__.py
--- a/admin/securedrop_admin/__init__.py
+++ b/admin/securedrop_admin/__init__.py
@@ -697,7 +697,8 @@ def update(args):
             try:
                 # We expect this to produce a non-zero exit code, which
                 # will produce a subprocess.CalledProcessError
-                subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+                subprocess.check_output(cmd, stderr=subprocess.STDOUT,
+                                        cwd=args.root)
                 sdlog.info("Signature verification failed.")
                 return 1
             except subprocess.CalledProcessError, e:


Test patch:
Repository: freedomofpress/securedrop

Issue:
Posting a Reply should return a UUID
## Description

The API should not return 204, but should return 200 with an object containing the UUID of the reply. Without doing that, there is no way for a client to sanely sync its local copy of a conversation with the server's copy.


Discussion:

Resolving patch:
diff --git a/securedrop/journalist_app/api.py b/securedrop/journalist_app/api.py
--- a/securedrop/journalist_app/api.py
+++ b/securedrop/journalist_app/api.py
@@ -244,7 +244,8 @@ def all_source_replies(source_uuid):
             db.session.add(reply)
             db.session.add(source)
             db.session.commit()
-            return jsonify({'message': 'Your reply has been stored'}), 201
+            return jsonify({'message': 'Your reply has been stored',
+                            'uuid': reply.uuid}), 201
 
     @api.route('/sources/<source_uuid>/replies/<reply_uuid>',
                methods=['GET', 'DELETE'])


Test patch:
diff --git a/securedrop/tests/test_journalist_api.py b/securedrop/tests/test_journalist_api.py
--- a/securedrop/tests/test_journalist_api.py
+++ b/securedrop/tests/test_journalist_api.py
@@ -4,6 +4,7 @@
 import os
 
 from pyotp import TOTP
+from uuid import UUID
 
 from flask import current_app, url_for
 from itsdangerous import TimedJSONWebSignatureSerializer
@@ -641,10 +642,14 @@ def test_authorized_user_can_add_reply(journalist_app, journalist_api_token,
                             headers=get_api_headers(journalist_api_token))
         assert response.status_code == 201
 
-    with journalist_app.app_context():  # Now verify everything was saved.
-        # Get most recent reply in the database
-        reply = Reply.query.order_by(Reply.id.desc()).first()
+    # ensure the uuid is present and valid
+    reply_uuid = UUID(response.json['uuid'])
+
+    # check that the uuid has a matching db object
+    reply = Reply.query.filter_by(uuid=str(reply_uuid)).one_or_none()
+    assert reply is not None
 
+    with journalist_app.app_context():  # Now verify everything was saved.
         assert reply.journalist_id == test_journo['id']
         assert reply.source_id == source_id
Repository: freedomofpress/securedrop

Issue:
Update i18n docs to account for Weblate / GitLab infrastructure migration
## Description

This must be done prior to updating the strings for the 0.11.0 release:

- Point to [our GitHub translations repo](https://github.com/freedomofpress/securedrop-i18n) in: [i18n_tool.py](https://github.com/freedomofpress/securedrop/blob/develop/securedrop/i18n_tool.py)
- Update soon-to-be-404ing link in the README here: https://github.com/freedomofpress/securedrop#acknowledgments
- Update i18n docs: https://github.com/freedomofpress/securedrop/blob/2455e4e50d83075fb9ce5a6945882410c3612d8d/docs/development/i18n.rst, which still refers to the GitLab workflow and .club URLs


Discussion:

Resolving patch:
diff --git a/securedrop/i18n_tool.py b/securedrop/i18n_tool.py
--- a/securedrop/i18n_tool.py
+++ b/securedrop/i18n_tool.py
@@ -26,7 +26,7 @@ class I18NTool(object):
     #
     # The database of support language, indexed by the language code
     # used by weblate (i.e. whatever shows as CODE in
-    # https://weblate.securedrop.club/projects/securedrop/securedrop/CODE/
+    # https://weblate.securedrop.org/projects/securedrop/securedrop/CODE/
     # is the index of the SUPPORTED_LANGUAGES database below.
     #
     # name: English name of the language to the documentation, not for
@@ -324,7 +324,7 @@ def set_update_from_weblate_parser(self, subps):
             default=root,
             help=('root of the SecureDrop git repository'
                   ' (default {})'.format(root)))
-        url = 'https://lab.securedrop.club/bot/securedrop.git'
+        url = 'https://github.com/freedomofpress/securedrop-i18n'
         parser.add_argument(
             '--url',
             default=url,


Test patch:
Repository: freedomofpress/securedrop

Issue:
Update Ansible to 2.4.5 or later
## Description

Ansible 2.4.2 is used in:
* https://github.com/freedomofpress/securedrop/blob/develop/securedrop/requirements/develop-requirements.txt
* https://github.com/freedomofpress/securedrop/blob/develop/admin/requirements.txt

There is currently a CVE associated to this version of Ansible: CVE-2018-10855:

>Ansible 2.5 prior to 2.5.5, and 2.4 prior to 2.4.5, do not honor the no_log task flag for failed tasks. When the no_log flag has been used to protect sensitive data passed to a task from being logged, and that task does not run successfully, Ansible will expose sensitive data in log files and on the terminal of the user running Ansible.

Securedrop does not use `no_log`, and as such this vulnerability does not directly affect Securedrop. Updating Ansible will be required for CI to pass safety checks.

## User Stories

As an admin, packages that are updated and do not have CVEs associated to them is good.



Discussion:
Ansible 2.4 is EOL [source](https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html#release-status). Shouldn't it be upgraded to the lowest supported version which is [2.5.9](https://github.com/ansible/ansible/blob/stable-2.5/changelogs/CHANGELOG-v2.5.rst)?
Ansible v2.5.9 is greater than v2.5.5 as mentioned at [CVE-2018-10855](https://www.cvedetails.com/cve/CVE-2018-10855/).
Good flag @ultimatecoder, we should indeed update to at least the lowest supported version
@redshiftzero Thanks!

Resolving patch:
diff --git a/install_files/ansible-base/callback_plugins/ansible_version_check.py b/install_files/ansible-base/callback_plugins/ansible_version_check.py
--- a/install_files/ansible-base/callback_plugins/ansible_version_check.py
+++ b/install_files/ansible-base/callback_plugins/ansible_version_check.py
@@ -21,7 +21,7 @@ class CallbackModule(CallbackBase):
     def __init__(self):
         # Can't use `on_X` because this isn't forwards compatible
         # with Ansible 2.0+
-        required_version = '2.4.2'  # Keep synchronized with requirements files
+        required_version = '2.6.8'  # Keep synchronized with requirements files
         if not ansible.__version__.startswith(required_version):
             print_red_bold(
                 "SecureDrop restriction: only Ansible {version}.*"


Test patch:
Repository: freedomofpress/securedrop

Issue:
Add Greek (el) language as supported language
## Description

Add [Greek](https://weblate.securedrop.org/projects/securedrop/securedrop/el/) language as a supported language.

## User Research Evidence

The language is fully translated and reviewed in the weblate.



Discussion:

Resolving patch:
diff --git a/securedrop/i18n_tool.py b/securedrop/i18n_tool.py
--- a/securedrop/i18n_tool.py
+++ b/securedrop/i18n_tool.py
@@ -38,6 +38,7 @@ class I18NTool(object):
         'de_DE': {'name': 'German', 'desktop': 'de_DE', },
         'es_ES': {'name': 'Spanish', 'desktop': 'es_ES', },
         'fr_FR': {'name': 'French', 'desktop': 'fr', },
+        'el': {'name': 'Greek', 'desktop': 'el', },
         'hi': {'name': 'Hindi', 'desktop': 'hi', },
         'it_IT': {'name': 'Italian', 'desktop': 'it', },
         'nb_NO': {'name': 'Norwegian', 'desktop': 'nb_NO', },


Test patch:
Repository: freedomofpress/securedrop

Issue:
Latest docs specify an RC instead of a Release version
## Description

`update_version.sh` bumps version string in docs even when an RC is created. There might be a small period of time (e.g. during release QA) where that Tag exists (albeit not signed)

## Steps to Reproduce

https://docs.securedrop.org/en/latest/set_up_admin_tails.html?highlight=git%20checkout and observe instructions to check out 0.12.0~rc1 tag

## Expected Behavior

The tag should be the latest release (as of today, 0.11.1)
## Actual Behavior

The tag is 0.12.0~rc1

## Comments




Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -57,9 +57,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.13.0~rc1'
+version = '0.12.1'
 # The full version, including alpha/beta/rc tags.
-release = '0.13.0~rc1'
+release = '0.12.1'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.


Test patch: