internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: freedomofpress/securedrop

Issue:
If a source's store directory is missing, subsequent submission attempts will fail with a FileNotFoundError exception
## Description

As per title.

## Steps to Reproduce

(in dev)
- log in to the SI and submit.
- in a shell on the dev docker container (`docker exec -it securedrop-dev-0 bash`) find and delete the source's directory under `/var/lib/securedrop/store/` 
- submit from the SI again as the same source 

## Expected Behavior
Either:
- submission succeeds and directory is recreated,
Or:
- Submission fails with an informative error

## Actual Behavior
uncaught FileNotFoundError and a stack trace (in dev)



Suggestions to fix, any other relevant information.



Discussion:
> submission succeeds and directory is recreated,
> Or:
> Submission fails with an informative error

All things considered, I would suggest going with the first option: recreate the directory.

1) This is not a state that should arise during normal operation, but
2) Admins may accidentally or intentionally wipe the `store` directory to clean out files and messages;
3) This approximates a "file and messages" deletion operation through the UI;
4) Locking out individual sources or large numbers of them can cause significant opsec issues in its own right as they recreate accounts and passphrases.

That said, it may be good to at least log an error or OSSEC alert in this case. Thoughts from others?
@eloquence Sounds good. Can I take this up?
@zenmonkeykstop @rmol @kushaldas May I take this up?
@prateekj117 It would be great if you could take a first stab at this, for sure!

Resolving patch:
diff --git a/securedrop/source_app/main.py b/securedrop/source_app/main.py
--- a/securedrop/source_app/main.py
+++ b/securedrop/source_app/main.py
@@ -188,6 +188,11 @@ def submit() -> werkzeug.Response:
         journalist_filename = g.source.journalist_filename
         first_submission = g.source.interaction_count == 0
 
+        if not os.path.exists(current_app.storage.path(g.filesystem_id)):
+            current_app.logger.debug("Store directory not found for source '{}', creating one."
+                                     .format(g.source.journalist_designation))
+            os.mkdir(current_app.storage.path(g.filesystem_id))
+
         if msg:
             g.source.interaction_count += 1
             fnames.append(


Test patch:
Repository: freedomofpress/securedrop

Issue:
Alembic operations fail with multiple head revisions
## Description

All Alembic operations fail with Alembic error:

    ERROR [alembic.util.messaging] Multiple head revisions are present for given argument 'head'; please specify a specific target revision, '<branchname>@head' to narrow to a specific head, or 'heads' for all heads

Cf. consistent recent failures of CI jobs `app-tests` and `staging-test-with-rebase` since #5974.

## Steps to Reproduce

`make test` on `develop`; open or push to a PR; etc.

## Expected Behavior

Alembic operations succeed and Alembic-based tests pass.

## Actual Behavior

All Alembic operations and tests fail with Alembic error:

    ERROR [alembic.util.messaging] Multiple head revisions are present for given argument 'head'; please specify a specific target revision, '<branchname>@head' to narrow to a specific head, or 'heads' for all heads

## Comments

This is essentially an Alembic-level merge-conflict.  PR forthcoming with the one-line fix.


Discussion:

Resolving patch:
diff --git a/securedrop/alembic/versions/1ddb81fb88c2_unique_index_for_instanceconfig_valid_.py b/securedrop/alembic/versions/1ddb81fb88c2_unique_index_for_instanceconfig_valid_.py
--- a/securedrop/alembic/versions/1ddb81fb88c2_unique_index_for_instanceconfig_valid_.py
+++ b/securedrop/alembic/versions/1ddb81fb88c2_unique_index_for_instanceconfig_valid_.py
@@ -11,7 +11,7 @@
 
 # revision identifiers, used by Alembic.
 revision = '1ddb81fb88c2'
-down_revision = '92fba0be98e9'
+down_revision = 'b060f38c0c31'
 branch_labels = None
 depends_on = None
 


Test patch:
Repository: freedomofpress/securedrop

Issue:
[QA] "securedrop-admin verify" does not support force arg
## Description
Running `./securedrop-admin verify` on `release/2.1.0` fails, because there's no prod-signed tag on that commit. 

## Steps to Reproduce
1. Check out `release/2.1.0` in Tails, configured VMs
2. Run `./securedrop-admin verify`, observe error message and friendly recommendation to try the "--force" flag
3. Run `./securedrop-admin --force verify` as recommended, observe error message about flag not being understood

## Expected Behavior

`./securedrop-admin --force verify` skips tag verification logic, just like `./securedrop-admin --force install` does.

## Actual Behavior

`./securedrop-admin --force verify` does not recognize the `--force` flag at all.


Discussion:
IMO `verify` shouldn't do the version check at all - it doesn't run any playbook that can change the workstation or server state, so it doesn't matter which version you're on locally (altho the onus is then on the user to make sure they're using the version they expect).

Resolving patch:
diff --git a/admin/securedrop_admin/__init__.py b/admin/securedrop_admin/__init__.py
--- a/admin/securedrop_admin/__init__.py
+++ b/admin/securedrop_admin/__init__.py
@@ -776,7 +776,6 @@ def install_securedrop(args: argparse.Namespace) -> int:
     )
 
 
-@update_check_required("verify")
 def verify_install(args: argparse.Namespace) -> int:
     """Run configuration tests against SecureDrop servers"""
 


Test patch:
Repository: freedomofpress/securedrop

Issue:
add `robots.txt` and a `nofollow` robots meta tag to dissuade legitimate crawlers from crawling tor2webb-ed instances



Discussion:

Resolving patch:
diff --git a/securedrop/source_app/main.py b/securedrop/source_app/main.py
--- a/securedrop/source_app/main.py
+++ b/securedrop/source_app/main.py
@@ -8,7 +8,7 @@
 
 import werkzeug
 from flask import (Blueprint, render_template, flash, redirect, url_for,
-                   session, current_app, request, Markup, abort, g)
+                   session, current_app, request, Markup, abort, g, make_response)
 from flask_babel import gettext
 
 import store
@@ -322,4 +322,11 @@ def logout() -> Union[str, werkzeug.Response]:
         else:
             return redirect(url_for('.index'))
 
+    @view.route('/robots.txt')
+    def robots_txt() -> werkzeug.Response:
+        """Tell robots we don't want them"""
+        resp = make_response("User-agent: *\nDisallow: /")
+        resp.headers["content-type"] = "text/plain"
+        return resp
+
     return view


Test patch:
diff --git a/securedrop/tests/test_source.py b/securedrop/tests/test_source.py
--- a/securedrop/tests/test_source.py
+++ b/securedrop/tests/test_source.py
@@ -869,3 +869,13 @@ def test_source_can_only_delete_own_replies(source_app, app_storage):
 
     reply = Reply.query.filter_by(filename=filename).one()
     assert reply.deleted_by_source
+
+
+def test_robots_txt(source_app):
+    """Test that robots.txt works"""
+    with source_app.test_client() as app:
+        # Not using url_for here because we care about the actual URL path
+        resp = app.get('/robots.txt')
+        assert resp.status_code == 200
+        text = resp.data.decode('utf-8')
+        assert 'Disallow: /' in text
Repository: freedomofpress/securedrop

Issue:
SI: focusing on #flashed message after submitting sometimes scrolls the viewport too far, may obscure message
## Description

First identified by @eloquence in #6306, the message we want to draw attention to may get scrolled out of view when we use `#flashed` as the anchor so screen-readers would know where to put their attention to first.

## Steps to Reproduce

* Visit index with Tor Browser, JavaScript disabled
* Inspect the `html` element with the **Web Developer Tools**
* Using the layout tab, make sure that the content size of the `html` element is 1000x540 (when no scrollbar is visible)
* Add a new submission, see most of the _Success!_ flashed message disappear

## Expected Behavior

Flashed message stays in view

## Actual Behavior

![flash](https://user-images.githubusercontent.com/213636/157574307-f2edd0bb-8234-49dd-8bd0-bb950c30962a.gif)

_Originally posted by @eloquence in https://github.com/freedomofpress/securedrop/issues/6306#issuecomment-1063580054_

## Comments

@cfm highlighted that this is probably due to changes from #6041, and IMO we don't want to leave screen-reader users to figure out themselves what's going on. The page 53 of [WCAG 2.1 Audit](https://docs.google.com/document/d/1jZlKco2YgOrrI4PmTkG_Q8lKuV5VxCds/edit) already describes an option that we can probably handle in time for 2.3.0:

We already have a _Skip to main content_ link which we could either combine with or add to with another link specifically to jump to `#flashed` if `get_flashed_messages()` isn't empty. 




Discussion:

Resolving patch:
diff --git a/securedrop/source_app/main.py b/securedrop/source_app/main.py
--- a/securedrop/source_app/main.py
+++ b/securedrop/source_app/main.py
@@ -171,7 +171,7 @@ def submit(logged_in_source: SourceUser) -> werkzeug.Response:
             for field, errors in form.errors.items():
                 for error in errors:
                     flash(error, "error")
-            return redirect(f"{url_for('main.lookup')}#flashed")
+            return redirect(url_for('main.lookup'))
 
         msg = request.form['msg']
         fh = None
@@ -186,7 +186,7 @@ def submit(logged_in_source: SourceUser) -> werkzeug.Response:
                       "error")
             else:
                 flash(gettext("You must enter a message."), "error")
-            return redirect(f"{url_for('main.lookup')}#flashed")
+            return redirect(url_for('main.lookup'))
 
         fnames = []
         logged_in_source_in_db = logged_in_source.get_db_record()
@@ -252,7 +252,7 @@ def submit(logged_in_source: SourceUser) -> werkzeug.Response:
 
         normalize_timestamps(logged_in_source)
 
-        return redirect(f"{url_for('main.lookup')}#flashed")
+        return redirect(url_for('main.lookup'))
 
     @view.route('/delete', methods=('POST',))
     @login_required


Test patch:
Repository: freedomofpress/securedrop

Issue:
Test securedrop-admin with Tails 5.0
## Description

https://tails.boum.org/news/test_5.0-beta1/

Tails 5.0 is based on Debian Bullseye, which means it's using a newer Python version (3.9) among plenty of other things.

It's probably worth walking through a full SD install + backup/restore to make sure it works as expected.


Discussion:
In-progress comment for notes as I go through a (virtualized) from-scratch setup on 5.0:
- :warning: menubar layout has changed (**Activities** is first option, screenshots and docs will need revising)
- :warning: initial ./securedrop-admin setup failed because the torified  torproject mirror was not available - could be a flake
- :x:  was a flake but looks like the `ccontrol` package has been dropped from Bullseye, so the initial installation of apt deps is gonna fail. Updating admin/bootstrap.py to remove it and retrying.
- :heavy_check_mark: with ccontrol gone, got as far as virtualenv install and failed a few times, getting name resolution failures for files.pythonhosted.org - finally venv setup succeeded 
- :x: `./securedrop-admin sdconfig` and `./securedrop-admin install` completed successfully, but `tailsconfig` fails as there is no `python` executable in Tails5, just `python3`. Updating the ansible task accordingly... and it works.
- :heavy_check_mark: GUI updater updates from `develop` to 2.3.1 with no issues, with the changes above.
- :heavy_check_mark: on an local-ssh install, `ssh app` and `ssh mon` work correctly
- :heavy_check_mark: Source and Journalist Interfaces are available via shortcuts
- :heavy_check_mark: backup and restore work without errors
-  :heavy_check_mark: testinfra deps installed without issue and testinfra tests run happily
Looks like ccontrol is a vestigial install left over from the days of Tails 3, removing it doesn't appear to have any effect on setup in Tails 4.29. Also, changing `python` to `python3` in in the tailsconfig task seems to work fine too. So with those changes basic functionality should be the same in 4 and 5, leaving aside some look-and-feel differences.

Resolving patch:
diff --git a/admin/bootstrap.py b/admin/bootstrap.py
--- a/admin/bootstrap.py
+++ b/admin/bootstrap.py
@@ -138,7 +138,6 @@ def install_apt_dependencies(args: argparse.Namespace) -> None:
                    python3-virtualenv \
                    python3-yaml \
                    python3-pip \
-                   ccontrol \
                    virtualenv \
                    libffi-dev \
                    libssl-dev \


Test patch:
Repository: freedomofpress/securedrop

Issue:
`map_locale_display_names()` does not set `use_display_name` for the first of multiple locales for a language
## Description

For languages like `pt` and `zh` for which we now have multiple translated locales, `map_locale_display_names()` sets `use_display_name` for only the second and following locales.  Prior to an overzealous refactoring in #6406, it would be set for all locales for a multi-locale language.

## Steps to Reproduce

Reported by @deeplow in <https://forum.securedrop.org/t/can-pt-br-become-portugues-brasil-instead-of-just-portugues/1455>.  *Test cases TK.*

## Expected Behavior

* `pt_BR` = `Português (Brasil)`
* `pt_PT` = `Português (Portugal)`

## Actual Behavior

* `pt_BR` = `Português`
* `pt_PT` = `Português (Portugal)`



Discussion:

Resolving patch:
diff --git a/securedrop/i18n.py b/securedrop/i18n.py
--- a/securedrop/i18n.py
+++ b/securedrop/i18n.py
@@ -16,7 +16,7 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 import collections
-from typing import List, Set
+from typing import Dict, List, Set
 
 from babel.core import (
     Locale,
@@ -175,18 +175,21 @@ def map_locale_display_names(config: SDConfig) -> None:
     to distinguish them. For languages with more than one translation,
     like Chinese, we do need the additional detail.
     """
-    seen: Set[str] = set()
+
+    language_locale_counts = collections.defaultdict(int)  # type: Dict[str, int]
+    for l in sorted(config.SUPPORTED_LOCALES):
+        locale = RequestLocaleInfo(l)
+        language_locale_counts[locale.language] += 1
+
     locale_map = collections.OrderedDict()
     for l in sorted(config.SUPPORTED_LOCALES):
         if Locale.parse(l) not in USABLE_LOCALES:
             continue
 
         locale = RequestLocaleInfo(l)
-        if locale.language in seen:
+        if language_locale_counts[locale.language] > 1:
             # Disambiguate translations for this language.
             locale.use_display_name = True
-        else:
-            seen.add(locale.language)
 
         locale_map[str(locale)] = locale
 


Test patch:
diff --git a/securedrop/tests/test_i18n.py b/securedrop/tests/test_i18n.py
--- a/securedrop/tests/test_i18n.py
+++ b/securedrop/tests/test_i18n.py
@@ -410,3 +410,17 @@ def test_html_attributes(journalist_app, config):
     )
     html = resp.data.decode("utf-8")
     assert '<html lang="en-US" dir="ltr">' in html
+
+
+def test_same_lang_diff_locale(journalist_app, config):
+    """
+    Verify that when two locales with the same lang are specified, the full locale
+    name is used for both.
+    """
+    del journalist_app
+    config.SUPPORTED_LOCALES = ["en_US", "pt_BR", "pt_PT"]
+    app = journalist_app_module.create_app(config).test_client()
+    resp = app.get("/", follow_redirects=True)
+    html = resp.data.decode("utf-8")
+    assert "português (Brasil)" in html
+    assert "português (Portugal)" in html
Repository: freedomofpress/securedrop

Issue:
Initial messages containing non-ascii characters fail if codename filtering is enabled.
## Description

Codename filtering was introduced in 2.3.0, allowing admins to block initial submissions containing only the user's codename, as they should not be shared with journalists. The filter uses the `compare_digest()` function to ensure constant-time comparison, but this fn will throw a `TypeError` if any of the strings being compared  contain Unicode.

## Steps to Reproduce

- start up `make dev` on 2.4.0
- visit the JI and enable codename filtering under Admin > Instance Config
- visit the SI, create a new source, and submit an initial message containing unicode, ie `Hallo! ö, ü, ä, or ß`

## Expected Behavior
- Message is submitted

## Actual Behavior
- 500 error, and (in dev) stack trace due to TypeError

## Comments

Suggestions to fix, any other relevant information.



Discussion:

Resolving patch:
diff --git a/securedrop/source_app/utils.py b/securedrop/source_app/utils.py
--- a/securedrop/source_app/utils.py
+++ b/securedrop/source_app/utils.py
@@ -23,7 +23,7 @@ def codename_detected(message: str, codename: str) -> bool:
     """
     message = message.strip()
 
-    return compare_digest(message.strip(), codename)
+    return compare_digest(message.strip().encode("utf-8"), codename.encode("utf-8"))
 
 
 def flash_msg(


Test patch:
diff --git a/securedrop/tests/test_source.py b/securedrop/tests/test_source.py
--- a/securedrop/tests/test_source.py
+++ b/securedrop/tests/test_source.py
@@ -356,7 +356,10 @@ def _dummy_submission(app):
     """
     return app.post(
         url_for("main.submit"),
-        data=dict(msg="Pay no attention to the man behind the curtain.", fh=(BytesIO(b""), "")),
+        data=dict(
+            msg="Hallo! ö, ü, ä, or ß...Pay no attention to the man behind the curtain.",
+            fh=(BytesIO(b""), ""),
+        ),
         follow_redirects=True,
     )
Repository: freedomofpress/securedrop

Issue:
Organization name gets escaped twice
## Description

When configuring an instance's name as (for example) "Hello & World", the `&` shows up as `&amp;` wherever the organization name is used, because it is escaped when it is being set and again when it is being displayed. 

## Comments

As far as I can tell, the organization name is the only user provided input that is escaped before it is set - oversight or intentional?


Discussion:
Probably more overzealousness!

Resolving patch:
diff --git a/securedrop/journalist_app/admin.py b/securedrop/journalist_app/admin.py
--- a/securedrop/journalist_app/admin.py
+++ b/securedrop/journalist_app/admin.py
@@ -2,7 +2,6 @@
 
 import binascii
 import os
-from html import escape
 from typing import Optional, Union
 
 import werkzeug
@@ -132,7 +131,7 @@ def update_org_name() -> Union[str, werkzeug.Response]:
         if form.validate_on_submit():
             try:
                 value = request.form["organization_name"]
-                InstanceConfig.set_organization_name(escape(value, quote=True))
+                InstanceConfig.set_organization_name(value)
                 flash(gettext("Preferences saved."), "org-name-success")
             except Exception:
                 flash(gettext("Failed to update organization name."), "org-name-error")


Test patch:
diff --git a/securedrop/tests/test_journalist.py b/securedrop/tests/test_journalist.py
--- a/securedrop/tests/test_journalist.py
+++ b/securedrop/tests/test_journalist.py
@@ -7,7 +7,6 @@
 import random
 import zipfile
 from base64 import b64decode
-from html import escape as htmlescape
 from io import BytesIO
 from pathlib import Path
 
@@ -2003,25 +2002,6 @@ def test_orgname_oversized_fails(config, journalist_app, test_admin, locale):
         assert InstanceConfig.get_current().organization_name == "SecureDrop"
 
 
-@flaky(rerun_filter=utils.flaky_filter_xfail)
-@pytest.mark.parametrize("locale", get_test_locales())
-def test_orgname_html_escaped(config, journalist_app, test_admin, locale):
-    t_name = '"> <a href=foo>'
-    with journalist_app.test_client() as app:
-        _login_user(app, test_admin["username"], test_admin["password"], test_admin["otp_secret"])
-        form = journalist_app_module.forms.OrgNameForm(organization_name=t_name)
-        assert InstanceConfig.get_current().organization_name == "SecureDrop"
-        with InstrumentedApp(journalist_app) as ins:
-            resp = app.post(
-                url_for("admin.update_org_name", l=locale), data=form.data, follow_redirects=True
-            )
-            assert page_language(resp.data) == language_tag(locale)
-            msgids = ["Preferences saved."]
-            with xfail_untranslated_messages(config, locale, msgids):
-                ins.assert_message_flashed(gettext(msgids[0]), "org-name-success")
-            assert InstanceConfig.get_current().organization_name == htmlescape(t_name, quote=True)
-
-
 def test_logo_default_available(journalist_app):
     # if the custom image is available, this test will fail
     custom_image_location = os.path.join(config.SECUREDROP_ROOT, "static/i/custom_logo.png")
Repository: freedomofpress/securedrop

Issue:
Clean up outdated references to Python 3.5
*This is a good first issue for new contributors to take on, if you have any questions, please ask on the task or in our [Gitter room](https://gitter.im/freedomofpress/securedrop)!*

## Description

SecureDrop now runs on focal, which uses Python 3.8. But there are still references to Python 3.5 that need to be cleaned up. Some should be dropped outright, others should be switched to 3.8.


Some examples:
```
$ rg python3\\.5
install_files/securedrop-grsec-focal/opt/securedrop/paxctld.conf
98:/usr/bin/python3.5		E

molecule/testinfra/vars/app-qubes-staging.yml
13:securedrop_venv_site_packages: "{{ securedrop_venv }}/lib/python3.5/site-packages"

molecule/testinfra/vars/prodVM.yml
12:securedrop_venv_site_packages: "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages"

install_files/ansible-base/roles/build-securedrop-app-code-deb-pkg/files/usr.sbin.apache2
71:  /etc/python3.5/sitecustomize.py r,
109:  /usr/local/lib/python3.5/dist-packages/ r,
117:  /opt/venvs/securedrop-app-code/lib/python3.5/ r,
118:  /opt/venvs/securedrop-app-code/lib/python3.5/** rm,

securedrop/scripts/rqrequeue
9:sys.path.insert(0, "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages")  # noqa: E402

securedrop/scripts/shredder
14:    0, "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages"

securedrop/scripts/source_deleter
14:    0, "/opt/venvs/securedrop-app-code/lib/python3.5/site-packages"
$ rg 3\\.5 --type=py
molecule/builder-focal/tests/test_build_dependencies.py
6:SECUREDROP_PYTHON_VERSION = os.environ.get("SECUREDROP_PYTHON_VERSION", "3.5")

setup.py
14:    python_requires=">=3.5",
```


Discussion:
Hey there can you assign me for this issue 
@donheshanthaka Go for it!

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -11,7 +11,7 @@
     long_description=long_description,
     long_description_content_type="text/markdown",
     license="AGPLv3+",
-    python_requires=">=3.5",
+    python_requires=">=3.8",
     url="https://github.com/freedomofpress/securedrop",
     classifiers=(
         "Development Status :: 5 - Stable",


Test patch:
diff --git a/molecule/builder-focal/tests/test_build_dependencies.py b/molecule/builder-focal/tests/test_build_dependencies.py
--- a/molecule/builder-focal/tests/test_build_dependencies.py
+++ b/molecule/builder-focal/tests/test_build_dependencies.py
@@ -3,7 +3,7 @@
 import pytest
 
 SECUREDROP_TARGET_DISTRIBUTION = os.environ.get("SECUREDROP_TARGET_DISTRIBUTION")
-SECUREDROP_PYTHON_VERSION = os.environ.get("SECUREDROP_PYTHON_VERSION", "3.5")
+SECUREDROP_PYTHON_VERSION = os.environ.get("SECUREDROP_PYTHON_VERSION", "3.8")
 DH_VIRTUALENV_VERSION = "1.2.2"
 
 testinfra_hosts = ["docker://{}-sd-app".format(SECUREDROP_TARGET_DISTRIBUTION)]