internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: freedomofpress/securedrop

Issue:
Codename with a ‘/‘ causes an exception (or ‘\’)
STR:
1. Visit `/login` on the source interface
2. Type "/" in the codename field and submit the form.

In development, you can see from the traceback that this raises a CryptoException. This is intentional - we intentionally check the codenames for dangerous characters and do not process codenames that contain them, since the dangerous characters will be passed to GPG and could lead to a shell injection. In prod, you would get a 500 Internal Server Error.

Instead of erroring out, we should flash a message to the user that says their codename was invalid, and redirect them back to the login page.



Discussion:

Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -277,11 +277,15 @@ def valid_codename(codename):
 def login():
     if request.method == 'POST':
         codename = request.form['codename']
-        if valid_codename(codename):
-            session.update(codename=codename, logged_in=True)
-            return redirect(url_for('lookup', from_login='1'))
+        try:
+            valid = valid_codename(codename)
+        except crypto_util.CryptoException:
+            pass
         else:
-            flash("Sorry, that is not a recognized codename.", "error")
+            if valid:
+                session.update(codename=codename, logged_in=True)
+                return redirect(url_for('lookup', from_login='1'))
+        flash("Sorry, that is not a recognized codename.", "error")
     return render_template('login.html')
 
 


Test patch:
Repository: freedomofpress/securedrop

Issue:
More readable timestamp format for replies in source interface
As a Source, if I receive a reply from a journalist, the timestamp which is included in the reply is formatted strangely (e.g.2014-09-06 21:45:10.916714).

It should be formatted in a clearly readable format, like Sep 06, 2014 09:45 PM.



Discussion:

Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -177,7 +177,7 @@ def lookup():
             except UnicodeDecodeError:
                 app.logger.error("Could not decode reply %s" % fn)
             else:
-                date = datetime.fromtimestamp(os.stat(store.path(g.sid, fn)).st_mtime)
+                date = datetime.fromtimestamp(os.stat(store.path(g.sid, fn)).st_mtime).strftime("%b %d, %Y %I:%M %p")
                 replies.append(dict(id=fn, date=date, msg=msg))
 
     def async_genkey(sid, codename):


Test patch:
Repository: freedomofpress/securedrop

Issue:
Can delete and download submissions without having selected any
On any source page in the journalist interface, the journalist is allowed to "Download selected" and "Delete selected" without having selected any submissions. In the case of downloading, the journalist receives an empty zip file. In the case of deleting, nothing happens, but the journalist receives a flashed "Deleted 0 files" message.

We should instead display an error message if the journalist attempts such a nonsensical action.



Discussion:

Resolving patch:
diff --git a/securedrop/journalist.py b/securedrop/journalist.py
--- a/securedrop/journalist.py
+++ b/securedrop/journalist.py
@@ -186,7 +186,7 @@ def col_delete_single(sid):
 def col_delete(cols_selected):
     """deleting multiple collections from the index"""
     if len(cols_selected) < 1:
-        flash("No collections selected to delete!", "warning")
+        flash("No collections selected to delete!", "error")
     else:
         for source_id in cols_selected:
             delete_collection(source_id)
@@ -249,6 +249,13 @@ def bulk():
     filenames_selected = [
         doc['name'] for doc in docs_selected]
 
+    if not docs_selected:
+        if action == 'download':
+            flash("No collections selected to download!", "error")
+        elif action == 'delete':
+            flash("No collections selected to delete!", "error")
+        return redirect(url_for('col', sid=g.sid))
+
     if action == 'download':
         return bulk_download(g.sid, filenames_selected)
     elif action == 'delete':


Test patch:
Repository: freedomofpress/securedrop

Issue:
Newlines shouldn't be stripped from source replies
When a journalist types a reply to a source, the newslines don't get converted to br tags when the source sees them. This is annoying, but if the reply is PGP-encrypted it makes it much more annoying for the source to decrypt.

Either we can convert newslines to brs, or put the reply in a pre tag, or something like that.

![screenshot from 2014-08-04 22 35 16](https://cloud.githubusercontent.com/assets/156128/3804618/08f54d20-1c29-11e4-85a8-9dff2eea19fd.png)



Discussion:
I am surprised this flew under the radar for so long! How does the document interface handle it?


Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -27,6 +27,7 @@
 import template_filters
 from db import db_session, Source, Submission
 from request_that_secures_file_uploads import RequestThatSecuresFileUploads
+from jinja2 import evalcontextfilter
 
 app = Flask(__name__, template_folder=config.SOURCE_TEMPLATES_DIR)
 app.request_class = RequestThatSecuresFileUploads
@@ -46,6 +47,7 @@
     app.jinja_env.globals['use_custom_header_image'] = False
 
 app.jinja_env.filters['datetimeformat'] = template_filters.datetimeformat
+app.jinja_env.filters['nl2br'] = evalcontextfilter(template_filters.nl2br)
 
 @app.teardown_appcontext
 def shutdown_session(exception=None):
diff --git a/securedrop/template_filters.py b/securedrop/template_filters.py
--- a/securedrop/template_filters.py
+++ b/securedrop/template_filters.py
@@ -1,5 +1,6 @@
 # -*- coding: utf-8 -*-
 from datetime import datetime
+from jinja2 import Markup, escape
 
 
 def datetimeformat(dt, fmt=None, relative=False):
@@ -34,3 +35,10 @@ def _relative_timestamp(dt):
         return '{} days'.format(int(max(diff / 86400, 2)))
     else:
         return None
+
+
+def nl2br(context, value):
+    formatted = u'<br>\n'.join(escape(value).split('\n'))
+    if context.autoescape:
+        formatted = Markup(formatted)
+    return formatted


Test patch:
Repository: freedomofpress/securedrop

Issue:
Database error if a source goes back and resubmits the /generate page
A IntegrityError is thrown by SqlAlchemy if a user goes back to the /generate form and resubmits it. There is an attempt to create another Source entry with a non unqiue filesystem_id/codename. Instead the user should probably just be redirected to their /lookup page



Discussion:

Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -17,6 +17,7 @@
 from flask_wtf.csrf import CsrfProtect
 
 from sqlalchemy.orm.exc import MultipleResultsFound, NoResultFound
+from sqlalchemy.exc import IntegrityError
 
 import config
 import version
@@ -157,9 +158,12 @@ def create():
 
     source = Source(sid, crypto_util.display_id())
     db_session.add(source)
-    db_session.commit()
-
-    os.mkdir(store.path(sid))
+    try:
+        db_session.commit()
+    except IntegrityError as e: 
+        app.logger.error("Attempt to create a source with duplicate codename: %s" % (e,))
+    else:
+        os.mkdir(store.path(sid))
 
     session['logged_in'] = True
     return redirect(url_for('lookup'))


Test patch:
Repository: freedomofpress/securedrop

Issue:
Rename all submissions for a source when their designation is changed
As a journalist, if I generate a new codename for the source, all of the files for the source should also be renamed so that they have the new codename in the file-name. Currently, new submissions will use the new codename, but old submission's filenames will retain the previous codename.



Discussion:

Resolving patch:
diff --git a/securedrop/journalist.py b/securedrop/journalist.py
--- a/securedrop/journalist.py
+++ b/securedrop/journalist.py
@@ -226,8 +226,14 @@ def reply():
 
 @app.route('/regenerate-code', methods=('POST',))
 def generate_code():
+    original_journalist_designation = g.source.journalist_designation
     g.source.journalist_designation = crypto_util.display_id()
+    
+    for doc in Submission.query.filter(Submission.source_id == g.source.id).all():
+        doc.filename = store.rename_submission(g.sid, doc.filename, g.source.journalist_filename())
     db_session.commit()
+
+    flash("The source '%s' has been renamed to '%s'" % (original_journalist_designation, g.source.journalist_designation), "notification")
     return redirect('/col/' + g.sid)
 
 
diff --git a/securedrop/store.py b/securedrop/store.py
--- a/securedrop/store.py
+++ b/securedrop/store.py
@@ -15,7 +15,7 @@
 from werkzeug import secure_filename
 
 VALIDATE_FILENAME = re.compile(
-    "^(reply-)?[a-z0-9-_]+(-msg|-doc\.zip|)\.gpg$").match
+    "^(?P<index>\d+)\-[a-z0-9-_]+?(?P<file_type>msg|doc\.zip|)\.gpg$").match
 
 
 class PathException(Exception):
@@ -96,6 +96,23 @@ def save_message_submission(sid, count, journalist_filename, message):
     return filename
 
 
+def rename_submission(sid, orig_filename, journalist_filename):
+    check_submission_name = VALIDATE_FILENAME(orig_filename)
+    if check_submission_name:
+        parsed_filename = check_submission_name.groupdict()
+        if parsed_filename.get('file_type'):
+            new_filename = "{}-{}-{}.gpg".format(
+                parsed_filename['index'], journalist_filename,
+                parsed_filename['file_type'])
+            try:
+                os.rename(path(sid, orig_filename), path(sid, new_filename))
+            except OSError:
+                pass
+            else:
+                return new_filename  # Only return new filename if successful
+    return orig_filename
+
+
 def secure_unlink(fn, recursive=False):
     verify(fn)
     command = ['srm']


Test patch:
Repository: freedomofpress/securedrop

Issue:
Large files can be downloaded before the file is completely processed
Scenario:

As a source, upload a >250 MB file (something large)
While this is uploading, from the journalist view, reload the page of “files received from the source”
Once the file shows up, immediately click on the filename in order to download it.

Result:

You will download part of the encrypted file, but possibly not all of it. It can take a while for large files to be encrypted, and in the interim you will get confusing partial copies of the encrypted file.

Expected behavior:

We should either get the whole file, or none of the file. (i.e.: it shouldn’t show up if it is not in a state available for downloading).



Discussion:

Resolving patch:
diff --git a/securedrop/journalist.py b/securedrop/journalist.py
--- a/securedrop/journalist.py
+++ b/securedrop/journalist.py
@@ -123,6 +123,11 @@ def index():
 def col(sid):
     source = get_source(sid)
     docs = get_docs(sid)
+    submissions = [submission.filename for submission in Submission.query.filter(Submission.source_id == source.id).all()]
+    # Only include documents loaded from the filesystem which are replies or which are also listed in the
+    # submissions table to avoid displaying partially uploaded files (#561).
+    docs = [doc for doc in docs if doc['name'] in submissions or doc['name'].endswith('reply.gpg')]
+
     haskey = crypto_util.getkey(sid)
     return render_template("col.html", sid=sid,
                            codename=source.journalist_designation, docs=docs, haskey=haskey,


Test patch:
Repository: freedomofpress/securedrop

Issue:
Add folder with the source's name to the .zip
When a journalist downloads a new submission and extracts the archive, she is left with a number of files in the same directory as the .zip file. It would be great if a folder with files was created instead, the name of the folder should be the code name given to the source.



Discussion:

Resolving patch:
diff --git a/securedrop/journalist.py b/securedrop/journalist.py
--- a/securedrop/journalist.py
+++ b/securedrop/journalist.py
@@ -289,9 +289,9 @@ def bulk_download(sid, docs_selected):
         except NoResultFound as e:
             app.logger.error("Could not mark " + doc + " as downloaded: %s" % (e,))
     db_session.commit()
-    zip = store.get_bulk_archive(filenames)
+    zip = store.get_bulk_archive(filenames, zip_directory=source.journalist_filename())
     return send_file(zip.name, mimetype="application/zip",
-                     attachment_filename=source.journalist_designation + ".zip",
+                     attachment_filename=source.journalist_filename() + ".zip",
                      as_attachment=True)
 
 
diff --git a/securedrop/store.py b/securedrop/store.py
--- a/securedrop/store.py
+++ b/securedrop/store.py
@@ -66,12 +66,15 @@ def path(*s):
     return absolute
 
 
-def get_bulk_archive(filenames):
+def get_bulk_archive(filenames, zip_directory=''):
     zip_file = tempfile.NamedTemporaryFile(prefix='tmp_securedrop_bulk_dl_')
     with zipfile.ZipFile(zip_file, 'w') as zip:
         for filename in filenames:
             verify(filename)
-            zip.write(filename, arcname=os.path.basename(filename))
+            zip.write(filename, arcname=os.path.join(
+                zip_directory,
+                os.path.basename(filename)
+            ))
     return zip_file
 
 


Test patch:
diff --git a/securedrop/tests/test_unit.py b/securedrop/tests/test_unit.py
--- a/securedrop/tests/test_unit.py
+++ b/securedrop/tests/test_unit.py
@@ -259,6 +259,9 @@ def test_bulk_download(self):
         self.assertEqual(rv.status_code, 200)
         self.assertEqual(rv.content_type, 'application/zip')
         self.assertTrue(zipfile.is_zipfile(StringIO(rv.data)))
+        self.assertTrue(zipfile.ZipFile(StringIO(rv.data)).getinfo(
+                os.path.join(source.journalist_filename(),files[0])
+            ))
 
 
 class TestIntegration(unittest.TestCase):
Repository: freedomofpress/securedrop

Issue:
Database error when trying to delete replies in the journalist interface
An error is thrown deleting replies in the journalist interface. An attempt is made to remove a record for the reply from the database but replies are only recorded on the filesystem.



Discussion:

Resolving patch:
diff --git a/securedrop/journalist.py b/securedrop/journalist.py
--- a/securedrop/journalist.py
+++ b/securedrop/journalist.py
@@ -276,7 +276,8 @@ def bulk_delete(sid, docs_selected):
     confirm_delete = bool(request.form.get('confirm_delete', False))
     if confirm_delete:
         for doc in docs_selected:
-            db_session.delete(Submission.query.filter(Submission.filename == doc['name']).one())
+            if not doc['name'].endswith('reply.gpg'):
+                db_session.delete(Submission.query.filter(Submission.filename == doc['name']).one())
             fn = store.path(sid, doc['name'])
             store.secure_unlink(fn)
         db_session.commit()


Test patch:
diff --git a/securedrop/tests/test_unit.py b/securedrop/tests/test_unit.py
--- a/securedrop/tests/test_unit.py
+++ b/securedrop/tests/test_unit.py
@@ -484,11 +484,13 @@ def helper_test_reply(self, test_reply, expected_success=True):
         # sending a reply
         _block_on_reply_keypair_gen(codename)
 
-        rv = self.journalist_app.post('/reply', data=dict(
-            sid=sid,
-            msg=test_reply
-        ), follow_redirects=True)
-        self.assertEqual(rv.status_code, 200)
+        # Create 2 replies to test deleting on journalist and source interface
+        for i in range(2): 
+            rv = self.journalist_app.post('/reply', data=dict(
+                sid=sid,
+                msg=test_reply
+            ), follow_redirects=True)
+            self.assertEqual(rv.status_code, 200)
 
         if not expected_success:
             pass
@@ -502,6 +504,11 @@ def helper_test_reply(self, test_reply, expected_success=True):
 
         _block_on_reply_keypair_gen(codename)
 
+        # Test deleting reply on the journalist interface
+        soup = BeautifulSoup(rv.data)
+        last_reply_number = len(soup.select('input[name="doc_names_selected"]')) - 1
+        self.helper_filenames_delete(soup, last_reply_number)
+
         with self.source_app as source_app:
             rv = source_app.post('/login', data=dict(codename=codename), follow_redirects=True)
             self.assertEqual(rv.status_code, 200)
Repository: freedomofpress/securedrop

Issue:
Location of replies unclear for sources
As a source, it it unclear where I should go to see messages from the journalist.

When I choose the “Check for response” and choose “continue” it should clearly show me a “Messages” area of the screen, and say “No replies from the journalist”, or list the messages (if there are any).

We currently flash a message that says "you received a new reply", but this is confusing. It only appears once, so if the user refreshes the page or navigates away and back again, they will not see that message (they will still see the reply, until it is deleted).

Another option would be to make “submit new documents/messages” a separate page from “check for replies from the journalist” page.



Discussion:

Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -190,13 +190,11 @@ def async_genkey(sid, codename):
     if not crypto_util.getkey(g.sid) and g.source.flagged:
         async_genkey(g.sid, g.codename)
 
-    # if this was a redirect from the login page, flash a message if there are
-    # no replies to clarify "check for replies" flow (#393)
-    if request.args.get('from_login') == '1' and len(replies) == 0:
-        flash("There are no replies at this time. You can submit more documents from this code name below.", "notification")
+    has_submissions = True if sum(g.source.documents_messages_count().values()) else False
 
     return render_template('lookup.html', codename=g.codename, replies=replies,
-            flagged=g.source.flagged, haskey=crypto_util.getkey(g.sid))
+            flagged=g.source.flagged, haskey=crypto_util.getkey(g.sid),
+            has_submissions=has_submissions)
 
 
 def normalize_timestamps(sid):


Test patch:
Repository: freedomofpress/securedrop

Issue:
Inconsistent use of "codename" vs "code name"
We should use either "codename" or "code name" consistently throughout. For example:
- on http://localhost:8080/generate it says “remember this code”
- on /generate it says "we're assigning you a unique code name.”
- on /generate it says “already have a codename?”
- on /login it says “enter your codename”
- on /lookup it says "You can submit more documents from this code name below.”
- on /lookup it says "Remember, your codename is”

I prefer "codename".



Discussion:
This is a duplicate of #495, but since this issue has more information I will go ahead and close the other one.


Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -192,7 +192,7 @@ def async_genkey(sid, codename):
     # if this was a redirect from the login page, flash a message if there are
     # no replies to clarify "check for replies" flow (#393)
     if request.args.get('from_login') == '1' and len(replies) == 0:
-        flash("There are no replies at this time. You can submit more documents from this code name below.", "notification")
+        flash("There are no replies at this time. You can submit more documents from this codename below.", "notification")
 
     return render_template('lookup.html', codename=g.codename, replies=replies,
             flagged=g.source.flagged, haskey=crypto_util.getkey(g.sid))


Test patch:
diff --git a/securedrop/tests/test_unit_source.py b/securedrop/tests/test_unit_source.py
--- a/securedrop/tests/test_unit_source.py
+++ b/securedrop/tests/test_unit_source.py
@@ -49,7 +49,7 @@ def test_generate(self):
             session_codename = session['codename']
         self.assertIn("Remember this code and keep it secret", rv.data)
         self.assertIn(
-            "To protect your identity, we're assigning you a unique code name.",
+            "To protect your identity, we're assigning you a unique codename.",
             rv.data)
         codename = self._find_codename(rv.data)
         # default codename length is 8 words
Repository: freedomofpress/securedrop

Issue:
Standardize on "SecureDrop" or "Securedrop"
In both the language on the web interface and in the documentation, we alternate between "SecureDrop" and "Securedrop." We should pick one and use it consistently. I prefer "SecureDrop".



Discussion:
+1 for "SecureDrop"

also +1 for "SecureDrop" -- thought we already standardized this


Resolving patch:
diff --git a/securedrop/manage.py b/securedrop/manage.py
--- a/securedrop/manage.py
+++ b/securedrop/manage.py
@@ -65,7 +65,7 @@ def test_unit():
 
 def reset():
     """
-    Clears the Securedrop development application's state, restoring it to the
+    Clears the SecureDrop development application's state, restoring it to the
     way it was immediately after running `setup_dev.sh`. This command:
     1. Erases the development sqlite database file
     2. Regenerates the database


Test patch:
Repository: freedomofpress/securedrop

Issue:
Location of replies unclear for sources
As a source, it it unclear where I should go to see messages from the journalist.

When I choose the “Check for response” and choose “continue” it should clearly show me a “Messages” area of the screen, and say “No replies from the journalist”, or list the messages (if there are any).

We currently flash a message that says "you received a new reply", but this is confusing. It only appears once, so if the user refreshes the page or navigates away and back again, they will not see that message (they will still see the reply, until it is deleted).

Another option would be to make “submit new documents/messages” a separate page from “check for replies from the journalist” page.



Discussion:

Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -189,11 +189,6 @@ def async_genkey(sid, codename):
     if not crypto_util.getkey(g.sid) and g.source.flagged:
         async_genkey(g.sid, g.codename)
 
-    # if this was a redirect from the login page, flash a message if there are
-    # no replies to clarify "check for replies" flow (#393)
-    if request.args.get('from_login') == '1' and len(replies) == 0:
-        flash("There are no replies at this time. You can submit more documents from this codename below.", "notification")
-
     return render_template('lookup.html', codename=g.codename, replies=replies,
             flagged=g.source.flagged, haskey=crypto_util.getkey(g.sid))
 
@@ -264,7 +259,7 @@ def delete():
     if msgid not in potential_files:
         abort(404)  # TODO are the checks necessary?
     store.secure_unlink(store.path(g.sid, msgid))
-    flash("Reply deleted.", "notification")
+    flash("Reply deleted", "notification")
 
     return redirect(url_for('lookup'))
 


Test patch: