internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: freedomofpress/securedrop

Issue:
Selected codename length resets to default when a new codename is requested
If a source wants to request a 10 word codename on /generate, she can do so by selecting "10 words" in the dropdown and clicking on the refresh-button on the right. If she is not happy with the new codename, she will have to select "10 words" in the dropdown again before clicking the refresh-button. This happens because the selected codename length resets to default when a new codename is requested. Seeing how we do not store any state, I am not sure if there is a good way to solve this.



Discussion:

Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -136,15 +136,15 @@ def generate():
     # page, or inform them that they're logged in.
     session.pop('logged_in', None)
 
-    number_words = 8
+    num_words = 8
     if request.method == 'POST':
-        number_words = int(request.form['number-words'])
-        if number_words not in range(7, 11):
+        num_words = int(request.form['number-words'])
+        if num_words not in range(7, 11):
             abort(403)
 
-    codename = generate_unique_codename(number_words)
+    codename = generate_unique_codename(num_words)
     session['codename'] = codename
-    return render_template('generate.html', codename=codename)
+    return render_template('generate.html', codename=codename, num_words=num_words)
 
 
 @app.route('/create', methods=['POST'])


Test patch:
Repository: freedomofpress/securedrop

Issue:
Seven word code name option is weaker than the recommended default
When the source generates a new code name, the default is to recommend a string that is eight words long. The source can select strings that contain more words, but the source can also select a string that has fewer words than the recommended default. Available options should ideally only be stronger than the default recommendation, not weaker.



Discussion:
I think we should change the recommended default to be 7 words instead of 8. Here are the reasons why:
1. 7 word codenames are easier to memorize. They seem (at least to me) to have a nice 4-3 rhythm to them, similar to a telephone number.
2. They provide a sufficient amount of entropy. `math.log(6699**7, 2) = 88.96811025580541`, and since we use scrypt with `N = 2**14`, the stored hashes have an additional 14 bits of entropy for a total of 102.96.
3. Available options will only be stronger than the default recommendation, as noted in the original comment.

I think 102.96 bits of entropy is probably good enough.

It would be great if we could make it 128 bits, because that's what cryptographers say makes it completely impossible to crack, assuming you could use all of the computing power available on Earth and hundreds of thousands of years. But 102.96 is really good anyway.

I think if sources want to memorize their codenames, making them 7 long is fine. If sources plan on writing them down, then they could do 9 words to pass the 128-bit threshold.


Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -132,7 +132,7 @@ def generate():
     # page, or inform them that they're logged in.
     session.pop('logged_in', None)
 
-    num_words = 8
+    num_words = 7
     if request.method == 'POST':
         num_words = int(request.form['number-words'])
         if num_words not in range(7, 11):


Test patch:
diff --git a/securedrop/tests/test_unit_source.py b/securedrop/tests/test_unit_source.py
--- a/securedrop/tests/test_unit_source.py
+++ b/securedrop/tests/test_unit_source.py
@@ -52,8 +52,8 @@ def test_generate(self):
             "To protect your identity, we're assigning you a unique codename.",
             rv.data)
         codename = self._find_codename(rv.data)
-        # default codename length is 8 words
-        self.assertEqual(len(codename.split()), 8)
+        # default codename length is 7 words
+        self.assertEqual(len(codename.split()), 7)
         # codename is also stored in the session - make sure it matches the
         # codename displayed to the source
         self.assertEqual(codename, escape(session_codename))
Repository: freedomofpress/securedrop

Issue:
Replies to the source are displayed in random order
The Source Interface displays replies from the journalist in a random order. From the top, I currently see a message received 7 minutes ago, a message received 1 minute ago, and a message received 3 minutes ago. The Document Interface displays the list of messages in the order they are sent to the source.



Discussion:
Hm, that sounds like a bug, possibly due to timestamp normalization.

This is not due to timestamp normalization, since we only normalize `source.submissions`. The random order is due to our use of Python's `os.listdir`, which returns [the directory entries in arbitrary order](https://docs.python.org/2/library/os.html#os.listdir).


Resolving patch:
diff --git a/securedrop/journalist.py b/securedrop/journalist.py
--- a/securedrop/journalist.py
+++ b/securedrop/journalist.py
@@ -443,7 +443,7 @@ def reply():
 def generate_code():
     original_journalist_designation = g.source.journalist_designation
     g.source.journalist_designation = crypto_util.display_id()
-    
+
     for doc in Submission.query.filter(Submission.source_id == g.source.id).all():
         doc.filename = store.rename_submission(g.sid, doc.filename, g.source.journalist_filename())
     db_session.commit()
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -151,7 +151,7 @@ def create():
     db_session.add(source)
     try:
         db_session.commit()
-    except IntegrityError as e: 
+    except IntegrityError as e:
         app.logger.error("Attempt to create a source with duplicate codename: %s" % (e,))
     else:
         os.mkdir(store.path(sid))
@@ -175,6 +175,9 @@ def lookup():
                 date = datetime.utcfromtimestamp(os.stat(store.path(g.sid, fn)).st_mtime)
                 replies.append(dict(id=fn, date=date, msg=msg))
 
+    # Sort the replies by date
+    replies.sort(key=lambda reply: reply['date'], reverse=True)
+
     def async_genkey(sid, codename):
         with app.app_context():
             background.execute(lambda: crypto_util.genkeypair(sid, codename))


Test patch:
Repository: freedomofpress/securedrop

Issue:
Don't armor encrypted submissions
SecureDrop currently armors encrypted submissions. This bloats the size of stored submissions significantly due to the encoding. For example, a 93 MB upload results in a 125.7 MB submission for the journalist to download.

Downloading anything over Tor is very slow (the aforementioned download took me, on average, 9 minutes to download). Therefore, unnecessarily increasing the size of submissions severely impacts usability. There is no reason that I can think of to ascii armor submissions - they are uploaded and downloaded over HTTP, which automatically handles encoding and de-encoding binary data.



Discussion:

Resolving patch:
diff --git a/securedrop/crypto_util.py b/securedrop/crypto_util.py
--- a/securedrop/crypto_util.py
+++ b/securedrop/crypto_util.py
@@ -158,7 +158,8 @@ def encrypt(plaintext, fingerprints, output=None):
     out = encrypt_fn(plaintext,
                      *fingerprints,
                      output=output,
-                     always_trust=True)
+                     always_trust=True,
+                     armor=False)
     if out.ok:
         return out.data
     else:


Test patch:
diff --git a/securedrop/tests/test_unit_integration.py b/securedrop/tests/test_unit_integration.py
--- a/securedrop/tests/test_unit_integration.py
+++ b/securedrop/tests/test_unit_integration.py
@@ -277,7 +277,7 @@ def _can_decrypt_with_key(self, msg, key_fpr, passphrase=None):
                     salt=crypto_util.SCRYPT_GPG_PEPPER)
         decrypted_data = gpg.decrypt(msg, passphrase=passphrase)
         self.assertTrue(decrypted_data.ok,
-                "Could not decrypt msg with key, gpg says: {}".format(decrypted_data.status))
+                "Could not decrypt msg with key, gpg says: {}".format(decrypted_data.stderr))
 
         # We have to clean up the temporary GPG dir
         shutil.rmtree(gpg_tmp_dir)
@@ -362,10 +362,10 @@ def helper_test_reply(self, test_reply, expected_success=True):
             doc_names_selected=checkbox_values
         ), follow_redirects=True)
         self.assertEqual(rv.status_code, 200)
-        pgp_msg_re = r'-----BEGIN PGP MESSAGE-----.*-----END PGP MESSAGE-----'
-        pgp_msg = re.search(pgp_msg_re, rv.data, re.MULTILINE|re.DOTALL).group(0)
-        self._can_decrypt_with_key(pgp_msg, config.JOURNALIST_KEY)
-        self._can_decrypt_with_key(pgp_msg, crypto_util.getkey(sid), codename)
+        zf = zipfile.ZipFile(StringIO(rv.data), 'r')
+        data = zf.read(zf.namelist()[0])
+        self._can_decrypt_with_key(data, config.JOURNALIST_KEY)
+        self._can_decrypt_with_key(data, crypto_util.getkey(sid), codename)
 
         # Test deleting reply on the journalist interface
         last_reply_number = len(soup.select('input[name="doc_names_selected"]')) - 1
Repository: freedomofpress/securedrop

Issue:
Add "stop", "restart" to manage.py



Discussion:

Resolving patch:
diff --git a/securedrop/manage.py b/securedrop/manage.py
--- a/securedrop/manage.py
+++ b/securedrop/manage.py
@@ -8,6 +8,7 @@
 import readline # makes the add_admin prompt kick ass
 from getpass import getpass
 import signal
+from time import sleep
 
 import qrcode
 import psutil
@@ -64,7 +65,7 @@ def start():
         print "Source interface:     localhost:8080"
         print "Journalist interface: localhost:8081"
     else:
-        print "The web application is already running.  Please use './manage.py stop' to stop."
+        print "The web application is already running.  Please use './manage.py restart' to stop and start again."
 
 
 def stop():
@@ -79,6 +80,12 @@ def stop():
         print "There was a problem stopping the web application."
 
 
+def restart():
+    stop()
+    sleep(0.1)
+    start()
+
+
 def test():
     """
     Runs the test suite
@@ -212,7 +219,7 @@ def listdir_fullpath(d):
 
 
 def main():
-    valid_cmds = ["start", "stop", "test_unit", "test", "reset", "add_admin", "clean_tmp"]
+    valid_cmds = ["start", "stop", "test_unit", "test", "restart", "reset", "add_admin", "clean_tmp"]
     help_str = "./manage.py {{{0}}}".format(','.join(valid_cmds))
 
     if len(sys.argv) != 2 or sys.argv[1] not in valid_cmds:


Test patch:
Repository: freedomofpress/securedrop

Issue:
Explain that encryption key is not generated until source logs back in
If a journalist wishes to reply to a source, the journalist has to (1) flag the source for a reply, and (2) wait for the source to log back in. If and when the source does log back in, (1) an encryption key is generated for the source and the journalist can write a reply.

While this process is explained to the journalist in the Document Interface, it is not made clear to the source in the Source Interface until _after_ the source has logged back in. Ideally, /lookup should explain the process for receiving and reading replies from the journalist. Additional bonus points for not cluttering the UI too much by adding more text. Maybe we can add a "Learn how the reply process work" link at the bottom?



Discussion:
Hey @garrettr, is there anything in 0.3 that explains this to the source if the alternative reply to flow kicks in?

If the source logs in or reloads the `/lookup` page after the journalist has flagged them for reply, the key will be generated and they will see a flashed notification:

> A journalist has been waiting for you to log in again so SecureDrop can generate a crypto key for you. Now that you have logged in, they are able to write you a reply. Check back later for replies.

I think that's pretty good, what do you think?

That's good. I wonder if it would be helpful to also add a message saying "Thanks! We have received your message. Please check back later for replies." (for submitted documents and messages). Thoughts?

Yeah, that's fine with me. I think it actually used to say that, and I removed it in the interest of brevity. I'm fine with adding it back in.

Now that 0.3 has a dedicated "Replies" section of the source interface "lookup" page, I don't think it's necessary to repeat "Please check back later for replies" every time a source submits something. However, it might be nice to show that extra text after their _first_ submission. What do you think, @runasand?


Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -216,6 +216,7 @@ def submit():
 
     fnames = []
     journalist_filename = g.source.journalist_filename()
+    first_submission = g.source.interaction_count == 0
 
     if msg:
         g.source.interaction_count += 1
@@ -228,6 +229,11 @@ def submit():
             journalist_filename, fh.filename, fh.stream))
         flash('{} "{}".'.format("Thanks! We received your document",
                                 fh.filename or '[unnamed]'), "notification")
+
+    if first_submission:
+        flash("Thanks for submitting something to SecureDrop! Please check back later for replies.",
+              "notification")
+
     for fname in fnames:
         submission = Submission(g.source, fname)
         db_session.add(submission)


Test patch:
Repository: freedomofpress/securedrop

Issue:
Ignore spaces in two-factor code on document interface login
A recent update to Google Authenticator on iPhone adds a space between the first and last 3 digits of the 6-digit TOTP code, presumably for readability. If you enter the code with the space in the login form on the Document Interface, login will fail due to the presence of the space. In general, two-factor clients often add spaces to their one-time codes in order to improve readability. This could be a pitfall for users with such a two-factor client.

The easy fix here is to strip spaces before checking the code.



Discussion:

Resolving patch:
diff --git a/securedrop/db.py b/securedrop/db.py
--- a/securedrop/db.py
+++ b/securedrop/db.py
@@ -245,7 +245,13 @@ def formatted_otp_secret(self):
         chunks = [ sec[i:i+4] for i in xrange(0, len(sec), 4) ]
         return ' '.join(chunks).lower()
 
+    def _format_token(self, token):
+        """Strips from authentication tokens the whitespace that many clients add for readability"""
+        return ''.join(token.split())
+
     def verify_token(self, token):
+        token = self._format_token(token)
+
         # Only allow each authentication token to be used once. This
         # prevents some MITM attacks.
         if token == self.last_token and LOGIN_HARDENING:


Test patch:
Repository: freedomofpress/securedrop

Issue:
A submission without a file upload or message should not count as a submission
If a source clicks the "Submit"-button without uploading a file or writing a message, the system still interprets this as a submission and (1) creates an entry in the list in the Document Interface, and (2) updates the counter to reflect that the source just did something. Would it be possible to check for and reject empty submissions?



Discussion:
Good catch! That should be very easy to fix.


Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -239,6 +239,11 @@ def submit():
     msg = request.form['msg']
     fh = request.files['fh']
 
+    # Don't bother submitting anything if it was an "empty" submission. #878.
+    if not (msg or fh):
+        flash("You must enter a message or choose a file to submit.", "error")
+        return redirect(url_for('lookup'))
+
     fnames = []
     journalist_filename = g.source.journalist_filename
     first_submission = g.source.interaction_count == 0


Test patch:
Repository: freedomofpress/securedrop

Issue:
Update kernel module blacklist
During an installation last week, we encountered an issue with the kernel module blacklist. The install was using the new generation of Intel NUCs ([NUC5i5RYK](http://www.amazon.com/dp/B00SD9ISIQ) and [NUC5i5RYH](http://www.amazon.com/dp/B00SD9IS1S/)). Unlike the previous generation of NUCs, which did not include wireless networking hardware by default, the new generation includes wireless networking hardware for Wifi and Bluetooth on the motherboard.

This means that Ubuntu running on the servers not only loaded the high-level kernel modules for wifi and bluetooth support (`iwlwifi` and `bluetooth`), it also loaded modules necessary for support on the specific (included) hardware: `iwlmvm` and `btusb`. When the `remove kernel modules` Ansible role ran, it failed with an error because it could not remove the top-level modules without removing their dependencies first.

A quickfix to get this working on the new hardware was to change `disabled_kernel_modules` in `group_vars/securedrop.yml` from:

``` yml
disabled_kernel_modules:
  - bluetooth
  - iwlwifi
```

to:

``` yml
disabled_kernel_modules:
  - btusb
  - bluetooth
  - iwlmvm
  - iwlwifi
```

The order of the modules is important! We need to make sure the the dependencies are removed prior to the target modules that depend on them.

This list is also likely specific to the new generation of Intel NUCs. If we want to support a wider variety of hardware, we may want to try being smart about removing kernel modules and their dependencies, e.g. something akin to this technique from [Stack Exchange](https://askubuntu.com/questions/317230/how-can-i-temporarily-disable-a-kernel-module).

Finally, we need to make sure this updated module blacklist still works on the old hardware as well.



Discussion:

Resolving patch:
diff --git a/securedrop/version.py b/securedrop/version.py
--- a/securedrop/version.py
+++ b/securedrop/version.py
@@ -1 +1 @@
-__version__ = '0.3.4'
+__version__ = '0.3.5'


Test patch:
Repository: freedomofpress/securedrop

Issue:
Totally unmemorizable stuff in the wordlist
With high probability a source get a phrase-like codename consisting of words. But sometimes one can get something like
`crest sill 86 crt plushy 13th rebut`.
Numbers are easy to forget. Words leave some trace in human memory (one can remember a word by its length/shape/etc). Numbers are not.
It's all because of [`wordlist`](https://github.com/freedomofpress/securedrop/blob/develop/securedrop/wordlist#L6515). A possible fix is to replace all numbers with words generated additionally.



Discussion:
I made a new diceware list (mostly by hand). This might be relevant here.
https://github.com/heartsucker/diceware


Resolving patch:
diff --git a/securedrop/_genwordlist.py b/securedrop/_genwordlist.py
deleted file mode 100644
--- a/securedrop/_genwordlist.py
+++ /dev/null
@@ -1,29 +0,0 @@
-"""
-Generates `wordlist` from The English Open Word List http://dreamsteep.com/projects/the-english-open-word-list.html
-Usage: Unzip the CSV files from the archive with the command `unzip EOWL-v1.1.2.zip EOWL-v1.1.2/CSV\ Format/*.csv`
-"""
-import re
-import string
-
-
-def just7(x):
-    return all(c in string.printable for c in x)
-
-words = set()
-
-for i in map(chr, range(65, 91)):
-    words.update(
-        x.strip() for x in file(
-            'EOWL-v1.1.2/CSV Format/%s Words.csv' %
-            i) if just7(x))
-
-fh = file('wordlist', 'w')
-for word in words:
-    if re.search('[^a-z0-9]', word):  # punctuation is right out
-        continue
-    if re.match(r'^([a-z])\1\1\1*$', word):  # yyyy is not a real word
-        continue
-    # EOWL contains bigrams xf, xg, xh, etc.
-    if re.match(r'^[a-z][a-z]$', word):
-        continue
-    fh.write('%s\n' % word)


Test patch:
Repository: freedomofpress/securedrop

Issue:
Add batch-delete functionality to replies in the Source Interface
Sources are encouraged to delete all replies when they are done with them. It would be great if the Source Interface had a button that deleted all replies, in addition to the button that deletes individual messages. 



Discussion:

Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -347,6 +347,20 @@ def delete():
     flash("Reply deleted", "notification")
     return redirect(url_for('lookup'))
 
+@app.route('/delete-all', methods=('POST',))
+@login_required
+def batch_delete():
+    replies = g.source.replies
+    if len(replies) == 0:
+        app.logger.error("Found no replies when at least one was expected")
+        return redirect(url_for('lookup'))
+    for reply in replies:
+        store.secure_unlink(store.path(g.sid, reply.filename))
+        db_session.delete(reply)
+    db_session.commit()
+
+    flash("All replies have been deleted", "notification")
+    return redirect(url_for('lookup'))
 
 def valid_codename(codename):
     # Ignore codenames that are too long to avoid DoS


Test patch:
Repository: freedomofpress/securedrop

Issue:
Merge changes from 0.3.7.post1 into develop
This is blocking on #1313.



Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -59,9 +59,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.3.5'
+version = '0.3.7'
 # The full version, including alpha/beta/rc tags.
-release = '0.3.5'
+release = '0.3.7'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.


Test patch:
Repository: freedomofpress/securedrop

Issue:
Merge changes from 0.3.8 into develop
This is blocking on #1313 and #1345.



Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -59,9 +59,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.3.7'
+version = '0.3.8'
 # The full version, including alpha/beta/rc tags.
-release = '0.3.7'
+release = '0.3.8'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
diff --git a/securedrop/version.py b/securedrop/version.py
--- a/securedrop/version.py
+++ b/securedrop/version.py
@@ -1 +1 @@
-__version__ = '0.3.7'
+__version__ = '0.3.8'


Test patch: