internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: freedomofpress/securedrop

Issue:
Version info should be displayed on the source's interface
The source's interface should display the securedrop's version info on each page.



Discussion:
The first version with version numbers will be 0.1. Right now (pre-release) I suppose the version is 0.1development. It would be great if we could record the version number somewhere and update it with each release, and then make it get displayed to sources (and maybe journalists too) so they know if the SecureDrop they're using is up-to-date or not.

Would a good place to record this be in `deadrop/config.py`?

Hmm, maybe not config.py since that's used for editable settings (and that really ought to be gitignored too, if it isn't-- I think). We don't want anyone to accidentally edit it. Maybe deadrop/version.py?


Resolving patch:
diff --git a/modules/deaddrop/files/deaddrop/source.py b/modules/deaddrop/files/deaddrop/source.py
--- a/modules/deaddrop/files/deaddrop/source.py
+++ b/modules/deaddrop/files/deaddrop/source.py
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 import os, uuid, datetime
 import web
-import config, crypto, background, store
+import config, crypto, background, store, version
 
 urls = (
   '/', 'index',
@@ -9,7 +9,8 @@
   '/lookup/', 'lookup',
 )
 
-render = web.template.render(config.SOURCE_TEMPLATES_DIR, base='base')
+render = web.template.render(config.SOURCE_TEMPLATES_DIR, base='base',
+                             globals={'version':version.__version__})
 
 class index:
   def GET(self):
diff --git a/modules/deaddrop/files/deaddrop/version.py b/modules/deaddrop/files/deaddrop/version.py
new file mode 100644
--- /dev/null
+++ b/modules/deaddrop/files/deaddrop/version.py
@@ -0,0 +1 @@
+__version__ = '0.1development'


Test patch:
Repository: freedomofpress/securedrop

Issue:
Collection links in journalist interface 404
Steps to reproduce:
1. Set up according to instructions in HACKING.md
2. As source, visit the site. "Submit Documents", "Continue". Enter a test message and "Submit".
3. Visit the journalist site. Click the link of the new submission.

Expected behavior: Page contains a link to download the new submission

Actual behavior: 404, blank page with the text "not found"



Discussion:

Resolving patch:
diff --git a/modules/deaddrop/files/deaddrop/journalist.py b/modules/deaddrop/files/deaddrop/journalist.py
--- a/modules/deaddrop/files/deaddrop/journalist.py
+++ b/modules/deaddrop/files/deaddrop/journalist.py
@@ -6,8 +6,8 @@
 urls = (
   '/', 'index',
   '/reply/', 'reply',
-  '/([a-f0-9]+)/', 'col',
-  '/([a-f0-9]+)/([0-9]+\.[0-9]+(?:_msg|_doc|)\.gpg)', 'doc'
+  '/([A-Z1-7]+)/', 'col',
+  '/([A-Z1-7]+)/([0-9]+\.[0-9]+(?:_msg|_doc|)\.gpg)', 'doc'
 )
 
 render = web.template.render(config.JOURNALIST_TEMPLATES_DIR, base='base')


Test patch:
Repository: freedomofpress/securedrop

Issue:
Unsafe hmac key generation
Please note how little entropy is included in the hmac key: https://github.com/freedomofpress/securedrop/blob/master/server_setup.sh#L198 -- it would not be difficult for an attacker to generate all 86400 possible datestamps if they can guess the date the server was installed. Since there are only a few hundred days since this software was released, brute forcing every second since release wouldn't even be all that difficult.

Best would use a key derivation function that takes a very long time, such as PBKDF2, in conjunction with more sources of entropy. Maybe even a base64 output from /dev/urandom or something similar...

Thanks



Discussion:
Good point about the entropy. An easy fix:

```
hmac_secret=`date | sha256sum | cut -d ' ' -f1`
```

to

```
hmac_secret=`head -c 128 /dev/urandom | sha256sum | cut -d ' ' -f1`
```

However, more to the point - I grep'd for HMAC_SECRET across the entire codebase (`grep -rnIi HMAC_SECRET *`) and I _don't see it actually being used anywhere!_ Can we just get rid of it?


Resolving patch:
diff --git a/modules/deaddrop/files/deaddrop/example_config.py b/modules/deaddrop/files/deaddrop/example_config.py
--- a/modules/deaddrop/files/deaddrop/example_config.py
+++ b/modules/deaddrop/files/deaddrop/example_config.py
@@ -7,7 +7,6 @@
 JOURNALIST_TEMPLATES_DIR='./journalist_templates'
 WORD_LIST='./wordlist'
 
-HMAC_SECRET='c75f391d54717cbfbd89d3a5597dacad0eab42242661512d18d8087a3c842128' # $ date | sha256sum
 BCRYPT_SALT='$2a$12$gLZnkcyhZBrWbCZKHKYgKee8g/Yb9O7.24/H.09Yu9Jt9hzW6n0Ky' # bcrypt.gensalt()
 
 # Use different settings for tests


Test patch:
Repository: freedomofpress/securedrop

Issue:
Sources should not be allowed to choose insecure codenames
A form was added in a607e9f4 to allow sources to choose the length of their randomly generated codenames. Sources are allowed to choose codenames from length 4 to 10.

The argument for allowing sources to choose shorter codenames is that they are easier to remember, and after a certain length sources will be forced to write them down, possibly causing a loss of plausible deniability. However, the codename lengths offered now are far too short.

We want source codenames to have a _minimum_ of 80 bits of entropy, and ideally they would have a 128 bit security level. There are currently 6969 words in our wordlist, so:

log(6969^10, 2) = 127.66 bits of entropy
log(6969^9, 2) = 114.9
log(6969^8, 2) = 102.13
log(6969^7, 2) = 89.36
log(6969^6, 2) = 76.60

So the minimum allowable length should be 7 words. 10 is a good maximum, as that pretty much gets to 128 bits.

Since the source's security relies entirely on the difficulty in guessing their codename, we should be strict about this. It is a given that most humans cannot remember passwords with sufficient entropy in today's security environment. Writing them down should be [encouraged](https://www.schneier.com/blog/archives/2005/06/write_down_your.html) - although we should also educate sources on what to do once they're done communicating with a journalist (including destroying any written copies of their codename).



Discussion:
Or increase the wordlist to about 12000
log( 12000^6, 2 ) = 81.30


Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -97,7 +97,7 @@ def generate():
     number_words = 8
     if request.method == 'POST':
         number_words = int(request.form['number-words'])
-        if number_words not in range(4, 11):
+        if number_words not in range(7, 11):
             abort(403)
     session['codename'] = crypto_util.genrandomid(number_words)
     return render_template('generate.html', codename=session['codename'])


Test patch:
diff --git a/securedrop/test.py b/securedrop/test.py
--- a/securedrop/test.py
+++ b/securedrop/test.py
@@ -109,7 +109,7 @@ def test_generate(self):
 
     def test_regenerate_valid_lengths(self):
         """Make sure we can regenerate all valid length codenames"""
-        for codename_len in xrange(4, 11):
+        for codename_len in xrange(7, 11):
             response = self.client.post('/generate', data={
                 'number-words': str(codename_len),
             })
Repository: freedomofpress/securedrop

Issue:
Possible path confusion / traversal via imprecise store.verify()
The method `store.verify()` checks file paths provided via URL and other ways and raises an exception if they are not matching the validation criteria.

A problem with this validation process was spotted: `os.path.commonprefix()` is not sufficient to check if the path is inside the configured store path. It only compares character by character. Thus allows to navigate into another folder when they share the same start string.

```
Example:     config.STORE_DIR = '/opt/store'
PoC:         store.verify('/opt/store_backup')
```

Mitigation has to make sure, that the path is inside the configured store folder. A mitigation could be to add another check in `store.verify()` with `os.path.relpath(p, config.STORE_DIR)`. If the absolute path p is not inside the store directory, `os.path.relpath()` will return a string starting with '../'.

Example:

```
os.path.relpath('/opt/store_backup', config.STORE_DIR) == '../store_backup'
```

**Reported as part of the cure53 audit of 0.2 as: SD-01-006**



Discussion:

Resolving patch:
diff --git a/securedrop/store.py b/securedrop/store.py
--- a/securedrop/store.py
+++ b/securedrop/store.py
@@ -35,13 +35,13 @@ def verify(p):
     if not p == os.path.abspath(p):
         raise PathException("The path is not absolute and/or normalized")
 
-    if os.path.commonprefix([config.STORE_DIR, p]) != config.STORE_DIR:
+    # Check that the path p is in config.STORE_DIR
+    if os.path.relpath(p, config.STORE_DIR).startswith('..'):
         raise PathException("Invalid directory %s" % (p, ))
 
-    filename = os.path.basename(p)
-    ext = os.path.splitext(filename)[-1]
-
     if os.path.isfile(p):
+        filename = os.path.basename(p)
+        ext = os.path.splitext(filename)[-1]
         if filename == '_FLAG':
             return True
         if ext != '.gpg':


Test patch:
diff --git a/securedrop/test.py b/securedrop/test.py
--- a/securedrop/test.py
+++ b/securedrop/test.py
@@ -530,6 +530,8 @@ def tearDown(self):
     def test_verify(self):
         with self.assertRaises(store.PathException):
             store.verify(os.path.join(config.STORE_DIR, '..', 'etc', 'passwd'))
+        with self.assertRaises(store.PathException):
+            store.verify(config.STORE_DIR + "_backup")
 
     def test_get_zip(self):
         sid = 'EQZGCJBRGISGOTC2NZVWG6LILJBHEV3CINNEWSCLLFTUWZJPKJFECLS2NZ4G4U3QOZCFKTTPNZMVIWDCJBBHMUDBGFHXCQ3R'
Repository: freedomofpress/securedrop

Issue:
Stuck in "Sorry, we couldn't locate what you requested" state
As a source, after generating a code and clicking the "Continue..." button and going back, then refreshing you get stuck on a page that says "Sorry, we couldn't locate what you requested". Going to any URL causes the same message to be displayed and you have to close the browser to be able to get back to a normal state.



Discussion:

Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -115,6 +115,7 @@ def generate():
         if number_words not in range(7, 11):
             abort(403)
     session['codename'] = crypto_util.genrandomid(number_words)
+    session.pop('logged_in', None)
     # TODO: make sure this codename isn't a repeat
     return render_template('generate.html', codename=session['codename'])
 


Test patch:
Repository: freedomofpress/securedrop

Issue:
securedrop_init script in Tails doesn't work right if you run it twice
It appends torrc-additions to torrc multiple times, and it should just append it once.

securedrop_init script in Tails doesn't work right if you run it twice
It appends torrc-additions to torrc multiple times, and it should just append it once.



Discussion:
Is there any reason to run an init script twice?
On Apr 17, 2014 10:13 AM, "Micah Lee" notifications@github.com wrote:

> It appends torrc-additions to torrc multiple times, and it should just
> append it once.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/freedomofpress/securedrop/issues/355
> .

Nope, but if you do the torrc won't reload because of the same HidServAuth line multiple times. People might accidentally run it more than once (I know I have-- especially if Tails has been running for a long time and I can't remember if I ran it when I first booted or not). But it should be a quick and easy fix.

Is there any reason to run an init script twice?
On Apr 17, 2014 10:13 AM, "Micah Lee" notifications@github.com wrote:

> It appends torrc-additions to torrc multiple times, and it should just
> append it once.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/freedomofpress/securedrop/issues/355
> .

Nope, but if you do the torrc won't reload because of the same HidServAuth line multiple times. People might accidentally run it more than once (I know I have-- especially if Tails has been running for a long time and I can't remember if I ran it when I first booted or not). But it should be a quick and easy fix.


Resolving patch:
diff --git a/tails_files/securedrop_init.py b/tails_files/securedrop_init.py
--- a/tails_files/securedrop_init.py
+++ b/tails_files/securedrop_init.py
@@ -31,7 +31,7 @@
         open(path_torrc_backup, 'w').write(torrc)
 
     # append the additions
-    open(path_torrc, 'a').write(torrc_additions)
+    open(path_torrc, 'w').write(torrc+torrc_additions)
 
     # reload tor
     subprocess.call(['/usr/sbin/service', 'tor', 'reload'])


Test patch:
Repository: freedomofpress/securedrop

Issue:
Display number of docs and messages per source in source list
> At the moment each source in the list displays: source codename, last updated. It would be helpful to also see: total # of messages/docs.

Extracted from #322



Discussion:
Currently to do this we would have to do a `get_docs` call, which in turn calls `os.listdir` for each source.  I guess this is okay, but maybe we should wait until some data about documents is stored in the db, so we don't have to be so disk-io heavy on this?

I see there is a submissions table, I'll work off of that.  Working on this now: https://github.com/hainish/securedrop/tree/issues/371


Resolving patch:
diff --git a/securedrop/db.py b/securedrop/db.py
--- a/securedrop/db.py
+++ b/securedrop/db.py
@@ -59,6 +59,19 @@ def journalist_filename(self):
         valid_chars = 'abcdefghijklmnopqrstuvwxyz1234567890-_'
         return ''.join([c for c in self.journalist_designation.lower().replace(' ', '_') if c in valid_chars])
 
+    def documents_messages_count(self):
+        try:
+            return self.docs_msgs_count
+        except AttributeError:
+            self.docs_msgs_count = {'messages': 0, 'documents': 0}
+            for submission in self.submissions:
+                if submission.filename.endswith('msg.gpg'):
+                    self.docs_msgs_count['messages'] += 1
+                elif submission.filename.endswith('doc.zip.gpg'):
+                    self.docs_msgs_count['documents'] += 1
+            return self.docs_msgs_count
+
+
 class Submission(Base):
     __tablename__ = 'submissions'
     id = Column(Integer, primary_key=True)


Test patch:
Repository: freedomofpress/securedrop

Issue:
Add flash message when there are no replies on a sumission
The current flow if there are no replies on a submission make it look like I am just back on the submit flow.

It would be great to work this flow out a bit better in the future.

For now, if we could add a flash message to give an indication of the flow that would be great.

Suggested flash message: "There are no replies as yet on this submission. You can submit more documents under this code name below."



Discussion:

Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -167,7 +167,12 @@ def async_genkey(sid, codename):
     if not crypto_util.getkey(g.sid) and g.source.flagged:
         async_genkey(g.sid, g.codename)
 
-    return render_template('lookup.html', codename=g.codename, msgs=replies,
+    # if this was a redirect from the login page, flash a message if there are
+    # no replies to clarify "check for replies" flow (#393)
+    if request.args.get('from_login') == '1' and len(replies) == 0:
+        flash("There are no replies at this time. You can submit more documents from this code name below.", "notification")
+
+    return render_template('lookup.html', codename=g.codename, replies=replies,
             flagged=g.source.flagged, haskey=crypto_util.getkey(g.sid))
 
 
@@ -251,7 +256,7 @@ def login():
         codename = request.form['codename']
         if valid_codename(codename):
             session.update(codename=codename, logged_in=True)
-            return redirect(url_for('lookup'))
+            return redirect(url_for('lookup', from_login='1'))
         else:
             flash("Sorry, that is not a recognized codename.", "error")
     return render_template('login.html')


Test patch:
Repository: freedomofpress/securedrop

Issue:
Make it quicker to do regular dev tasks
There should be a script that does these things in vagrant for you:
- start servers
- restart servers
- run tests
- delete then regen database / SECUREDROP_ROOT
- maybe some other stuff

Right now you have to `vagrant ssh` and do all this stuff manually.



Discussion:
We should call this `manage.py`, which will be familiar to Django developers among others.


Resolving patch:
diff --git a/securedrop/manage.py b/securedrop/manage.py
new file mode 100755
--- /dev/null
+++ b/securedrop/manage.py
@@ -0,0 +1,58 @@
+#!/usr/bin/env python
+
+import sys
+import os
+import shutil
+import subprocess
+
+import config
+import db
+
+def start():
+    subprocess.Popen(['python', 'source.py'])
+    subprocess.Popen(['python', 'journalist.py'])
+    print "The web application is running, and available on your Vagrant host at the following addresses:"
+    print "Source interface:     localhost:8080"
+    print "Journalist interface: localhost:8081"
+
+def test():
+    """
+    Runs the test suite
+    """
+    # TODO: we could implement test.sh's functionality here, and get rid of
+    # test.sh (now it's just clutter, and confusing)
+    subprocess.call(["./test.sh"])
+
+def reset():
+    """
+    Clears the Securedrop development application's state, restoring it to the
+    way it was immediately after running `setup_dev.sh`. This command:
+    1. Erases the development sqlite database file ($SECUREDROP_ROOT/db.sqlite)
+    2. Regenerates the database
+    3. Erases stored submissions and replies from $SECUREDROP_ROOT/store
+    """
+    # Erase the development db file
+    assert hasattr(config, 'DATABASE_FILE'), "TODO: ./manage.py doesn't know how to clear the db if the backend is not sqlite"
+    os.remove(config.DATABASE_FILE)
+
+    # Regenerate the database
+    db.init_db()
+
+    # Clear submission/reply storage
+    for source_dir in os.listdir(config.STORE_DIR):
+        # Each entry in STORE_DIR is a directory corresponding to a source
+        shutil.rmtree(os.path.join(config.STORE_DIR, source_dir))
+
+def main():
+    valid_cmds = ["start", "test", "reset"]
+    help_str = "./manage.py {{{0}}}".format(','.join(valid_cmds))
+
+    if len(sys.argv) != 2 or sys.argv[1] not in valid_cmds:
+        print help_str
+        sys.exit(1)
+
+    cmd = sys.argv[1]
+    getattr(sys.modules[__name__], cmd)()
+
+if __name__ == "__main__":
+    main()


Test patch:
Repository: freedomofpress/securedrop

Issue:
codename collisions
If a codename collision is detected the application only logs a message it does not prevent that codename from being used by a the second source. While this situation should not exist under normal working conditions it is a defense in depth option that should be implemented.
mentioned by @iSec.



Discussion:

Resolving patch:
diff --git a/securedrop/source.py b/securedrop/source.py
--- a/securedrop/source.py
+++ b/securedrop/source.py
@@ -115,17 +115,34 @@ def index():
     return render_template('index.html')
 
 
+def generate_unique_codename(num_words):
+    """Generate random codenames until we get an unused one"""
+    while True:
+        codename = crypto_util.genrandomid(num_words)
+        sid = crypto_util.hash_codename(codename) # scrypt (slow)
+        matching_sources = Source.query.filter(Source.filesystem_id == sid).all()
+        if len(matching_sources) == 0:
+            return codename
+
+
 @app.route('/generate', methods=('GET', 'POST'))
 def generate():
+    # Popping this key prevents errors when a logged in user returns to /generate.
+    # TODO: is this the best experience? A logged in user will be automatically
+    # logged out if they navigate to /generate by accident, which could be
+    # confusing. It might be better to instead redirect them to the lookup
+    # page, or inform them that they're logged in.
+    session.pop('logged_in', None)
+
     number_words = 8
     if request.method == 'POST':
         number_words = int(request.form['number-words'])
         if number_words not in range(7, 11):
             abort(403)
-    session['codename'] = crypto_util.genrandomid(number_words)
-    session.pop('logged_in', None)
-    # TODO: make sure this codename isn't a repeat
-    return render_template('generate.html', codename=session['codename'])
+
+    codename = generate_unique_codename(number_words)
+    session['codename'] = codename
+    return render_template('generate.html', codename=codename)
 
 
 @app.route('/create', methods=['POST'])
@@ -136,11 +153,7 @@ def create():
     db_session.add(source)
     db_session.commit()
 
-    if os.path.exists(store.path(sid)):
-        # if this happens, we're not using very secure crypto
-        log.warning("Got a duplicate ID '%s'" % sid)
-    else:
-        os.mkdir(store.path(sid))
+    os.mkdir(store.path(sid))
 
     session['logged_in'] = True
     return redirect(url_for('lookup'))


Test patch:
Repository: freedomofpress/securedrop

Issue:
./manage.py reset deletes the test db, not development db
<pre>
vagrant@precise64:/vagrant/securedrop$ ls -al /home/vagrant/.securedrop/db.sqlite 
-rw-r--r-- 1 vagrant vagrant 6144 Jun 16 02:45 /home/vagrant/.securedrop/db.sqlite
vagrant@precise64:/vagrant/securedrop$ ./manage.py reset
vagrant@precise64:/vagrant/securedrop$ ls -al /home/vagrant/.securedrop/db.sqlite 
-rw-r--r-- 1 vagrant vagrant 6144 Jun 16 02:45 /home/vagrant/.securedrop/db.sqlite
</pre>



Discussion:
This is because the `import config` on line 45 is not run - `config` has already been imported by line 21 of `test_unit.py`, right after the env has been set as `test` on line 20.

Moving `from tests import test_unit, test_journalist, test_single_star` into the `test` function will fix this.


Resolving patch:
diff --git a/securedrop/manage.py b/securedrop/manage.py
--- a/securedrop/manage.py
+++ b/securedrop/manage.py
@@ -5,7 +5,8 @@
 import shutil
 import subprocess
 import unittest
-from tests import test_unit, test_journalist, test_single_star
+
+import config
 
 os.environ['SECUREDROP_ENV'] = 'development'
 
@@ -22,13 +23,6 @@ def test():
     """
     Runs the test suite
     """
-    # TODO: we could implement test.sh's functionality here, and get rid of
-    # test.sh (now it's just clutter, and confusing)
-    test_suites = [test_unit, test_journalist, test_single_star]
-    for test_suite in test_suites:
-        test_loader = unittest.defaultTestLoader.loadTestsFromModule(test_suite)
-        test_runner = unittest.TextTestRunner(verbosity=2)
-        test_runner.run(test_loader)
     subprocess.call(["./test.sh"])
 
 
@@ -40,9 +34,6 @@ def reset():
     2. Regenerates the database
     3. Erases stored submissions and replies from $SECUREDROP_ROOT/store
     """
-    # HACK: import here so they don't interfere with the test setup
-    # (environment variable hasn't been set yet)
-    import config
     import db
 
     # Erase the development db file


Test patch: