Repository: freedomofpress/securedrop

Issue:
Bulking downloading folder structure
I've noticed that in practice, it's really helpful if each message/document is in it's own folder. The reason is because in Tails, you double-click a .gpg file to decrypt it, and it creates the new file in the same folder. If it's a doc, then you right-click on the .zip to extract it to get the original file that was uploaded. Sometimes (if the source is savvy and follows the journalist key instructions) this is another .gpg file with a different name, and you double-click on that to decrypt it. And sometimes, it decrypts into a .zip or .tar.gz or something that the source sent, so you extract that. Etc, etc.

Basically, each message generates at least 2 files, sometimes 3 (if they post a gpg-encrypted message in the box). And each doc generates at least 3 files (.zip.gpg, .zip, and whatever is extracted from the .zip), sometimes way more. It gets really disorganized to have all of the files just sitting there in the same folder.

(Also, sometimes a source tries to upload the same document twice because they weren't sure if it worked the first time. If the two .zip.gpg files are in the same folder, you end up extracting the zip files over each other when the contents have the same filename.)

So anyway, I propose that the bulk download feature (and any future bulk download features, such as #372) shouldn't just put all the messages/docs into a zip file, but rather should build a directory structure like this:

```
| source_codename
|--| year-month-date
|--|--| doc_number
|--|--|--| document.gpg
```

So for example, if the source codename is "Ottoman mate" and they uploaded a message and then a document on May 22, the directory structure would be:

```
ottoman-mate
ottoman-mate/2014-05-22
ottoman-mate/2014-05-22/1
ottoman-mate/2014-05-22/1/1-ottoman_mate-msg.gpg
ottoman-mate/2014-05-22/2
ottoman-mate/2014-05-22/2/2-ottoman_mate-doc.zip.gpg
```

Make sense? I don't think the directory structure needs to be exactly like this, as long as it's organized sequentially and each message/doc is in it's own folder. (And the dates are super useful, especially if there's weeks of back-and-forth between journalist and source, to look back -- but the date could just be the date of last activity.)



Discussion:
@fowlslegs and I discussed this in person and decided this was a good way to organize the folder structure without having to make the changes to the db to save timestamps on each submission (which would be needed for the original structure) and could/would leak information as discussed in #822 

Structure:

```
| source_codename
|--| <doc_number>_<Source.last_updated>
|--|--| document.gpg
```

Example:

```
ottoman-mate
ottoman-mate/1_2014-05-22/
ottoman-mate/1_2014-05-22/1-ottoman_mate-msg.gpg
ottoman-mate/2_2014-05-22/
ottoman-mate/2_2014-05-22/2-ottoman_mate-doc.zip.gpg
```


Resolving patch:
diff --git a/securedrop/journalist.py b/securedrop/journalist.py
--- a/securedrop/journalist.py
+++ b/securedrop/journalist.py
@@ -687,19 +687,16 @@ def download(zip_basename, submissions):
     :param list submissions: A list of :class:`db.Submission`s to
                              include in the zipfile.
     """
-    # Mark the submissions that are about to be downloaded as such
+    zf = store.get_bulk_archive(submissions,
+                                zip_directory=zip_basename)
+    attachment_filename = "{}--{}.zip".format(
+        zip_basename, datetime.utcnow().strftime("%Y-%m-%d--%H-%M-%S"))
+
+    # Mark the submissions that have been downloaded as such
     for submission in submissions:
         submission.downloaded = True
     db_session.commit()
 
-    filenames = [store.path(submission.source.filesystem_id,
-                            submission.filename)
-                 for submission in submissions]
-
-    zf = store.get_bulk_archive(filenames,
-                                zip_directory=zip_basename)
-    attachment_filename = "{}--{}.zip".format(
-        zip_basename, datetime.utcnow().strftime("%Y-%m-%d--%H-%M-%S"))
     return send_file(zf.name, mimetype="application/zip",
                      attachment_filename=attachment_filename,
                      as_attachment=True)
diff --git a/securedrop/store.py b/securedrop/store.py
--- a/securedrop/store.py
+++ b/securedrop/store.py
@@ -68,17 +68,29 @@ def path(*s):
     return absolute
 
 
-def get_bulk_archive(filenames, zip_directory=''):
+def get_bulk_archive(selected_submissions, zip_directory=''):
+    """Generate a zip file from the selected submissions"""
     zip_file = tempfile.NamedTemporaryFile(prefix='tmp_securedrop_bulk_dl_',
                                            dir=config.TEMP_DIR,
                                            delete=False)
+    sources = set([i.source.journalist_designation for i in selected_submissions])
+    # The below nested for-loops are there to create a more usable
+    # folder structure per #383
     with zipfile.ZipFile(zip_file, 'w') as zip:
-        for filename in filenames:
-            verify(filename)
-            zip.write(filename, arcname=os.path.join(
-                zip_directory,
-                os.path.basename(filename)
-            ))
+        for source in sources:
+            submissions = [s for s in selected_submissions if s.source.journalist_designation == source]
+            for submission in submissions:
+                filename = path(submission.source.filesystem_id,
+                                submission.filename)
+                verify(filename)
+                document_number = submission.filename.split('-')[0]
+                zip.write(filename, arcname=os.path.join(
+                    zip_directory,
+                    source,
+                    "%s_%s" % (document_number,
+                               submission.source.last_updated.date()),
+                    os.path.basename(filename)
+                ))
     return zip_file
 
 


Test patch:
diff --git a/securedrop/tests/test_unit_journalist.py b/securedrop/tests/test_unit_journalist.py
--- a/securedrop/tests/test_unit_journalist.py
+++ b/securedrop/tests/test_unit_journalist.py
@@ -554,37 +554,51 @@ def test_delete_source_deletes_docs_on_disk(self):
 
     def test_download_selected_submissions_from_source(self):
         source, _ = utils.db_helper.init_source()
-        submissions = set(utils.db_helper.submit(source, 4))
-
+        submissions = utils.db_helper.submit(source, 4)
         selected_submissions = random.sample(submissions, 2)
         selected_fnames = [submission.filename
                            for submission in selected_submissions]
+        selected_fnames.sort()
 
         self._login_user()
         resp = self.client.post(
             '/bulk', data=dict(action='download',
                                sid=source.filesystem_id,
                                doc_names_selected=selected_fnames))
+
         # The download request was succesful, and the app returned a zipfile
         self.assertEqual(resp.status_code, 200)
         self.assertEqual(resp.content_type, 'application/zip')
         self.assertTrue(zipfile.is_zipfile(StringIO(resp.data)))
+
         # The submissions selected are in the zipfile
         for filename in selected_fnames:
-            self.assertTrue(zipfile.ZipFile(StringIO(resp.data)).getinfo(
-                os.path.join(source.journalist_filename, filename)))
+            self.assertTrue(
+                # Check that the expected filename is in the zip file
+                zipfile.ZipFile(StringIO(resp.data)).getinfo(
+                    os.path.join(
+                        source.journalist_filename,
+                        source.journalist_designation,
+                        "%s_%s" % (filename.split('-')[0], source.last_updated.date()),
+                        filename
+                    ))
+                )
+
         # The submissions not selected are absent from the zipfile
-        not_selected_submissions = submissions.difference(selected_submissions)
+        not_selected_submissions = set(submissions).difference(selected_submissions)
         not_selected_fnames = [submission.filename
                                for submission in not_selected_submissions]
+
         for filename in not_selected_fnames:
-            try:
+            with self.assertRaises(KeyError):
                 zipfile.ZipFile(StringIO(resp.data)).getinfo(
-                    os.path.join(source.journalist_filename, filename))
-            except KeyError:
-                pass
-            else:
-                self.assertTrue(False)
+                    os.path.join(
+                        source.journalist_filename,
+                        source.journalist_designation,
+                        "%s_%s" % (filename.split('-')[0],
+                                   source.last_updated.date()),
+                        filename
+                    ))
 
     def _bulk_download_setup(self):
         """Create a couple sources, make some submissions on their behalf,
@@ -592,19 +606,21 @@ def _bulk_download_setup(self):
         sources."""
         self.source0, _ = utils.db_helper.init_source()
         self.source1, _ = utils.db_helper.init_source()
-        self.submissions0 = set(utils.db_helper.submit(self.source0, 2))
-        self.submissions1 = set(utils.db_helper.submit(self.source1, 3))
+        self.journo0, _ = utils.db_helper.init_journalist()
+        self.submissions0 = utils.db_helper.submit(self.source0, 2)
+        self.submissions1 = utils.db_helper.submit(self.source1, 3)
         self.downloaded0 = random.sample(self.submissions0, 1)
         utils.db_helper.mark_downloaded(*self.downloaded0)
-        self.not_downloaded0 = self.submissions0.difference(self.downloaded0)
+        self.not_downloaded0 = set(self.submissions0).difference(self.downloaded0)
         self.downloaded1 = random.sample(self.submissions1, 2)
         utils.db_helper.mark_downloaded(*self.downloaded1)
-        self.not_downloaded1 = self.submissions1.difference(self.downloaded1)
+        self.not_downloaded1 = set(self.submissions1).difference(self.downloaded1)
 
 
     def test_download_unread_all_sources(self):
         self._bulk_download_setup()
         self._login_user()
+
         # Download all unread messages from all sources
         self.resp = self.client.post(
             '/col/process',
@@ -616,50 +632,98 @@ def test_download_unread_all_sources(self):
         self.assertEqual(self.resp.status_code, 200)
         self.assertEqual(self.resp.content_type, 'application/zip')
         self.assertTrue(zipfile.is_zipfile(StringIO(self.resp.data)))
+
         # All the not dowloaded submissions are in the zipfile
-        for submission in self.not_downloaded0.union(self.not_downloaded1):
+        for submission in self.not_downloaded0:
             self.assertTrue(
                 zipfile.ZipFile(StringIO(self.resp.data)).getinfo(
-                    os.path.join('unread', submission.filename))
+                    os.path.join(
+                        "unread",
+                        self.source0.journalist_designation,
+                        "%s_%s" % (submission.filename.split('-')[0],
+                                   self.source0.last_updated.date()),
+                        submission.filename
+                    ))
                 )
+        for submission in self.not_downloaded1:
+            self.assertTrue(
+                zipfile.ZipFile(StringIO(self.resp.data)).getinfo(
+                    os.path.join(
+                        "unread",
+                        self.source1.journalist_designation,
+                        "%s_%s" % (submission.filename.split('-')[0],
+                                   self.source1.last_updated.date()),
+                        submission.filename
+                    ))
+                )
+
         # All the downloaded submissions are absent from the zipfile
-        for submission in self.downloaded0 + self.downloaded1:
-            try:
+        for submission in self.downloaded0:
+            with self.assertRaises(KeyError):
+                zipfile.ZipFile(StringIO(self.resp.data)).getinfo(
+                    os.path.join(
+                        "unread",
+                        self.source0.journalist_designation,
+                        "%s_%s" % (submission.filename.split('-')[0],
+                                   self.source0.last_updated.date()),
+                        submission.filename
+                    ))
+
+        for submission in self.downloaded1:
+            with self.assertRaises(KeyError):
                 zipfile.ZipFile(StringIO(self.resp.data)).getinfo(
-                    os.path.join('unread', submission.filename))
-            except KeyError:
-                pass
-            else:
-                self.assertTrue(False)
+                    os.path.join(
+                        "unread",
+                        self.source1.journalist_designation,
+                        "%s_%s" % (submission.filename.split('-')[0],
+                                   self.source1.last_updated.date()),
+                        submission.filename
+                    ))
+
 
     def test_download_all_selected_sources(self):
         self._bulk_download_setup()
         self._login_user()
+
         # Dowload all messages from self.source1
         self.resp = self.client.post(
             '/col/process',
             data=dict(action='download-all',
                       cols_selected=[self.source1.filesystem_id]))
 
+        resp = self.client.post('/col/process',
+                                data=dict(action='download-all',
+                                          cols_selected=[self.source1.filesystem_id]))
+
         # The download request was succesful, and the app returned a zipfile
-        self.assertEqual(self.resp.status_code, 200)
-        self.assertEqual(self.resp.content_type, 'application/zip')
-        self.assertTrue(zipfile.is_zipfile(StringIO(self.resp.data)))
+        self.assertEqual(resp.status_code, 200)
+        self.assertEqual(resp.content_type, 'application/zip')
+        self.assertTrue(zipfile.is_zipfile(StringIO(resp.data)))
+
         # All messages from self.source1 are in the zipfile
         for submission in self.submissions1:
             self.assertTrue(
-                zipfile.ZipFile(StringIO(self.resp.data)).getinfo(
-                    os.path.join('all', submission.filename))
+                zipfile.ZipFile(StringIO(resp.data)).getinfo(
+                    os.path.join(
+                        "all",
+                        self.source1.journalist_designation,
+                        "%s_%s" % (submission.filename.split('-')[0],
+                                   self.source1.last_updated.date()),
+                        submission.filename)
+                    )
                 )
-        # All messages from self.source2 are absent from the zipfile
+
+        # All messages from self.source0 are absent from the zipfile
         for submission in self.submissions0:
-            try:
-                zipfile.ZipFile(StringIO(self.resp.data)).getinfo(
-                    os.path.join('all', submission.filename))
-            except KeyError:
-                pass
-            else:
-                self.assertTrue(False)
+            with self.assertRaises(KeyError):
+                zipfile.ZipFile(StringIO(resp.data)).getinfo(
+                    os.path.join(
+                        "all",
+                        self.source0.journalist_designation,
+                        "%s_%s" % (submission.filename.split('-')[0],
+                                   self.source0.last_updated.date()),
+                        submission.filename)
+                    )
 
     def test_add_star_redirects_to_index(self):
         source, _ = utils.db_helper.init_source()
diff --git a/securedrop/tests/test_unit_store.py b/securedrop/tests/test_unit_store.py
--- a/securedrop/tests/test_unit_store.py
+++ b/securedrop/tests/test_unit_store.py
@@ -39,7 +39,7 @@ def test_get_zip(self):
                                   submission.filename)
                      for submission in submissions]
 
-        archive = zipfile.ZipFile(store.get_bulk_archive(filenames))
+        archive = zipfile.ZipFile(store.get_bulk_archive(submissions))
         archivefile_contents = archive.namelist()
 
         for archived_file, actual_file in zip(archivefile_contents, filenames):