Repository: UAlbanyArchives/mailbagit (language: python)

Issue:
Pyinstaller executables not loading choices
**Describe the bug**
The GUI works fine natively in python, but after compiling it with pyinstaller, the format and derivative registries don't load correctly, so the `input_types` and `derivative_types` variables in `__init__.py` are empty lists. This makes no options available in the GUI. This also affects `mailbagit.exe` as well since it checks against those empty lists and raises an error when it doesn't include them, but I think that is more easily addressed.

**To Reproduce**
Steps to reproduce the behavior:
1. Install dev dependency (pyinstaller) with `pip install -e .[dev]` 
2. make a .exe with `pyinstaller --onefile mailbagit-gui.py` (same without `--onefile`)
3. Run `mailbagit-gui.exe`

**Expected behavior**
The GUI should list all possible parsers and derivatives for users. I imaging that it would be tough to have the .exe detect if `wkhtmltopdf` or `google-chrome` are installed, but I'd expect it to show all the formats and derivatives available at build time.

**Screenshots**

![pyinstaller_error](https://user-images.githubusercontent.com/10030353/167014497-1e29fe11-59ae-4635-b9af-34dd14fa2244.png)

**Environment (please complete the following information):**
_ GUI or command line?: GUI
- OS: Win10
- Python version: 3.9.12


Discussion / hints:
Poked at this a bit more and it looks like with pyinstaller the [formats](https://github.com/UAlbanyArchives/mailbag/blob/develop/mailbag/email_account.py#L30-L31) and [derivative
 ](https://github.com/UAlbanyArchives/mailbag/blob/develop/mailbag/derivative.py#L30-L31) registries don't load. 

If you have `$MAILBAG_LOG_LEVEL` set to `debug`, then it logs these registries. It logs both registries correctly when building the .exe, but when you actually run the .exe, it logs them both as empty dicts. 

In other words, this issue prevents both `mailbagit` and `mailbagit-gui` from running as pyinstaller executables and I don't know a work around at this point.

Fix patch:
diff --git a/data/msg-Digitization Archiving Solutions/1/Attachments/1/Content_ID.txt b/data/msg-Digitization Archiving Solutions/1/Attachments/1/Content_ID.txt
new file mode 100644
index 0000000..9b83e84
--- /dev/null
+++ b/data/msg-Digitization Archiving Solutions/1/Attachments/1/Content_ID.txt
@@ -0,0 +1,1 @@
+b77f8b0f48324f4ea767df91f1a85ff8
\ No newline at end of file
diff --git a/data/msg-Digitization Archiving Solutions/1/Attachments/2/Content_ID.txt b/data/msg-Digitization Archiving Solutions/1/Attachments/2/Content_ID.txt
new file mode 100644
index 0000000..7cf0165
--- /dev/null
+++ b/data/msg-Digitization Archiving Solutions/1/Attachments/2/Content_ID.txt
@@ -0,0 +1,1 @@
+1eed379e88f1438f9517c20f558bbff8
\ No newline at end of file
diff --git a/data/msg-Digitization Archiving Solutions/1/Attachments/3/Content_ID.txt b/data/msg-Digitization Archiving Solutions/1/Attachments/3/Content_ID.txt
new file mode 100644
index 0000000..161a59d
--- /dev/null
+++ b/data/msg-Digitization Archiving Solutions/1/Attachments/3/Content_ID.txt
@@ -0,0 +1,1 @@
+64ecedb978594310ac86265701e24b29
\ No newline at end of file
diff --git a/data/msg-Digitization Archiving Solutions/1/Derivatives_Path.txt b/data/msg-Digitization Archiving Solutions/1/Derivatives_Path.txt
index e69de29..945c9b4 100644
--- a/data/msg-Digitization Archiving Solutions/1/Derivatives_Path.txt
+++ b/data/msg-Digitization Archiving Solutions/1/Derivatives_Path.txt
@@ -0,0 +1,1 @@
+.
\ No newline at end of file
diff --git a/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/1/Derivatives_Path.txt b/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/1/Derivatives_Path.txt
index f3e1d0c..58c0c3a 100644
--- a/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/1/Derivatives_Path.txt
+++ b/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/1/Derivatives_Path.txt
@@ -1,1 +1,1 @@
-outlook2019_MSO_16.0.10377.20023_64-bit/outlook2019_MSO_16.0.10377.20023_64-bit.pst/Today at UAlbany
\ No newline at end of file
+outlook2019_MSO_16.0.10377.20023_64-bit/Top of Outlook data file/Inbox/Today at UAlbany
\ No newline at end of file
diff --git a/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/1/Message_Path.txt b/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/1/Message_Path.txt
index f3e1d0c..8287894 100644
--- a/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/1/Message_Path.txt
+++ b/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/1/Message_Path.txt
@@ -1,1 +1,1 @@
-outlook2019_MSO_16.0.10377.20023_64-bit/outlook2019_MSO_16.0.10377.20023_64-bit.pst/Today at UAlbany
\ No newline at end of file
+Top of Outlook data file/Inbox/Today at UAlbany
\ No newline at end of file
diff --git a/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/2/Derivatives_Path.txt b/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/2/Derivatives_Path.txt
index f3e1d0c..58c0c3a 100644
--- a/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/2/Derivatives_Path.txt
+++ b/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/2/Derivatives_Path.txt
@@ -1,1 +1,1 @@
-outlook2019_MSO_16.0.10377.20023_64-bit/outlook2019_MSO_16.0.10377.20023_64-bit.pst/Today at UAlbany
\ No newline at end of file
+outlook2019_MSO_16.0.10377.20023_64-bit/Top of Outlook data file/Inbox/Today at UAlbany
\ No newline at end of file
diff --git a/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/2/Message_Path.txt b/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/2/Message_Path.txt
index f3e1d0c..8287894 100644
--- a/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/2/Message_Path.txt
+++ b/data/pst-outlook2019_MSO_16.0.10377.20023_64-bit/2/Message_Path.txt
@@ -1,1 +1,1 @@
-outlook2019_MSO_16.0.10377.20023_64-bit/outlook2019_MSO_16.0.10377.20023_64-bit.pst/Today at UAlbany
\ No newline at end of file
+Top of Outlook data file/Inbox/Today at UAlbany
\ No newline at end of file
diff --git a/docs/Gemfile.lock b/docs/Gemfile.lock
index abab7fa..39ff3a4 100644
--- a/docs/Gemfile.lock
+++ b/docs/Gemfile.lock
@@ -74,7 +74,7 @@ DEPENDENCIES
   jekyll (~> 4.1.0)
   jekyll-feed (~> 0.6)
   just-the-docs
-  kramdown (~> 2.3.1)
+  kramdown (>= 2.3.1)
   tzinfo-data
 
 BUNDLED WITH
diff --git a/docs/install.md b/docs/install.md
index 1e0c240..2e1e85d 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -10,11 +10,12 @@ has_children: true
 
 Mailbag contains multiple tools for packaging mailbags according to the [Mailbag Specification]({{ site.baseurl }}/spec/).
 
-There are two different ways to install and run `mailbagit`.
+There are three different ways to install and run `mailbagit`.
 
 1. [Python install with pip]({{ site.baseurl}}/python)
-2. [Using a Docker image]({{ site.baseurl}}/docker)
+2. [Using Windows executables]({{ site.baseurl}}/exe)
+3. [Using a Docker image]({{ site.baseurl}}/docker)
 
-If you install `mailbagit` with Python, you will need to install either wkhtmltopdf or Google Chrome to create [PDF derivatives]({{ site.baseurl}}/pdf).
+If you install `mailbagit` with Python or use the Windows executables, you will need to install either wkhtmltopdf or Google Chrome to create [PDF derivatives]({{ site.baseurl}}/pdf).
 
-The Docker image comes with all dependencies preinstalled.
\ No newline at end of file
+The Docker image comes with all dependencies preinstalled.
diff --git a/docs/installing_mailbagit/docker.md b/docs/installing_mailbagit/docker.md
index 4376cdd..5315252 100755
--- a/docs/installing_mailbagit/docker.md
+++ b/docs/installing_mailbagit/docker.md
@@ -3,7 +3,7 @@ layout: page
 title: Docker Images
 permalink: /docker/
 parent: Installing mailbagit
-nav_order: 3
+nav_order: 4
 ---
 
 # Running mailbagit in a Docker container
diff --git a/docs/installing_mailbagit/exe.md b/docs/installing_mailbagit/exe.md
new file mode 100644
index 0000000..5ec5207
--- /dev/null
+++ b/docs/installing_mailbagit/exe.md
@@ -0,0 +1,23 @@
+---
+layout: page
+title: Windows Executables
+permalink: /exe/
+parent: Installing mailbagit
+nav_order: 3
+---
+
+# Windows Executables
+
+Executables are available for Windows that contain all dependencies except the [PDF dependencies]({{ site.baseurl }}/pdf). Download the latest `mailbagit.exe` and `mailbagit-gui.exe` files from the [Github releases](https://github.com/UAlbanyArchives/mailbag/releases).
+
+You can run `mailbagit.exe` in the command line [just as you would use the `mailbagit` command]({{ site.baseurl }}/mailbagit):
+
+```
+mailbagit.exe path/to/export.pst -i pst -d eml warc -m my_mailbag -r`
+```
+
+`mailbagit-gui.exe` will run just by double-clicking it.
+
+These executables are unsigned, so Windows will likely give you as "Windows protected your PC" warning from Microsoft Defender SmartScreen. You will need sufficient permissions to allow unsigned executables on you machine.
+
+![Screenshot Microsoft Defender SmartScreen preventing mailbagit-gui.exe from running due to an unsigned executable.]({{ site.baseurl }}/img/smart_screen.png)
diff --git a/docs/installing_mailbagit/pdf.md b/docs/installing_mailbagit/pdf.md
index 7b2aa82..9d6569f 100755
--- a/docs/installing_mailbagit/pdf.md
+++ b/docs/installing_mailbagit/pdf.md
@@ -3,7 +3,7 @@ layout: page
 title: PDF Derivatives
 permalink: /pdf/
 parent: Installing mailbagit
-nav_order: 3
+nav_order: 5
 ---
 
 # PDF derivatives
diff --git a/docs/installing_mailbagit/python.md b/docs/installing_mailbagit/python.md
index e4e392b..2e5f2e0 100755
--- a/docs/installing_mailbagit/python.md
+++ b/docs/installing_mailbagit/python.md
@@ -44,6 +44,8 @@ You should now be able install the PST dependencies with:
 pip install 'mailbagit[pst]'
 ```
 
+If you have trouble installing the PST dependencies on Windows, try using the [Windows executables]({{ site.baseurl }}/exe).
+
 ## Using the Mailbagit Graphical User Interface (GUI)
 
 To install the mailbagit GUI, in addition to `pip install mailbagit`, you need to run:
diff --git a/docs/using_mailbagit/mailbagit-gui.md b/docs/using_mailbagit/mailbagit-gui.md
index 0201aa5..87a1dbd 100755
--- a/docs/using_mailbagit/mailbagit-gui.md
+++ b/docs/using_mailbagit/mailbagit-gui.md
@@ -24,7 +24,9 @@ Instead, we plan to implement a "guided" command line option, which we hope may
 
 There is a [known issue](https://github.com/UAlbanyArchives/mailbagit/issues/155) where the GUI does not fully run on Windows using the `mailbagit-gui` command. It will boot and lets you enter options but then fails to run when you click "Start."
 
-If you are runnig it natively with python, you can also try creating a file called `mailbagit-gui.py` with the contents:
+The GUI will run on Windows without this issue using the `mailbagit-gui.exe` [executables]({{ site.baseurl }}/exe).
+
+If you are running it natively with python, you can also try creating a file called `mailbagit-gui.py` with the contents:
 
 ```
 from mailbagit import gui
diff --git a/dumpData.py b/dumpData.py
index 275db7f..2ad6d85 100644
--- a/dumpData.py
+++ b/dumpData.py
@@ -10,9 +10,10 @@
 
 for parser in parsers:
     args.input = parser
-    args.derivatives = ["example"]
+    args.derivatives = ["html"]
     args.dry_run = True
     args.mailbag_name = "test_data"
+    args.companion_files = False
 
     controller = Controller(args)
 
diff --git a/mailbagit/__init__.py b/mailbagit/__init__.py
index 561397e..699b21f 100644
--- a/mailbagit/__init__.py
+++ b/mailbagit/__init__.py
@@ -30,7 +30,7 @@
 # Formats and derivatives are loaded from:
 #   1. formats/derivatives directories inside the package (built-in)
 #   2. .mailbagit/{formats,derivatives} in user home directory
-#   3. {formats,dirivatives} subdirectories in plugin dir set in environment variable
+#   3. {formats,derivatives} subdirectories in plugin dir set in environment variable
 plugin_dirs = {"formats": [], "derivatives": []}
 for plugin_type, dirs in plugin_dirs.items():
     dirs.append(Path(f"~/.mailbagit/{plugin_type}").expanduser())
@@ -165,10 +165,13 @@
 # add mailbagit-specific optional args here
 mailbagit_options.add_argument("--css", help="Path to a CSS file to customize PDF derivatives.", nargs=None)
 mailbagit_options.add_argument(
-    "-c", "--compress", help="Compress the mailbag as ZIP, TAR, or TAR.GZ", nargs=None, choices=["tar", "zip", "tar.gz"]
+    "-c", "--compress", help="Compress the mailbag as ZIP, TAR, or TAR.GZ.", nargs=None, choices=["tar", "zip", "tar.gz"]
 )
 mailbagit_options.add_argument(
-    "-r", "--dry_run", help="A dry run performs a trial run with no changes made", default=False, action="store_true"
+    "-r", "--dry_run", help="A dry run performs a trial run with no changes made.", default=False, action="store_true"
+)
+mailbagit_options.add_argument(
+    "-f", "--companion_files", help="Will copy all files in the path provided to mailbagit in to a mailbag regardless of extention.", default=False, action="store_true"
 )
 # Yet-to-be-implemented:
 """
diff --git a/mailbagit/controller.py b/mailbagit/controller.py
index bdc56c3..2b554f2 100644
--- a/mailbagit/controller.py
+++ b/mailbagit/controller.py
@@ -9,7 +9,7 @@
 from dataclasses import dataclass, asdict, field, InitVar
 from pathlib import Path
 import os, shutil, glob
-import mailbagit.helper as helper
+import mailbagit.helper.controller as controller
 import mailbagit.globals as globals
 from time import time
 import uuid
@@ -158,7 +158,7 @@ def generate_mailbag(self):
             if len(message.Attachments) > 0:
                 if not os.path.isdir(attachments_dir) and not self.args.dry_run:
                     os.mkdir(attachments_dir)
-                helper.saveAttachmentOnDisk(self.args.dry_run, attachments_dir, message)
+                controller.writeAttachmentsToDisk(self.args.dry_run, attachments_dir, message)
 
             # Setting up CSV data
             # checking if the count of messages exceed 100000 and creating a new portion if it exceeds
@@ -194,7 +194,9 @@ def generate_mailbag(self):
             is_last = mailbag_message_id == total_messages
             if total_messages / 100 < 1 or is_first or is_last or mailbag_message_id % int(total_messages / 100) == 0:
                 print_End = "\n" if globals.log_level == "DEBUG" or is_last else "\r"
-                helper.progress(mailbag_message_id, total_messages, start_time, prefix="Progress ", suffix="Complete", print_End=print_End)
+                controller.progress(
+                    mailbag_message_id, total_messages, start_time, prefix="Progress ", suffix="Complete", print_End=print_End
+                )
 
         # End thread and server for WARC derivatives
         for d in derivatives:
@@ -225,7 +227,7 @@ def generate_mailbag(self):
                         writer.writerows(portion)
                         f.close()
 
-        helper.progressMessage("Writing CSVs...")
+        controller.progressMessage("Writing CSVs...")
         log.debug("Writing error.csv to " + str(error_dir))
         if len(error_csv) > 1:
             filename = os.path.join(error_dir, "error.csv")
@@ -243,11 +245,11 @@ def generate_mailbag(self):
             now = datetime.datetime.now()
             bag.info["Bagging-Timestamp"] = now.strftime("%Y-%m-%dT%H:%M:%S")
             bag.info["Bagging-Date"] = now.strftime("%Y-%m-%d")
-            helper.progressMessage("Generating manifests...")
+            controller.progressMessage("Generating manifests...")
             bag.save(manifests=True)
 
         if self.args.compress and not self.args.dry_run:
-            helper.progressMessage("Compressing mailbag...")
+            controller.progressMessage("Compressing mailbag...")
             log.info("Compressing Mailbag")
             compressionFormats = {"tar": "tar", "zip": "zip", "tar.gz": "gztar"}
             shutil.make_archive(mailbag_dir, compressionFormats[self.args.compress], mailbag_dir)
@@ -257,6 +259,6 @@ def generate_mailbag(self):
                 # Deleting the mailbag if compressed files are present
                 shutil.rmtree(mailbag_dir)
 
-        helper.progressMessage("Finished packaging mailbag.", print_End="\n")
+        controller.progressMessage("Finished packaging mailbag.", print_End="\n")
 
         return mail_account.messages()
diff --git a/mailbagit/derivative.py b/mailbagit/derivative.py
index 36450ba..244dbc3 100644
--- a/mailbagit/derivative.py
+++ b/mailbagit/derivative.py
@@ -49,11 +49,19 @@ def do_task_per_message(self, message):
         pass
 
 
-def import_derivatives(additional_dirs=None):
-    if not additional_dirs:
-        additional_dirs = []
-
-    dirs = [join(dirname(__file__), "derivatives"), *additional_dirs]
+def import_derivatives(dirs=None):
+    if not dirs:
+        dirs = []
+
+    # Due to pyinstaller not picking up dynamic imports inside the packed exe,
+    # all built-in derivatives MUST be manually imported here.
+    import mailbagit.derivatives.eml
+    import mailbagit.derivatives.html
+    import mailbagit.derivatives.mbox
+    import mailbagit.derivatives.pdf_chrome
+    import mailbagit.derivatives.pdf
+    import mailbagit.derivatives.txt
+    import mailbagit.derivatives.warc
 
     for derivatives_dir in dirs:
         if not exists(derivatives_dir):
diff --git a/mailbagit/derivatives/eml.py b/mailbagit/derivatives/eml.py
index 6bb1b73..b455df6 100644
--- a/mailbagit/derivatives/eml.py
+++ b/mailbagit/derivatives/eml.py
@@ -1,6 +1,7 @@
 # This is Eml derivative
 from os.path import join
-import mailbagit.helper as helper
+import mailbagit.helper.common as common
+import mailbagit.helper.derivative as derivative
 import os, glob
 import mailbox
 from mailbagit.email_account import EmailAccount
@@ -20,7 +21,7 @@
 class EmlDerivative(Derivative):
     derivative_name = "eml"
     derivative_format = "eml"
-    derivative_agent = email.__name__
+    derivative_agent = "email"
     derivative_agent_version = platform.python_version()
 
     def __init__(self, email_account, **kwargs):
@@ -50,7 +51,7 @@ def do_task_per_message(self, message):
             # Build msg
             if not message.Message and not message.Headers:
                 desc = "Unable to create EML as no body or headers present for " + str(message.Mailbag_Message_ID)
-                errors = helper.handle_error(errors, None, desc, "error")
+                errors = common.handle_error(errors, None, desc, "error")
             else:
                 fullObjectWrite = False
                 if message.Message:
@@ -73,9 +74,9 @@ def do_task_per_message(self, message):
                                 outfile.close()
                             fullObjectWrite = True
                     except Exception as e:
-                        helper.deleteFile(filename + ".eml")
+                        derivative.deleteFile(filename + ".eml")
                         desc = "Error writing full email object to EML, generating it from the model instead"
-                        errors = helper.handle_error(errors, e, desc, "warn")
+                        errors = common.handle_error(errors, e, desc, "warn")
                         fullObjectWrite = False
 
                 if fullObjectWrite == False:
@@ -95,7 +96,7 @@ def do_task_per_message(self, message):
                                 msg["X-Folder"] = message.Message_Path
                         except Exception as e:
                             desc = "Error writing headers for EML derivative"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
                         # Add message body
                         try:
@@ -109,28 +110,47 @@ def do_task_per_message(self, message):
                                 msg.attach(alt)
                             else:
                                 desc = "No body present for " + str(message.Mailbag_Message_ID) + ". Created EML without message body"
-                                errors = helper.handle_error(errors, None, desc, "warn")
+                                errors = common.handle_error(errors, None, desc, "warn")
                         except Exception as e:
                             desc = "Error writing body for EML derivative"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
+
+                        # Get list of <img> files that might reference attachments
+                        if message.HTML_Body:
+                            inline_files = derivative.inlineAttachments(message.HTML_Body, message.HTML_Encoding)
+                        else:
+                            inline_files = {}
 
                         # Attachments
                         try:
                             for attachment in message.Attachments:
                                 mimeType = attachment.MimeType
                                 if mimeType is None:
-                                    mimeType = "text/plain"
+                                    mimeType = "application/octet-stream"
                                     desc = "Mime type not found for attachment, set as " + mimeType
-                                    errors = helper.handle_error(errors, None, desc, "warn")
+                                    errors = common.handle_error(errors, None, desc, "error")
                                 mimeType = mimeType.split("/")
                                 part = MIMEBase(mimeType[0], mimeType[1])
                                 part.set_payload(attachment.File)
                                 encoders.encode_base64(part)
-                                part.add_header("Content-Disposition", "attachment", filename=attachment.Name)
+
+                                # Check if the attachment is inline in the HTML
+                                if attachment.Content_ID in inline_files.values():
+                                    content_disposition = "inline"
+                                    part.add_header("Content-ID", attachment.Content_ID)
+                                elif attachment.Name in inline_files.keys():
+                                    # If the source was MSG or PST, we generated attachment.Content_ID so it won't match
+                                    content_disposition = "inline"
+                                    part.add_header("Content-ID", "<" + inline_files[attachment.Name] + ">")
+                                else:
+                                    content_disposition = "attachment"
+                                    part.add_header("Content-ID", attachment.Content_ID)
+
+                                part.add_header("Content-Disposition", content_disposition, filename=attachment.Name)
                                 msg.attach(part)
                         except Exception as e:
                             desc = "Error writing attachment(s) to EML derivative"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
                         # Write generated EML to disk
                         log.debug("Writing generated EML to " + str(out_dir))
@@ -144,15 +164,15 @@ def do_task_per_message(self, message):
                                     outfile.close()
                         except Exception as e:
                             desc = "Error writing EML derivative"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
                     else:
                         desc = "Unable to create EML as no body or headers present for " + str(message.Mailbag_Message_ID)
-                        errors = helper.handle_error(errors, None, desc, "error")
+                        errors = common.handle_error(errors, None, desc, "error")
 
         except Exception as e:
             desc = "Error creating EML derivative"
-            errors = helper.handle_error(errors, e, desc)
+            errors = common.handle_error(errors, e, desc)
 
         message.Error.extend(errors["msg"])
         message.StackTrace.extend(errors["stack_trace"])
diff --git a/mailbagit/derivatives/html.py b/mailbagit/derivatives/html.py
index 794d0c5..095f978 100644
--- a/mailbagit/derivatives/html.py
+++ b/mailbagit/derivatives/html.py
@@ -1,6 +1,7 @@
 # Makes html derivatives just containing message bodies
 import os
-import mailbagit.helper as helper
+import mailbagit.helper.derivative as derivative
+import mailbagit.helper.common as common
 from structlog import get_logger
 
 log = get_logger()
@@ -43,16 +44,16 @@ def do_task_per_message(self, message):
 
             if message.HTML_Body is None and message.Text_Body is None:
                 desc = "No HTML or plain text body for " + str(message.Mailbag_Message_ID) + ", no HTML derivative created"
-                errors = helper.handle_error(errors, None, desc, "warn")
+                errors = common.handle_error(errors, None, desc, "warn")
             else:
                 log.debug("Writing html derivative to " + filename)
                 # Calling helper function to get formatted html
                 html_formatted = None
                 try:
-                    html_formatted, encoding = helper.htmlFormatting(message, self.args.css, headers=False)
+                    html_formatted, encoding = derivative.htmlFormatting(message, self.args.css, headers=False)
                 except Exception as e:
                     desc = "Error formatting HTML for HTML derivative"
-                    errors = helper.handle_error(errors, e, desc)
+                    errors = common.handle_error(errors, e, desc)
 
                 if not self.args.dry_run:
                     if html_formatted:
@@ -64,11 +65,11 @@ def do_task_per_message(self, message):
                                 f.close()
                         except Exception as e:
                             desc = "Error writing HTML derivative"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
         except Exception as e:
             desc = "Error creating HTML derivative"
-            errors = helper.handle_error(errors, e, desc)
+            errors = common.handle_error(errors, e, desc)
 
         message.Error.extend(errors["msg"])
         message.StackTrace.extend(errors["stack_trace"])
diff --git a/mailbagit/derivatives/mbox.py b/mailbagit/derivatives/mbox.py
index ccf5d29..6e7259b 100644
--- a/mailbagit/derivatives/mbox.py
+++ b/mailbagit/derivatives/mbox.py
@@ -1,6 +1,7 @@
 from structlog import get_logger
 import mailbox
 import os
+import mailbagit.helper.common as common
 from email.mime.multipart import MIMEMultipart
 from email.mime.base import MIMEBase
 from email.mime.text import MIMEText
@@ -51,14 +52,14 @@ def do_task_per_message(self, message):
                     filename = os.path.join(out_dir, os.path.basename(message.Derivatives_Path.strip(os.sep)) + ".mbox")
             except Exception as e:
                 desc = "Error setting up MBOX derivative"
-                errors = helper.handle_error(errors, e, desc)
+                errors = common.handle_error(errors, e, desc)
 
             log.debug("Writing message to " + str(filename))
             if self.args.dry_run:
                 # Checks for bodies to raise warnings even during a dry run
                 if not message.HTML_Body and not message.Text_Body:
                     desc = "No body present for " + str(message.Mailbag_Message_ID) + ". Added message to MBOX without message body"
-                    errors = helper.handle_error(errors, None, desc, "warn")
+                    errors = common.handle_error(errors, None, desc, "warn")
             else:
                 if not os.path.isdir(out_dir):
                     os.makedirs(out_dir)
@@ -88,7 +89,7 @@ def do_task_per_message(self, message):
                                 msg["X-Folder"] = message.Message_Path
                         except Exception as e:
                             desc = "Error writing headers for MBOX derivative"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
                         # Add message body
                         try:
@@ -106,10 +107,10 @@ def do_task_per_message(self, message):
                                     + str(message.Mailbag_Message_ID)
                                     + ". Added message to MBOX without message body"
                                 )
-                                errors = helper.handle_error(errors, None, desc, "warn")
+                                errors = common.handle_error(errors, None, desc, "warn")
                         except Exception as e:
                             desc = "Error writing body for MBOX derivative"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
                         # Attachments
                         try:
@@ -122,28 +123,41 @@ def do_task_per_message(self, message):
                                 part = MIMEBase(mimeType[0], mimeType[1])
                                 part.set_payload(attachment.File)
                                 encoders.encode_base64(part)
-                                part.add_header("Content-Disposition", "attachment", filename=attachment.Name)
+
+                                # Check if the attachment is inline in the HTML
+                                if attachment.Content_ID in inline_files.values():
+                                    content_disposition = "inline"
+                                    part.add_header("Content-ID", attachment.Content_ID)
+                                elif attachment.Name in inline_files.keys():
+                                    # If the source was MSG or PST, we generated attachment.Content_ID so it won't match
+                                    content_disposition = "inline"
+                                    part.add_header("Content-ID", "<" + inline_files[attachment.Name] + ">")
+                                else:
+                                    content_disposition = "attachment"
+                                    part.add_header("Content-ID", attachment.Content_ID)
+
+                                part.add_header("Content-Disposition", content_disposition, filename=attachment.Name)
                                 msg.attach(part)
                         except Exception as e:
                             desc = "Error writing attachment(s) to MBOX derivative"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
                         mbox.add(msg)
 
                     else:
                         desc = "Unable to create MBOX as no body or headers present for " + str(message.Mailbag_Message_ID)
-                        errors = helper.handle_error(errors, None, desc, "error")
+                        errors = common.handle_error(errors, None, desc, "error")
 
                     mbox.flush()
                     mbox.unlock()
 
                 except Exception as e:
                     desc = "Error writing MBOX derivative"
-                    errors = helper.handle_error(errors, e, desc)
+                    errors = common.handle_error(errors, e, desc)
 
         except Exception as e:
             desc = "Error creating MBOX derivative"
-            errors = helper.handle_error(errors, e, desc)
+            errors = common.handle_error(errors, e, desc)
 
         message.Error.extend(errors["msg"])
         message.StackTrace.extend(errors["stack_trace"])
diff --git a/mailbagit/derivatives/pdf.py b/mailbagit/derivatives/pdf.py
index a37fd90..847fe6d 100644
--- a/mailbagit/derivatives/pdf.py
+++ b/mailbagit/derivatives/pdf.py
@@ -3,7 +3,8 @@
 import distutils.spawn
 from mailbagit.derivative import Derivative
 from structlog import get_logger
-import mailbagit.helper as helper
+import mailbagit.helper.derivative as derivative
+import mailbagit.helper.common as common
 
 # only create format if pypff is successfully importable -
 # pst is not supported otherwise
@@ -62,15 +63,15 @@ def do_task_per_message(self, message):
 
                 if message.HTML_Body is None and message.Text_Body is None:
                     desc = "No HTML or plain text body for " + str(message.Mailbag_Message_ID) + ", no PDF derivative created"
-                    errors = helper.handle_error(errors, None, desc, "warn")
+                    errors = common.handle_error(errors, None, desc, "warn")
                 else:
                     log.debug("Writing HTML to " + str(html_name) + " and converting to " + str(pdf_name))
                     # Calling helper function to get formatted html
                     try:
-                        html_formatted, encoding = helper.htmlFormatting(message, self.args.css)
+                        html_formatted, encoding = derivative.htmlFormatting(message, self.args.css)
                     except Exception as e:
                         desc = "Error formatting HTML for PDF derivative"
-                        errors = helper.handle_error(errors, e, desc)
+                        errors = common.handle_error(errors, e, desc)
 
                     if not self.args.dry_run:
                         try:
@@ -95,19 +96,24 @@ def do_task_per_message(self, message):
                                 if stdout:
                                     log.debug("Output converting to " + str(message.Mailbag_Message_ID) + ".pdf: " + stdout.decode("utf-8"))
                                 if stderr:
-                                    desc = "Error converting to " + str(message.Mailbag_Message_ID) + ".pdf: " + stderr.decode("utf-8")
-                                    errors = helper.handle_error(errors, None, desc, "warn")
+                                    desc = (
+                                        "Error converting to "
+                                        + str(message.Mailbag_Message_ID)
+                                        + ".pdf: "
+                                        + stderr.decode("utf-8").replace("\r", "\n").replace("\n\n", "\n")
+                                    )
+                                    errors = common.handle_error(errors, None, desc, "warn")
                             # delete the HTML file
                             if os.path.isfile(pdf_name):
                                 os.remove(html_name)
 
                         except Exception as e:
                             desc = "Error writing HTML and converting to PDF derivative"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
             except Exception as e:
                 desc = "Error creating PDF derivative"
-                errors = helper.handle_error(errors, e, desc)
+                errors = common.handle_error(errors, e, desc)
 
             message.Error.extend(errors["msg"])
             message.StackTrace.extend(errors["stack_trace"])
diff --git a/mailbagit/derivatives/pdf-chrome.py b/mailbagit/derivatives/pdf_chrome.py
similarity index 91%
rename from mailbagit/derivatives/pdf-chrome.py
rename to mailbagit/derivatives/pdf_chrome.py
index 2766cdb..6e6fb37 100644
--- a/mailbagit/derivatives/pdf-chrome.py
+++ b/mailbagit/derivatives/pdf_chrome.py
@@ -3,7 +3,8 @@
 import distutils.spawn
 from mailbagit.derivative import Derivative
 from structlog import get_logger
-import mailbagit.helper as helper
+import mailbagit.helper.derivative as derivative
+import mailbagit.helper.common as common
 
 
 skip_registry = False
@@ -58,10 +59,10 @@ def do_task_per_message(self, message):
                     log.debug("Writing HTML to " + str(html_name) + " and converting to " + str(pdf_name))
                     # Calling helper function to get formatted html
                     try:
-                        html_formatted, encoding = helper.htmlFormatting(message, self.args.css)
+                        html_formatted, encoding = derivative.htmlFormatting(message, self.args.css)
                     except Exception as e:
                         desc = "Error formatting HTML for PDF derivative"
-                        errors = helper.handle_error(errors, e, desc)
+                        errors = common.handle_error(errors, e, desc)
 
                     if not self.args.dry_run:
                         try:
@@ -96,18 +97,18 @@ def do_task_per_message(self, message):
                                     log.warn("Output converting to " + str(message.Mailbag_Message_ID) + ".pdf: " + str(stdout))
                                 if stderr:
                                     desc = "Error converting to " + str(message.Mailbag_Message_ID) + ".pdf: " + str(stderr)
-                                    errors = helper.handle_error(errors, None, desc, "error")
+                                    errors = common.handle_error(errors, None, desc, "error")
                             # delete the HTML file
                             if os.path.isfile(pdf_name):
                                 os.remove(html_name)
 
                         except Exception as e:
                             desc = "Error writing HTML and converting to PDF derivative"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
             except Exception as e:
                 desc = "Error creating PDF derivative with chrome"
-                errors = helper.handle_error(errors, e, desc)
+                errors = common.handle_error(errors, e, desc)
 
             message.Error.extend(errors["msg"])
             message.StackTrace.extend(errors["stack_trace"])
diff --git a/mailbagit/derivatives/txt.py b/mailbagit/derivatives/txt.py
index a5bc260..8aef252 100644
--- a/mailbagit/derivatives/txt.py
+++ b/mailbagit/derivatives/txt.py
@@ -1,6 +1,6 @@
 # Makes txt file derivatives just containing message bodies
 import os
-import mailbagit.helper as helper
+import mailbagit.helper.common as common
 from structlog import get_logger
 
 log = get_logger()
@@ -40,7 +40,7 @@ def do_task_per_message(self, message):
 
             if message.Text_Body is None:
                 desc = "No plain text body for " + str(message.Mailbag_Message_ID) + ", no TXT derivative created"
-                errors = helper.handle_error(errors, None, desc, "warn")
+                errors = common.handle_error(errors, None, desc, "warn")
             else:
                 log.debug("Writing txt derivative to " + filename)
                 if not self.args.dry_run:
@@ -53,11 +53,11 @@ def do_task_per_message(self, message):
                                 f.close()
                     except Exception as e:
                         desc = "Error writing plain text body to file"
-                        errors = helper.handle_error(errors, e, desc)
+                        errors = common.handle_error(errors, e, desc)
 
         except Exception as e:
             desc = "Error creating plain text derivative"
-            errors = helper.handle_error(errors, e, desc)
+            errors = common.handle_error(errors, e, desc)
 
         message.Error.extend(errors["msg"])
         message.StackTrace.extend(errors["stack_trace"])
diff --git a/mailbagit/derivatives/warc.py b/mailbagit/derivatives/warc.py
index d09a655..e9a9e08 100644
--- a/mailbagit/derivatives/warc.py
+++ b/mailbagit/derivatives/warc.py
@@ -1,7 +1,7 @@
 import os
 import warcio
-from structlog import get_logger
-import mailbagit.helper as helper
+import mailbagit.helper.derivative as derivative
+import mailbagit.helper.common as common
 from warcio.capture_http import capture_http
 from warcio import WARCWriter
 from warcio.statusandheaders import StatusAndHeaders
@@ -10,6 +10,8 @@
 import http.server
 import socketserver
 
+from structlog import get_logger
+
 log = get_logger()
 
 from mailbagit.derivative import Derivative
@@ -39,7 +41,7 @@ def __init__(self, email_account, **kwargs):
         if not self.args.dry_run:
             os.makedirs(self.warc_dir)
 
-            self.server_thread = Thread(target=helper.startServer, args=(self.args.dry_run, self.httpd, self.port))
+            self.server_thread = Thread(target=derivative.startServer, args=(self.args.dry_run, self.httpd, self.port))
             # Make it a daemon so it will stop after ctrl+c
             self.server_thread.daemon = True
             self.server_thread.start()
@@ -49,7 +51,7 @@ def terminate(self):
         # Terminate the process
         try:
             if not self.args.dry_run:
-                helper.stopServer(self.args.dry_run, self.httpd[0])
+                derivative.stopServer(self.args.dry_run, self.httpd[0])
                 self.server_thread.join()
         except SystemExit:
             pass
@@ -71,7 +73,7 @@ def do_task_per_message(self, message):
 
             if message.HTML_Body is None and message.Text_Body is None:
                 desc = "No HTML or plain text body for " + str(message.Mailbag_Message_ID) + ", no WARC derivative created"
-                errors = helper.handle_error(errors, None, desc, "warn")
+                errors = common.handle_error(errors, None, desc, "warn")
             else:
                 out_dir = os.path.join(self.warc_dir, message.Derivatives_Path)
                 filename = os.path.join(out_dir, str(message.Mailbag_Message_ID) + ".warc.gz")
@@ -84,13 +86,13 @@ def do_task_per_message(self, message):
                     with open(filename, "wb") as output:
 
                         try:
-                            html_formatted, encoding = helper.htmlFormatting(message, self.args.css, headers=False)
+                            html_formatted, encoding = derivative.htmlFormatting(message, self.args.css, headers=False)
                             with open(self.tmp_file, "w", encoding="utf-8") as f:
                                 f.write(html_formatted)
                                 f.close()
                         except Exception as e:
                             desc = "Error formatting HTML for WARC derivative"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
                         writer = WARCWriter(output, gzip=True)
                         try:
@@ -103,7 +105,7 @@ def do_task_per_message(self, message):
 
                         except requests.exceptions.HTTPError as e:
                             desc = "Error requesting HTML for WARC derivative"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
                         try:
                             # get raw headers from urllib3
@@ -119,20 +121,20 @@ def do_task_per_message(self, message):
                             )
                         except Exception as e:
                             desc = "Error writing WARC headers"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
                         try:
                             writer.write_record(record)
                         except Exception as e:
                             desc = "Error writing WARC derivative"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
                     output.close()
-                    helper.deleteFile(self.tmp_file)
+                    derivative.deleteFile(self.tmp_file)
 
         except Exception as e:
             desc = "Error creating WARC derivative"
-            errors = helper.handle_error(errors, e, desc)
+            errors = common.handle_error(errors, e, desc)
 
         message.Error.extend(errors["msg"])
         message.StackTrace.extend(errors["stack_trace"])
diff --git a/mailbagit/email_account.py b/mailbagit/email_account.py
index 9b5aa58..864a65d 100644
--- a/mailbagit/email_account.py
+++ b/mailbagit/email_account.py
@@ -47,11 +47,16 @@ def messages(self):
         pass
 
 
-def import_formats(additional_dirs=None):
-    if not additional_dirs:
-        additional_dirs = []
-
-    dirs = [join(dirname(__file__), "formats"), *additional_dirs]
+def import_formats(dirs=None):
+    if not dirs:
+        dirs = []
+
+    # Since pyinstaller can't pick up dynamically install modules within the exe,
+    # all built-in formats must be manually imported
+    import mailbagit.formats.eml
+    import mailbagit.formats.mbox
+    import mailbagit.formats.msg
+    import mailbagit.formats.pst
 
     for formats_dir in reversed(dirs):
         if not exists(formats_dir):
diff --git a/mailbagit/formats/eml.py b/mailbagit/formats/eml.py
index 829718d..8c405cc 100644
--- a/mailbagit/formats/eml.py
+++ b/mailbagit/formats/eml.py
@@ -1,14 +1,14 @@
 import datetime
 import json
-from os.path import join
-import mailbagit.helper as helper
+from pathlib import Path
+import mailbagit.helper.format as format
+import mailbagit.helper.common as common
 from structlog import get_logger
 from email import parser
 from mailbagit.email_account import EmailAccount
 from mailbagit.models import Email, Attachment
 import email
 import os
-from email import policy
 import platform
 
 log = get_logger()
@@ -29,6 +29,7 @@ def __init__(self, target_account, args, **kwargs):
         self.path = target_account
         self.dry_run = args.dry_run
         self.mailbag_name = args.mailbag_name
+        self.companion_files = args.companion_files
         self.iteration_only = False
         log.info("Reading : ", Path=self.path)
 
@@ -38,10 +39,17 @@ def account_data(self):
     def messages(self):
 
         fileList = []
+        companion_files = []
         for root, dirs, files in os.walk(self.path):
             for file in files:
-                if file.lower().endswith("." + self.format_name):
-                    fileList.append(os.path.join(root, file))
+                mailbag_path = os.path.join(self.path, self.mailbag_name) + os.sep
+                fileRoot = root + os.sep
+                # don't count the newly-created mailbag
+                if not fileRoot.startswith(mailbag_path):
+                    if file.lower().endswith("." + self.format_name):
+                        fileList.append(os.path.join(root, file))
+                    elif self.companion_files:
+                        companion_files.append(os.path.join(root, file))
 
         for filePath in fileList:
 
@@ -49,7 +57,7 @@ def messages(self):
                 yield None
                 continue
 
-            originalFile = helper.relativePath(self.path, filePath)
+            originalFile = Path(format.relativePath(self.path, filePath)).as_posix()
 
             attachments = []
             errors = {}
@@ -57,7 +65,7 @@ def messages(self):
             errors["stack_trace"] = []
             try:
                 with open(filePath, "rb") as f:
-                    msg = email.message_from_binary_file(f, policy=policy.default)
+                    msg = email.message_from_binary_file(f, policy=email.policy.default)
 
                     try:
                         # Parse message bodies
@@ -68,33 +76,35 @@ def messages(self):
                         bodies["text_encoding"] = None
                         if msg.is_multipart():
                             for part in msg.walk():
-                                bodies, attachments, errors = helper.parse_part(part, bodies, attachments, errors)
+                                bodies, attachments, errors = format.parse_part(part, bodies, attachments, errors)
                         else:
-                            bodies, attachments, errors = helper.parse_part(msg, bodies, attachments, errors)
+                            bodies, attachments, errors = format.parse_part(msg, bodies, attachments, errors)
 
                     except Exception as e:
                         desc = "Error parsing message parts"
-                        errors = helper.handle_error(errors, e, desc)
+                        errors = common.handle_error(errors, e, desc)
 
                     # Look for message arrangement
                     try:
-                        messagePath = helper.messagePath(msg)
+                        messagePath = Path(format.messagePath(msg)).as_posix()
+                        if messagePath == ".":
+                            messagePath = ""
                         unsafePath = os.path.join(os.path.dirname(originalFile), messagePath)
-                        derivativesPath = helper.normalizePath(unsafePath)
+                        derivativesPath = format.normalizePath(unsafePath)
                     except Exception as e:
                         desc = "Error reading message path from headers"
-                        errors = helper.handle_error(errors, e, desc)
+                        errors = common.handle_error(errors, e, desc)
 
                     message = Email(
                         Error=errors["msg"],
-                        Message_ID=helper.parse_header(msg["message-id"]),
+                        Message_ID=format.parse_header(msg["message-id"]),
                         Original_File=originalFile,
                         Message_Path=messagePath,
                         Derivatives_Path=derivativesPath,
-                        Date=helper.parse_header(msg["date"]),
-                        From=helper.parse_header(msg["from"]),
-                        To=helper.parse_header(msg["to"]),
-                        Subject=helper.parse_header(msg["subject"]),
+                        Date=format.parse_header(msg["date"]),
+                        From=format.parse_header(msg["from"]),
+                        To=format.parse_header(msg["to"]),
+                        Subject=format.parse_header(msg["subject"]),
                         Content_Type=msg.get_content_type(),
                         Headers=msg,
                         HTML_Body=bodies["html_body"],
@@ -108,9 +118,14 @@ def messages(self):
 
             except (email.errors.MessageParseError, Exception) as e:
                 desc = "Error parsing message"
-                errors = helper.handle_error(errors, e, desc)
+                errors = common.handle_error(errors, e, desc)
                 message = Email(Error=errors["msg"], StackTrace=errors["stack_trace"])
 
             # Move EML to new mailbag directory structure
             yield message
-            new_path = helper.moveWithDirectoryStructure(self.dry_run, self.path, self.mailbag_name, self.format_name, filePath)
+            new_path = format.moveWithDirectoryStructure(self.dry_run, self.path, self.mailbag_name, self.format_name, filePath)
+
+        if self.companion_files:
+            # Move all files into mailbag directory structure
+            for companion_file in companion_files:
+                new_path = format.moveWithDirectoryStructure(self.dry_run, self.path, self.mailbag_name, self.format_name, companion_file)
diff --git a/mailbagit/formats/mbox.py b/mailbagit/formats/mbox.py
index c2a3829..b47f91e 100644
--- a/mailbagit/formats/mbox.py
+++ b/mailbagit/formats/mbox.py
@@ -8,7 +8,8 @@
 
 from mailbagit.email_account import EmailAccount
 from mailbagit.models import Email, Attachment
-import mailbagit.helper as helper
+import mailbagit.helper.format as format
+import mailbagit.helper.common as common
 import platform
 
 log = get_logger()
@@ -29,6 +30,7 @@ def __init__(self, target_account, args, **kwargs):
         self.path = target_account
         self.dry_run = args.dry_run
         self.mailbag_name = args.mailbag_name
+        self.companion_files = args.companion_files
         self.iteration_only = False
         log.info("Reading : ", Path=self.path)
 
@@ -37,6 +39,7 @@ def account_data(self):
 
     def messages(self):
 
+        companion_files = []
         if os.path.isfile(self.path):
             parent_dir = os.path.dirname(self.path)
             fileList = [self.path]
@@ -45,11 +48,22 @@ def messages(self):
             fileList = []
             for root, dirs, files in os.walk(self.path):
                 for file in files:
-                    if file.lower().endswith("." + self.format_name):
-                        fileList.append(os.path.join(root, file))
+                    mailbag_path = os.path.join(self.path, self.mailbag_name) + os.sep
+                    fileRoot = root + os.sep
+                    # don't count the newly-created mailbag
+                    if not fileRoot.startswith(mailbag_path):
+                        if file.lower().endswith("." + self.format_name):
+                            fileList.append(os.path.join(root, file))
+                        elif self.companion_files:
+                            companion_files.append(os.path.join(root, file))
 
         for filePath in fileList:
-            originalFile = helper.relativePath(self.path, filePath)
+            rel_path = format.relativePath(self.path, filePath)
+            if len(rel_path) < 1:
+                originalFile = Path(filePath).name
+            else:
+                originalFile = Path(os.path.normpath(rel_path)).as_posix()
+            # original file is now the relative path to the MBOX from the provided path
 
             data = mailbox.mbox(filePath)
             for mail in data.itervalues():
@@ -62,6 +76,7 @@ def messages(self):
                 errors = {}
                 errors["msg"] = []
                 errors["stack_trace"] = []
+
                 try:
                     mailObject = email.message_from_bytes(mail.as_bytes(), policy=email.policy.default)
 
@@ -74,37 +89,35 @@ def messages(self):
                         bodies["text_encoding"] = None
                         if mailObject.is_multipart():
                             for part in mailObject.walk():
-                                bodies, attachments, errors = helper.parse_part(part, bodies, attachments, errors)
+                                bodies, attachments, errors = format.parse_part(part, bodies, attachments, errors)
                         else:
-                            bodies, attachments, errors = helper.parse_part(mailObject, bodies, attachments, errors)
+                            bodies, attachments, errors = format.parse_part(mailObject, bodies, attachments, errors)
                     except Exception as e:
                         desc = "Error parsing message parts"
-                        errors = helper.handle_error(errors, e, desc)
+                        errors = common.handle_error(errors, e, desc)
 
                     # Look for message arrangement
                     try:
-                        messagePath = helper.messagePath(mailObject)
-                        if len(messagePath) > 0:
-                            unsafePath = os.path.join(os.path.splitext(originalFile)[0], messagePath)
-                        else:
-                            unsafePath = os.path.splitext(originalFile)[0]
-                        derivativesPath = helper.normalizePath(unsafePath)
+                        messagePath = Path(format.messagePath(mailObject)).as_posix()
+                        if messagePath == ".":
+                            messagePath = ""
+                        derivativesPath = Path(os.path.splitext(originalFile)[0], format.normalizePath(messagePath)).as_posix()
                     except Exception as e:
                         desc = "Error reading message path from headers"
-                        errors = helper.handle_error(errors, e, desc)
+                        errors = common.handle_error(errors, e, desc)
 
                     message = Email(
                         Error=errors["msg"],
-                        Message_ID=helper.parse_header(mail["Message-ID"]),
+                        Message_ID=format.parse_header(mail["Message-ID"]),
                         Original_File=originalFile,
                         Message_Path=messagePath,
                         Derivatives_Path=derivativesPath,
-                        Date=helper.parse_header(mail["Date"]),
-                        From=helper.parse_header(mail["From"]),
-                        To=helper.parse_header(mail["To"]),
-                        Cc=helper.parse_header(mail["Cc"]),
-                        Bcc=helper.parse_header(mail["Bcc"]),
-                        Subject=helper.parse_header(mail["Subject"]),
+                        Date=format.parse_header(mail["Date"]),
+                        From=format.parse_header(mail["From"]),
+                        To=format.parse_header(mail["To"]),
+                        Cc=format.parse_header(mail["Cc"]),
+                        Bcc=format.parse_header(mail["Bcc"]),
+                        Subject=format.parse_header(mail["Subject"]),
                         Content_Type=mailObject.get_content_type(),
                         Headers=mail,
                         HTML_Body=bodies["html_body"],
@@ -117,7 +130,7 @@ def messages(self):
                     )
                 except (email.errors.MessageParseError, Exception) as e:
                     desc = "Error parsing message"
-                    errors = helper.handle_error(errors, e, desc)
+                    errors = common.handle_error(errors, e, desc)
                     message = Email(Error=errors["msg"], StackTrace=errors["stack_trace"])
 
                 yield message
@@ -126,4 +139,9 @@ def messages(self):
             data.close()
             # Move MBOX to new mailbag directory structure
             if not self.iteration_only:
-                new_path = helper.moveWithDirectoryStructure(self.dry_run, parent_dir, self.mailbag_name, self.format_name, filePath)
+                new_path = format.moveWithDirectoryStructure(self.dry_run, parent_dir, self.mailbag_name, self.format_name, filePath)
+
+        if self.companion_files:
+            # Move all files into mailbag directory structure
+            for companion_file in companion_files:
+                new_path = format.moveWithDirectoryStructure(self.dry_run, self.path, self.mailbag_name, self.format_name, companion_file)
diff --git a/mailbagit/formats/msg.py b/mailbagit/formats/msg.py
index bdf72a4..d5455f4 100644
--- a/mailbagit/formats/msg.py
+++ b/mailbagit/formats/msg.py
@@ -1,14 +1,16 @@
 import extract_msg
 import os
+from pathlib import Path
 from email import parser
 from structlog import get_logger
-from RTFDE.deencapsulate import DeEncapsulator
 import email.errors
 from mailbagit.email_account import EmailAccount
 from mailbagit.models import Email, Attachment
-import mailbagit.helper as helper
+import mailbagit.helper.format as format
+import mailbagit.helper.common as common
 import mailbagit.globals as globals
 from extract_msg import attachment
+import uuid
 
 log = get_logger()
 
@@ -28,6 +30,7 @@ def __init__(self, target_account, args, **kwargs):
         self.path = target_account
         self.dry_run = args.dry_run
         self.mailbag_name = args.mailbag_name
+        self.companion_files = args.companion_files
         self.iteration_only = False
         log.info("Reading :", Path=self.path)
 
@@ -36,18 +39,26 @@ def account_data(self):
 
     def messages(self):
         fileList = []
+        companion_files = []
         for root, dirs, files in os.walk(self.path):
             for file in files:
-                if file.lower().endswith("." + self.format_name):
-                    fileList.append(os.path.join(root, file))
+                mailbag_path = os.path.join(self.path, self.mailbag_name) + os.sep
+                fileRoot = root + os.sep
+                # don't count the newly-created mailbag
+                if not fileRoot.startswith(mailbag_path):
+                    if file.lower().endswith("." + self.format_name):
+                        fileList.append(os.path.join(root, file))
+                    elif self.companion_files:
+                        companion_files.append(os.path.join(root, file))
 
         for filePath in fileList:
+            # Parse email matching the input file extension
 
             if self.iteration_only:
                 yield None
                 continue
 
-            originalFile = helper.relativePath(self.path, filePath)
+            originalFile = Path(format.relativePath(self.path, filePath)).as_posix()
 
             attachments = []
             errors = {}
@@ -70,16 +81,18 @@ def messages(self):
                         text_encoding = mail.stringEncoding
                 except Exception as e:
                     desc = "Error parsing message body"
-                    errors = helper.handle_error(errors, e, desc)
+                    errors = common.handle_error(errors, e, desc)
 
                 # Look for message arrangement
                 try:
-                    messagePath = helper.messagePath(mail.header)
+                    messagePath = Path(format.messagePath(mail.header)).as_posix()
+                    if messagePath == ".":
+                        messagePath = ""
                     unsafePath = os.path.join(os.path.dirname(originalFile), messagePath)
-                    derivativesPath = helper.normalizePath(unsafePath)
+                    derivativesPath = Path(format.normalizePath(unsafePath)).as_posix()
                 except Exception as e:
                     desc = "Error reading message path from headers"
-                    errors = helper.handle_error(errors, e, desc)
+                    errors = common.handle_error(errors, e, desc)
 
                 try:
                     for mailAttachment in mail.attachments:
@@ -90,21 +103,42 @@ def messages(self):
                         elif mailAttachment.shortFilename:
                             attachmentName = mailAttachment.shortFilename
                         else:
-                            attachmentName = str(len(attachments))
-                            desc = "No filename found for attachment " + attachmentName + " for message " + str(message.Mailbag_Message_ID)
-                            errors = helper.handle_error(errors, e, desc)
+                            attachmentName = None
+                            desc = "No filename found for attachment, integer will be used instead"
+                            errors = common.handle_error(errors, None, desc)
+
+                        # Handle attachments.csv conflict
+                        # helper.controller.writeAttachmentsToDisk() handles this
+                        if attachmentName:
+                            if attachmentName.lower() == "attachments.csv":
+                                desc = "attachment " + attachmentName + " will be renamed to avoid filename conflict with mailbag spec"
+                                errors = common.handle_error(errors, None, desc, "warn")
+
+                        # Try to get the mime, guess it if this doesn't work
+                        mime = None
+                        try:
+                            mime = mailAttachment._ensureSet("_contentType", "__substg1.0_370e")
+                        except Exception as e:
+                            desc = "Error reading mime type, guessing it instead"
+                            errors = common.handle_error(errors, e, desc, "warn")
+                        if mime is None:
+                            mime = format.guessMimeType(attachmentName)
+
+                        # MSGs don't seem to have a reliable content ID so we make one since emails may have multiple attachments with the same filename
+                        contentID = uuid.uuid4().hex
 
                         attachment = Attachment(
                             Name=attachmentName,
                             File=mailAttachment.data,
-                            MimeType=helper.guessMimeType(attachmentName),
+                            MimeType=mime,
+                            Content_ID=contentID,
                         )
                         attachments.append(attachment)
 
                 except Exception as e:
                     desc = "Error parsing attachments"
-                    errors = helper.handle_error(errors, e, desc)
-                print(mail.subject)
+                    errors = common.handle_error(errors, e, desc)
+
                 message = Email(
                     Error=errors["msg"],
                     Message_ID=mail.messageId.strip(),
@@ -134,11 +168,16 @@ def messages(self):
 
             except (email.errors.MessageParseError, Exception) as e:
                 desc = "Error parsing message"
-                errors = helper.handle_error(errors, e, desc)
+                errors = common.handle_error(errors, e, desc)
                 message = Email(Error=errors["msg"], StackTrace=errors["stack_trace"])
                 # Make sure the MSG file is closed
                 mail.close()
 
             # Move MSG to new mailbag directory structure
-            new_path = helper.moveWithDirectoryStructure(self.dry_run, self.path, self.mailbag_name, self.format_name, filePath)
+            new_path = format.moveWithDirectoryStructure(self.dry_run, self.path, self.mailbag_name, self.format_name, filePath)
             yield message
+
+        if self.companion_files:
+            # Move all files into mailbag directory structure
+            for companion_file in companion_files:
+                new_path = format.moveWithDirectoryStructure(self.dry_run, self.path, self.mailbag_name, self.format_name, companion_file)
diff --git a/mailbagit/formats/pst.py b/mailbagit/formats/pst.py
index 1ce6cf8..25395d9 100644
--- a/mailbagit/formats/pst.py
+++ b/mailbagit/formats/pst.py
@@ -7,7 +7,9 @@
 from email import parser
 from mailbagit.email_account import EmailAccount
 from mailbagit.models import Email, Attachment
-import mailbagit.helper as helper
+import mailbagit.helper.format as format
+import mailbagit.helper.common as common
+import uuid
 
 # only create format if pypff is successfully importable -
 # pst is not supported otherwise
@@ -38,6 +40,7 @@ def __init__(self, target_account, args, **kwargs):
             self.path = target_account
             self.dry_run = args.dry_run
             self.mailbag_name = args.mailbag_name
+            self.companion_files = args.companion_files
             self.iteration_only = False
             log.info("Reading :", Path=self.path)
 
@@ -47,10 +50,9 @@ def account_data(self):
         def folders(self, folder, path, originalFile):
             # recursive function that calls itself on any subfolders and
             # returns a generator of messages
-            # path is a list that you can create the filepath with os.path.join()
+            # path is the email folder path of the message, separated by "/"
             if folder.number_of_sub_messages:
                 log.debug("Reading folder: " + folder.name)
-                path.append(folder.name)
                 for index in range(folder.number_of_sub_messages):
 
                     if self.iteration_only:
@@ -68,7 +70,7 @@ def folders(self, folder, path, originalFile):
                             headers = headerParser.parsestr(messageObj.transport_headers)
                         except Exception as e:
                             desc = "Error parsing message body"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
                         try:
                             # Parse message bodies
@@ -89,44 +91,41 @@ def folders(self, folder, path, originalFile):
                                         if entry.data:
                                             value = entry.get_data_as_integer()
                                             # Use the extract_msg code page in constants.py
-                                            encodings["PidTagInternetCodepage"] = CODE_PAGES[value]
+                                            encodings[1] = {"name": CODE_PAGES[value], "label": "PidTagInternetCodepage"}
                                     if entry.entry_type == LIBPFF_ENTRY_TYPE_MESSAGE_CODEPAGE:
                                         if entry.data:
                                             value = entry.get_data_as_integer()
                                             # Use the extract_msg code page in constants.py
-                                            encodings["PidTagMessageCodepage"] = CODE_PAGES[value]
+                                            encodings[2] = {"name": CODE_PAGES[value], "label": "PidTagMessageCodepage"}
 
                             if messageObj.html_body:
-                                if encodings["PidTagInternetCodepage"]:
-                                    html_encoding = encodings["PidTagInternetCodepage"]
-                                elif encodings["PidTagMessageCodepage"]:
-                                    html_encoding = encodings["PidTagMessageCodepage"]
-                                else:
-                                    html_encoding = chardet.detect(messageObj.html_body)["encoding"]
-                                html_body = messageObj.html_body.decode(html_encoding)
+                                html_body, html_encoding, errors = format.safely_decode("HTML", messageObj.html_body, encodings, errors)
                             if messageObj.plain_text_body:
-                                if encodings["PidTagInternetCodepage"]:
-                                    text_encoding = encodings["PidTagInternetCodepage"]
-                                elif encodings["PidTagMessageCodepage"]:
-                                    text_encoding = encodings["PidTagMessageCodepage"]
-                                else:
-                                    text_encoding = chardet.detect(messageObj.plain_text_body)["encoding"]
-                                text_encoding = chardet.detect(messageObj.plain_text_body)["encoding"]
-                                text_body = messageObj.plain_text_body.decode(text_encoding)
+                                encodings[len(encodings.keys()) + 1] = {
+                                    "name": "utf-8",
+                                    "label": "manual",
+                                }
+                                encodings[len(encodings.keys()) + 2] = {
+                                    "name": chardet.detect(messageObj.plain_text_body)["encoding"],
+                                    "label": "detected",
+                                }
+                                text_body, text_encoding, errors = format.safely_decode(
+                                    "plain text", messageObj.plain_text_body, encodings, errors
+                                )
 
                         except Exception as e:
                             desc = "Error parsing message body"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
                         # Build message and derivatives paths
                         try:
-                            messagePath = os.path.join(os.path.splitext(originalFile)[0], *path)
+                            messagePath = path
                             if len(messagePath) > 0:
                                 messagePath = Path(messagePath).as_posix()
-                            derivativesPath = helper.normalizePath(messagePath)
+                            derivativesPath = Path(os.path.splitext(originalFile)[0], format.normalizePath(messagePath)).as_posix()
                         except Exception as e:
                             desc = "Error reading message path"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
                         try:
                             total_attachment_size_bytes = 0
@@ -160,46 +159,59 @@ def folders(self, folder, path, originalFile):
                                     elif len(attachmentShort) > 0:
                                         attachmentName = attachmentShort
                                     else:
-                                        print(message.Mailbag_Message_ID)
-                                        raise ValueError("No attachment name found.")
+                                        attachmentName = None
+                                        desc = "No filename found for attachment, integer will be used instead"
+                                        errors = common.handle_error(errors, None, desc)
+
+                                    # Handle attachments.csv conflict
+                                    # helper.controller.writeAttachmentsToDisk() handles this
+                                    if attachmentName:
+                                        if attachmentName.lower() == "attachments.csv":
+                                            desc = (
+                                                "attachment "
+                                                + attachmentName
+                                                + " will be renamed to avoid filename conflict with mailbag spec"
+                                            )
+                                            errors = common.handle_error(errors, None, desc, "warn")
 
                                     # Guess the mime if we can't find it
                                     if mime is None:
-                                        mime = helper.guessMimeType(attachmentName)
+                                        mime = format.guessMimeType(attachmentName)
+
+                                    # MSGs don't seem to have a reliable content ID so we make one since emails may have multiple attachments with the same filename
+                                    contentID = uuid.uuid4().hex
 
                                 except Exception as e:
                                     attachmentName = str(len(attachments))
                                     desc = (
-                                        "No filename found for attachment "
-                                        + attachmentName
-                                        + " for message "
-                                        + str(message.Mailbag_Message_ID)
+                                        "No filename found for attachment " + attachmentName + " for message " + str(headers["Message-ID"])
                                     )
-                                    errors = helper.handle_error(errors, e, desc)
+                                    errors = common.handle_error(errors, e, desc)
 
                                 attachment = Attachment(
                                     Name=attachmentName,
                                     File=attachment_content,
                                     MimeType=mime,
+                                    Content_ID=contentID,
                                 )
                                 attachments.append(attachment)
 
                         except Exception as e:
                             desc = "Error parsing attachments"
-                            errors = helper.handle_error(errors, e, desc)
+                            errors = common.handle_error(errors, e, desc)
 
                         message = Email(
                             Error=errors["msg"],
-                            Message_ID=helper.parse_header(headers["Message-ID"]),
+                            Message_ID=format.parse_header(headers["Message-ID"]),
                             Original_File=originalFile,
                             Message_Path=messagePath,
                             Derivatives_Path=derivativesPath,
-                            Date=helper.parse_header(headers["Date"]),
-                            From=helper.parse_header(headers["From"]),
-                            To=helper.parse_header(headers["To"]),
-                            Cc=helper.parse_header(headers["Cc"]),
-                            Bcc=helper.parse_header(headers["Bcc"]),
-                            Subject=helper.parse_header(headers["Subject"]),
+                            Date=format.parse_header(headers["Date"]),
+                            From=format.parse_header(headers["From"]),
+                            To=format.parse_header(headers["To"]),
+                            Cc=format.parse_header(headers["Cc"]),
+                            Bcc=format.parse_header(headers["Bcc"]),
+                            Subject=format.parse_header(headers["Subject"]),
                             Content_Type=headers.get_content_type(),
                             Headers=headers,
                             HTML_Body=html_body,
@@ -213,7 +225,7 @@ def folders(self, folder, path, originalFile):
 
                     except (Exception) as e:
                         desc = "Error parsing message"
-                        errors = helper.handle_error(errors, e, desc)
+                        errors = common.handle_error(errors, e, desc)
                         message = Email(Error=errors["msg"], StackTrace=errors["stack_trace"])
 
                     yield message
@@ -222,12 +234,15 @@ def folders(self, folder, path, originalFile):
             if folder.number_of_sub_folders:
                 for folder_index in range(folder.number_of_sub_folders):
                     subfolder = folder.get_sub_folder(folder_index)
-                    yield from self.folders(subfolder, path, originalFile)
-            else:
-                # gotta return empty directory to controller somehow
-                log.warn("Empty folder " + folder.name + " not handled.")
+                    yield from self.folders(subfolder, path + "/" + subfolder.name, originalFile)
+            # else:
+            # if not self.iteration_only:
+            # This is an email folder that does not contain any messages.
+            # We are currently ignoring these per issue #117
+            # log.warn("Empty folder " + folder.name + " not handled.")
 
         def messages(self):
+            companion_files = []
             if os.path.isfile(self.path):
                 parent_dir = os.path.dirname(self.path)
                 fileList = [self.path]
@@ -236,15 +251,22 @@ def messages(self):
                 fileList = []
                 for root, dirs, files in os.walk(self.path):
                     for file in files:
-                        if file.lower().endswith("." + self.format_name):
-                            fileList.append(os.path.join(root, file))
+                        mailbag_path = os.path.join(self.path, self.mailbag_name) + os.sep
+                        fileRoot = root + os.sep
+                        # don't count the newly-created mailbag
+                        if not fileRoot.startswith(mailbag_path):
+                            if file.lower().endswith("." + self.format_name):
+                                fileList.append(os.path.join(root, file))
+                            elif self.companion_files:
+                                companion_files.append(os.path.join(root, file))
 
             for filePath in fileList:
-                originalFile = helper.relativePath(self.path, filePath)
-                if len(originalFile) < 1:
-                    pathList = []
+                rel_path = format.relativePath(self.path, filePath)  # returns "" when path is a file
+                if len(rel_path) < 1:
+                    originalFile = Path(filePath).name
                 else:
-                    pathList = os.path.normpath(originalFile).split(os.sep)
+                    originalFile = Path(os.path.normpath(rel_path)).as_posix()
+                # original file is now the relative path to the PST from the provided path
 
                 pst = pypff.file()
                 pst.open(filePath)
@@ -252,18 +274,27 @@ def messages(self):
                 for folder in root.sub_folders:
                     if folder.number_of_sub_folders:
                         # call recursive function to parse email folder
-                        yield from self.folders(folder, pathList, os.path.basename(filePath))
-                    else:
-                        # gotta return empty directory to controller somehow
-                        log.warn("Empty folder " + folder.name + " not handled.")
+                        yield from self.folders(folder, folder.name, originalFile)
+                    # else:
+                    # if not self.iteration_only:
+                    # This is an email folder that does not contain any messages.
+                    # We are currently ignoring these per issue #117
+                    # log.warn("Empty folder " + folder.name + " not handled.")
                 pst.close()
 
                 # Move PST to new mailbag directory structure
                 if not self.iteration_only:
-                    new_path = helper.moveWithDirectoryStructure(
+                    new_path = format.moveWithDirectoryStructure(
                         self.dry_run,
                         parent_dir,
                         self.mailbag_name,
                         self.format_name,
                         filePath,
                     )
+
+            if self.companion_files:
+                # Move all files into mailbag directory structure
+                for companion_file in companion_files:
+                    new_path = format.moveWithDirectoryStructure(
+                        self.dry_run, self.path, self.mailbag_name, self.format_name, companion_file
+                    )
diff --git a/mailbagit/helper.py b/mailbagit/helper.py
deleted file mode 100644
index 2557d51..0000000
--- a/mailbagit/helper.py
+++ /dev/null
@@ -1,735 +0,0 @@
-import base64
-import urllib.parse
-from pathlib import Path
-import os, shutil, glob
-import datetime
-from time import time
-from bs4 import BeautifulSoup, Doctype
-from structlog import get_logger
-import mailbagit.globals as globals
-from mailbagit.models import Attachment
-import mimetypes
-import traceback
-import chardet, codecs
-from email.header import Header, decode_header, make_header
-import http.server
-import socketserver
-import html
-
-log = get_logger()
-
-
-def moveFile(dry_run, oldPath, newPath):
-    os.makedirs(os.path.dirname(newPath), exist_ok=True)
-    try:
-        log.debug("from: " + str(oldPath))
-        log.debug("to: " + str(newPath))
-        shutil.move(oldPath, newPath)
-    except IOError as e:
-        log.error("Unable to move file. %s" % e)
-
-
-def progress(current, total, start_time, prefix="", suffix="", decimals=1, length=100, fill="█", print_End="\r"):
-    """
-    Call in a loop to create terminal progress bar
-
-    Parameters:
-        current (int): current progress
-        total (int): total iterations
-        start_time (float): start time
-        prefix (String): prefix string
-        suffix (String): suffix string
-        decimals (int): positive number of decimals in percent complete
-        length (int): character length of bar (Int)
-        fill (String): bar fill character (Str)
-        printEnd (String): end character (e.g. "\r", "\r\n")
-    """
-
-    time_spent = time() - start_time
-    remaining_time = round(time_spent * (total / current - 1), 2)
-    e = datetime.datetime.now()
-    percent = ("{0:." + str(decimals) + "f}").format(100 * (current / float(total)))
-    # filledLength = int(length * current // total)
-    style = globals.style
-    # bar = fill * filledLength + '-' * (length - filledLength)
-
-    dt = f"{e.year}-{e.month:02d}-{e.day:02d} {e.hour:02d}:{e.minute:02d}.{e.second:02d}"
-    message_type = f'[{style["cy"][0]}{prefix}{style["b"][1]}]'
-    # deco_prefix = f'{style["b"][0]}{prefix}{style["b"][1]}'
-    # statusBar = f'|{bar}| {percent}% [{current}MB out of {total}MB] {suffix}'
-    status = f"{percent}% [{current} / {total} messages] {remaining_time}s remaining"
-
-    print(f"\r{dt} {message_type} {status}", end=print_End)
-
-
-def progressMessage(msg, print_End="\r"):
-    """
-    Shows a message as progress. Useful since it make take awhile to save large bags
-    even after all messages have been processed.
-
-    Parameters:
-        msg (String): A message to show as progress
-    """
-    e = datetime.datetime.now()
-    style = globals.style
-    dt = f"{e.year}-{e.month:02d}-{e.day:02d} {e.hour:02d}:{e.minute:02d}.{e.second:02d}"
-    message_type = f'[{style["cy"][0]}{"Progress "}{style["b"][1]}]'
-    print(f"\r{dt} {message_type} {msg}", end=print_End)
-
-
-def processedFile(filePath):
-    """
-    Add size of this file to counter
-
-    Parameters:
-        filePath (String): Location to find the file
-    """
-
-    increment = os.path.getsize(filePath) / (1024**2)
-    globals.processedSize += increment
-    globals.processedSize = round(globals.processedSize, 2)
-
-
-def getDirectorySize(directory, format):
-    """
-    Calculate total size of relevant files in the directory
-
-    Parameters:
-        directory (String): Location to find files
-        format (String): format of the mail type
-
-    Returns:
-        String: emailFolder
-    """
-
-    filePath = os.path.join(directory, "**", "*." + format)
-    total = sum(os.path.getsize(f) for f in glob.iglob(filePath, recursive=True) if os.path.isfile(f))
-    total = round(total / (1024**2), 2)
-    log.debug(f"Total files to be processed: {total}MB")
-    return total
-
-
-def relativePath(mainPath, file):
-    """
-    Gets the relative path of an input file within the input directory structure
-    Useful for getting the path of messages within an email account
-
-    Parameters:
-        mainPath (String): Parent or provided directory path
-        file (String): Email file path
-
-    Returns:
-        String: emailFolder
-    """
-
-    fullPath = Path(mainPath).resolve()
-    fullFilePath = Path(file).resolve()
-    relPath = str(fullFilePath.relative_to(fullPath))
-    if relPath == ".":
-        return ""
-    else:
-        return relPath
-
-
-def messagePath(headers):
-    """
-    Tries to read any email folder arragement from headers
-    Useful for getting the path of messages within an email account
-
-    Parameters:
-        headers (email.message.Message): Can be used as a dict of email headers
-
-    Returns:
-        String: messagePath (must at least return an empty string)
-    """
-
-    if headers["X-Folder"]:
-        messagePath = Path(headers["X-Folder"]).as_posix()
-    else:
-        messagePath = ""
-    return messagePath
-
-
-def getFileBeforeAfterPath(mainPath, mailbag_name, input, file):
-    """
-    Creates file paths for input mail files and new paths
-
-    Parameters:
-        mainPath (String) :
-        mailbag_name (String) :
-        input (String) :
-        file (String) :
-    """
-    fullPath = Path(mainPath).resolve()
-    fullFilePath = Path(file).resolve()
-    relPath = fullFilePath.relative_to(fullPath).parents[0]
-    filename = fullFilePath.name
-    folder_new = os.path.join(fullPath, mailbag_name, "data", input)
-    file_new_path = os.path.join(folder_new, relPath, filename)
-
-    return fullPath, fullFilePath, file_new_path, relPath
-
-
-def moveWithDirectoryStructure(dry_run, mainPath, mailbag_name, input, file):
-    """
-    Create new mailbag directory structure while maintaining the input data's directory structure.
-    Useful for MBOX, EML, and MSG files.
-
-    Parameters:
-        dry_run (Boolean):
-        mainPath (String): Parent or provided directory path
-        mailbag_name (String): Mailbag name
-        input (String): Mail type
-        emailFolder (String): Path of the email export file relative to mainPath
-        file (String): Email file path
-
-    Returns:
-        file_new_path (Path): The path where the file was moved
-    """
-
-    fullPath, fullFilePath, file_new_path, relPath = getFileBeforeAfterPath(mainPath, mailbag_name, input, file)
-    log.debug("Moving: " + str(fullFilePath) + " to: " + str(file_new_path) + " SubFolder: " + str(relPath))
-
-    if not dry_run:
-        moveFile(dry_run, fullFilePath, file_new_path)
-        # clean up old directory structure
-        p = fullFilePath.parents[0]
-        while p != p.root and p != fullPath:
-            if not os.listdir(p):
-                log.debug("Cleaning: " + str(p))
-                os.rmdir(p)
-                # dirty hack since rmdir is not synchronous on Windows
-                if os.name == "nt":
-                    import time
-
-                    time.sleep(0.01)
-            p = p.parent
-
-    return file_new_path
-
-
-def saveAttachments(part):
-    return (part.get_filename(), part.get_payload(decode=True))
-
-
-def deleteFile(filePath):
-    if os.path.exists(filePath):
-        os.remove(filePath)
-
-
-def startServer(dry_run, httpdShared, port=5000):
-    class QuietHandler(http.server.SimpleHTTPRequestHandler):
-        def log_message(self, format, *args):
-            pass
-
-    log.debug("Starting Server")
-    if not dry_run:
-        Handler = http.server.SimpleHTTPRequestHandler
-        with socketserver.TCPServer(("127.0.0.1", port), QuietHandler) as httpd:
-            httpdShared.append(httpd)
-            httpd.serve_forever()
-
-
-def stopServer(dry_run, httpd):
-    log.debug("Stopping Server")
-    if not dry_run:
-        httpd.shutdown()
-        httpd.server_close()
-
-
-def handle_error(errors, exception, desc, level="error"):
-    """
-    Is called when an exception is raised in the parsers.
-    returns a dict of readable and full trace errors that can be appended to.
-
-    Parameters:
-        errors (dict):
-            "msg" contains a list of human readable error messages
-            "stack_trace" contains a list of full stack traces
-        exception (Exception): The exception raised
-        desc (String): A full email message object desribed in models.py
-        level (String): Whether to log as error or warn. Defaults to error.
-    Returns:
-        errors (dict):
-            "msg" contains a list of human readable error messages
-            "stack_trace" contains a list of full stack traces
-    """
-    if exception:
-        error_msg = level.upper() + ": " + desc + ": " + repr(exception)
-        stack_header = (
-            "**************************************************************************\n"
-            + error_msg
-            + "\n**************************************************************************\n"
-        )
-        errors["stack_trace"].append(stack_header + traceback.format_exc())
-    else:
-        error_msg = level.upper() + ": " + desc + "."
-        errors["stack_trace"].append(desc + ".")
-    errors["msg"].append(error_msg)
-
-    print()
-    log_msg = (error_msg[:150] + "..") if len(error_msg) > 150 else error_msg
-    if level == "warn":
-        log.warn(log_msg)
-    else:
-        log.error(log_msg)
-
-    return errors
-
-
-def parse_part(part, bodies, attachments, errors):
-    """
-    Used for EML and MBOX parsers
-    Parses a part of an email message for multipart messages or a full message with a single part
-
-    Parameters:
-        part (email.Message.message part):
-            "msg" contains a list of human readable error messages
-            "stack_trace" contains a list of full stack traces
-        bodies (dict):
-            "msg" contains a list of human readable error messages
-            "stack_trace" contains a list of full stack traces
-        attachments (list): a list of attachment object as defined in models.py
-        errors (dict):
-            "msg" contains a list of human readable error messages
-            "stack_trace" contains a list of full stack traces
-    Returns:
-        bodies (dict):
-            "msg" contains a list of human readable error messages
-            "stack_trace" contains a list of full stack traces
-        attachments (list): a list of attachment object as defined in models.py
-        errors (dict):
-            "msg" contains a list of human readable error messages
-            "stack_trace" contains a list of full stack traces
-    """
-    content_type = part.get_content_type()
-    content_disposition = part.get_content_disposition()
-
-    # Extract body
-    try:
-        if content_disposition != "attachment":
-            if content_type == "text/html" or content_type == "text/plain":
-                part_encoding = part.get_content_charset()
-                enc_source = "read"
-                if part_encoding:
-                    try:
-                        part_encoding = codecs.lookup(part_encoding).name.lower()
-                    except:
-                        part_encoding = chardet.detect(part.get_payload(decode=True))["encoding"]
-                        enc_source = "detected"
-                else:
-                    part_encoding = chardet.detect(part.get_payload(decode=True))["encoding"]
-
-                try:
-                    message_body = part.get_payload(decode=True).decode(part_encoding)
-                except UnicodeDecodeError as e:
-                    if enc_source == "read":
-                        print("lies!")
-                        # lies!
-                        detected_encoding = chardet.detect(part.get_payload(decode=True))["encoding"]
-                        try:
-                            message_body = part.get_payload(decode=True).decode(detected_encoding)
-                            desc = (
-                                "Failed to decode message body with listed encoding "
-                                + part_encoding
-                                + " (lies!), but successfully decoded with detected encoding "
-                                + detected_encoding
-                            )
-                            errors = handle_error(errors, e, desc, "warn")
-                            part_encoding = detected_encoding
-                        except UnicodeDecodeError as e:
-                            desc = "Error decoding message body with " + part_encoding + " (" + enc_source + ")"
-                            errors = handle_error(errors, e, desc)
-                            message_body = part.get_payload(decode=True).decode(part_encoding, errors="replace")
-                    else:
-                        desc = "Error decoding message body with " + part_encoding + " (" + enc_source + ")"
-                        errors = handle_error(errors, e, desc)
-                        message_body = part.get_payload(decode=True).decode(part_encoding, errors="replace")
-
-                if content_type == "text/html":
-                    bodies["html_encoding"] = part_encoding
-                    bodies["html_body"] = message_body
-                elif content_type == "text/plain":
-                    bodies["text_encoding"] = part_encoding
-                    bodies["text_body"] = message_body
-    except Exception as e:
-        desc = "Error parsing message body"
-        errors = handle_error(errors, e, desc)
-
-    # Extract attachments
-    if part.get_content_maintype() == "multipart":
-        pass
-    elif content_disposition is None:
-        pass
-    else:
-        try:
-            if not part.get_payload(decode=True):
-                if part.get_filename():
-                    desc = "Missing attachment content, failed to read attachment " + part.get_filename()
-                else:
-                    desc = "Missing attachment content and filename, failed to read attachment"
-                errors = handle_error(errors, None, desc)
-            else:
-                attachmentFile = part.get_payload(decode=True)
-                if part.get_filename():
-                    attachmentName = part.get_filename()
-                else:
-                    desc = "Missing attachment name, using integer"
-                    errors = handle_error(errors, None, desc)
-                    attachmentName = str(len(attachments))
-
-                attachment = Attachment(
-                    Name=attachmentName,
-                    File=attachmentFile,
-                    MimeType=content_type,
-                )
-                attachments.append(attachment)
-        except Exception as e:
-            desc = "Error parsing attachments"
-            errors = handle_error(errors, e, desc)
-
-    return bodies, attachments, errors
-
-
-def decode_header_part(header):
-    """
-    Used for to decode strings according to RFC 1342.
-    If the string is not encoded, just return it.
-    For encoded strings it tries to decode it with email.header.decode_header().
-    If we don't get a real encoding, it tries it best to detect it.
-    Errors are logged and documented in the error report.
-
-    Parameters:
-        header (email.header.Header or string or None):
-    Returns:
-        header_string (str): A as-best-as-we-can-do decoded string
-    """
-    # headerObj, encoding = decode_header(header)[0]
-    header_string = []
-    for part in decode_header(header):
-        headerObj, encoding = part
-        if encoding:
-            # Did we get a real encoding?
-            try:
-                encoding = codecs.lookup(encoding).name.lower()
-            except:
-                # If not, might as well try to detect it.
-                encoding = chardet.detect(headerObj)["encoding"]
-            try:
-                header_string.append(headerObj.decode(encoding))
-            except UnicodeDecodeError as e:
-                # Document that the header isn't valid
-                desc = "Error decoding header with " + encoding
-                errors = handle_error(errors, e, desc)
-                header_string.append(headerObj.decode(encoding, errors="replace"))
-        else:
-            if isinstance(headerObj, str):
-                # Not encoded
-                header_string.append(headerObj)
-            else:
-                # Oddly, there are weird cases where quotes (") return a binary with no encoding.
-                # Dunno what to do here so just hopefully safely decode it?
-                header_string.append(headerObj.decode(errors="replace"))
-
-    return "".join(header_string)
-
-
-def parse_header(header):
-    """
-    Used to handle headers that have RFC 1342 encoding.
-    Sometimes the whole header is encoded, while
-    sometimes only part of the header string is encoded.
-    In some cases we also get a Header object that
-    decode_header_part() also handles.
-
-    Parameters:
-        header (email.header.Header or string or None):
-    Returns:
-        header_string (str or None): A as-best-as-we-can-do decoded string
-    """
-    if header is None:
-        header_string = None
-    else:
-        if isinstance(header, str):
-            header_list = []
-            for header_part in header.split(" "):
-                header_list.append(decode_header_part(header_part))
-            header_string = " ".join(header_list)
-        else:
-            header_string = html.unescape(decode_header_part(header))
-
-    return header_string
-
-
-def saveAttachmentOnDisk(dry_run, attachments_dir, message):
-    """
-    Takes an email message object and writes any attachments in the model
-    to the attachments subdirectory according to the mailbag spec
-
-    Parameters:
-        dry_run (Boolean): Option to do a test run without writing changes
-        attachments_dir (Path): Path to the attachments subdirectory
-        message (Email): A full email message object desribed in models.py
-    """
-
-    if not dry_run:
-        message_attachments_dir = os.path.join(attachments_dir, str(message.Mailbag_Message_ID))
-        os.mkdir(message_attachments_dir)
-
-    for attachment in message.Attachments:
-        if not attachment.File:
-            desc = "Missing attachment content"
-            errors = handle_error(errors, None, desc)
-        else:
-            log.debug("Saving Attachment:" + str(attachment.Name))
-            log.debug("Type:" + str(attachment.MimeType))
-            if not dry_run:
-                attachment_path = os.path.join(message_attachments_dir, attachment.Name)
-                f = open(attachment_path, "wb")
-                f.write(attachment.File)
-                f.close()
-
-
-def guessMimeType(filename):
-    """
-    Takes an file name and uses mimetypes to guess the mime type
-
-    Parameters:
-        filename (String): Attachment filename
-
-    Returns:
-        Mimetype (String)
-    """
-    return mimetypes.guess_type(filename)[0]
-
-
-def normalizePath(path):
-    # this is not sufficent yet
-    if os.name == "nt":
-        specials = ["<", ">", ":", '"', "/", "|", "?", "*"]
-        special_names = [
-            "CON",
-            "PRN",
-            "AUX",
-            "NUL",
-            "COM1",
-            "COM2",
-            "COM3",
-            "COM4",
-            "COM5",
-            "COM6",
-            "COM",
-            "COM8",
-            "COM9",
-            "LPT1",
-            "LPT2",
-            "LPT3",
-            "LPT4",
-            "LPT5",
-            "LPT6",
-            "LPT7",
-            "LPT8",
-            "LPT9",
-        ]
-        new_path = []
-        for name in os.path.normpath(path).split(os.sep):
-            illegal = False
-            for char in specials:
-                if char in name:
-                    illegal = True
-            if illegal:
-                new_path.append(urllib.parse.quote_plus(name))
-            else:
-                new_path.append(name)
-        out_path = Path(os.path.join(*new_path)).as_posix()
-    else:
-        out_path = path
-
-    if out_path == ".":
-        return ""
-    else:
-        return out_path
-
-
-def htmlFormatting(message, external_css, headers=True):
-    """
-    Creates a formatted html file using message text or html body
-    inserts any additional styling given by user
-    Useful for html and pdf derivative
-
-    Parameters:
-        message:message object
-        external_css(string): path of css file to customize derivative
-        headers(boolean):option to add table of headers
-
-    Returns:
-        String: formatted html for pdf, html derivatives
-    """
-
-    # check to see which body to use
-    html_content = False
-    encoding = False
-    if message.HTML_Body:
-        html_content = message.HTML_Body
-        encoding = message.HTML_Encoding
-    elif message.Text_Body:
-        #  converting text body to html body
-        html_content = (
-            """
-            <!DOCTYPE html>
-            <html>
-                <head>
-                    <style>
-                        #mailbagPlainText{white-space: pre;}
-                    </style>
-                </head>
-                <body>
-                    <section id='mailbagPlainText'>"""
-            + message.Text_Body
-            + """<section>
-                </body>
-            </html>
-        """
-        )
-        # using utf-8 for ascii plain text bodies as we're adding non-ascii whitespace
-        if codecs.lookup(message.Text_Encoding).name.lower() == "ascii":
-            log.debug("Using utf-8 to better handle whitespace for plain text ascii body message " + str(message.Mailbag_Message_ID))
-            encoding = "utf-8"
-        else:
-            encoding = message.Text_Encoding
-    else:
-        log.warn("Unable to format HTML, no message body found for " + str(message.Mailbag_Message_ID))
-
-    if html_content:
-
-        # Formatting HTML with beautiful soup
-        soup = BeautifulSoup(html_content.encode(encoding), "html.parser", from_encoding=encoding)
-
-        # Checking if message contains partial html
-        if not soup.html:
-            html_content = "<html>" + html_content + "</html>"
-            soup = BeautifulSoup(html_content.encode(encoding), "html.parser", from_encoding=encoding)
-        if not soup.head:
-            head = soup.new_tag("head")
-            soup.html.insert(0, head)
-        if not soup.body:
-            body = soup.new_tag("body")
-            soup.head.next_element.wrap(body)
-
-        # Check Doctype
-        doctype = False
-        for item in soup.contents:
-            if isinstance(item, Doctype):
-                doctype = True
-        if doctype is False:
-            tag = Doctype("html")
-            soup.insert(0, tag)
-
-        # optionally adds headers table to html
-        if headers:
-            # Make headers table
-            table = "<table id='mailbagHeadersTable'>"
-            if message.Subject:
-                h2 = "<h2>" + message.Subject + "</h2>"
-            else:
-                h2 = ""
-            # Headers to display
-            headerFields = [
-                "Mailbag_Message_ID",
-                "Message_ID",
-                "From",
-                "Date",
-                "To",
-                "Cc",
-                "Bcc",
-                "Subject",
-            ]
-            # Getting the values of the attrbutes and appending to HTML string
-            for headerField in headerFields:
-                value = getattr(message, headerField)
-                if not value is None and value != []:
-                    table += "<tr>"
-                    table += "<td class='header'>" + str(headerField) + "</td>"
-                    table += "<td>" + str(getattr(message, headerField)).replace("<", "&lt;").replace(">", "&gt;") + "</td>"
-                    table += "</tr>"
-            if len(message.Attachments) > 0:
-                attachmentNumber = len(message.Attachments)
-                table += "<tr>"
-                table += "<td class='header'>Attachments</td><td>"
-                for i, attachment in enumerate(message.Attachments):
-                    table += attachment.Name
-                    if i + 1 < attachmentNumber:
-                        table += "<br/>"
-                table += "</td>"
-                table += "</tr>"
-            tableSection = "<section id='mailbagHeaders'>" + h2 + table + "</table></section>"
-            soupTable = BeautifulSoup(tableSection, "html.parser")
-
-            # Add headers table to HTML body
-            soup.body.insert(0, soupTable)
-
-        # Embedding Encoding with meta
-        meta = soup.new_tag("meta")
-        meta["charset"] = "utf-8"
-        soup.head.insert(0, meta)
-
-        # Embedding default styling
-        default_css = """
-            @media print {
-                /* for Chrome margins */
-                @page { margin: 10; }
-            }
-            section#mailbagHeaders table#mailbagHeadersTable {
-                width: 100%;
-                margin-bottom: 35px;
-                text-align: left;
-                border-top: 4px solid #000000;
-                padding-top: 8px;
-                border-collapse: separate;
-                border-spacing: 0 1px;
-            }
-            section#mailbagHeaders table#mailbagHeadersTable > tbody > tr > td {
-                font: 16px Arial sans-serif;
-                color:  #000000;
-                padding: 2px 5px;
-            }
-            section#mailbagHeaders table#mailbagHeadersTable tbody tr:nth-of-type(even) {
-                background-color: #f3f3f3;
-            }
-            section#mailbagHeaders h2 {
-                margin-bottom: 2px;
-                font-size: 24px;
-                font-family: Arial sans-serif;
-        """
-        style = soup.new_tag("style")
-        style.string = default_css
-        soup.head.append(style)
-
-        # Adding external_css
-        if external_css:
-            link = soup.new_tag("link")
-            link["rel"] = "stylesheet"
-            link["type"] = "text/css"
-            link["href"] = "file:///" + external_css
-            soup.head.append(link)
-
-        # Embedding Images
-        # HT to extract_msg for this approach
-        # https://github.com/TeamMsgExtractor/msg-extractor/blob/6bed8213de1a7a41739fcf5c9363322508711fce/extract_msg/message_base.py#L403-L414
-        tags = (tag for tag in soup.findAll("img") if tag.get("src") and tag.get("src").startswith("cid:"))
-        data = None
-        for tag in tags:
-            # Iterate through the attachments until we get the right one.
-            cid = tag["src"][4:]
-
-            for attachment in message.Attachments:
-                if attachment.Name in cid:
-                    data = attachment.File
-
-            # If we found anything, inject it.
-            if data:
-                tag["src"] = (b"data:image;base64," + base64.b64encode(data)).decode(encoding)
-
-        html_content = soup.prettify(encoding).decode(encoding)
-
-    return html_content, encoding
diff --git a/mailbagit/helper/__init__.py b/mailbagit/helper/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/mailbagit/helper/common.py b/mailbagit/helper/common.py
new file mode 100644
index 0000000..c0cc9b6
--- /dev/null
+++ b/mailbagit/helper/common.py
@@ -0,0 +1,45 @@
+import traceback
+
+from structlog import get_logger
+
+log = get_logger()
+
+
+def handle_error(errors, exception, desc, level="error"):
+    """
+    Is called when an exception is raised in the parsers.
+    returns a dict of readable and full trace errors that can be appended to.
+
+    Parameters:
+        errors (dict):
+            "msg" contains a list of human readable error messages
+            "stack_trace" contains a list of full stack traces
+        exception (Exception): The exception raised
+        desc (String): A full email message object desribed in models.py
+        level (String): Whether to log as error or warn. Defaults to error.
+    Returns:
+        errors (dict):
+            "msg" contains a list of human readable error messages
+            "stack_trace" contains a list of full stack traces
+    """
+    if exception:
+        error_msg = level.upper() + ": " + desc + ": " + repr(exception)
+        stack_header = (
+            "**************************************************************************\n"
+            + error_msg
+            + "\n**************************************************************************\n"
+        )
+        errors["stack_trace"].append(stack_header + traceback.format_exc())
+    else:
+        error_msg = level.upper() + ": " + desc + "."
+        errors["stack_trace"].append(desc + ".")
+    errors["msg"].append(error_msg)
+
+    print()
+    log_msg = (error_msg[:150] + "..") if len(error_msg) > 150 else error_msg
+    if level == "warn":
+        log.warn(log_msg)
+    else:
+        log.error(log_msg)
+
+    return errors
diff --git a/mailbagit/helper/controller.py b/mailbagit/helper/controller.py
new file mode 100644
index 0000000..d6d8129
--- /dev/null
+++ b/mailbagit/helper/controller.py
@@ -0,0 +1,113 @@
+import os, shutil, glob
+import datetime
+from time import time
+import csv
+
+import mailbagit.globals as globals
+
+from structlog import get_logger
+
+log = get_logger()
+
+
+def progress(current, total, start_time, prefix="", suffix="", decimals=1, length=100, fill="█", print_End="\r"):
+    """
+    Call in a loop to create terminal progress bar
+
+    Parameters:
+        current (int): current progress
+        total (int): total iterations
+        start_time (float): start time
+        prefix (String): prefix string
+        suffix (String): suffix string
+        decimals (int): positive number of decimals in percent complete
+        length (int): character length of bar (Int)
+        fill (String): bar fill character (Str)
+        printEnd (String): end character (e.g. "\r", "\r\n")
+    """
+
+    time_spent = time() - start_time
+    remaining_time = round(time_spent * (total / current - 1), 2)
+    e = datetime.datetime.now()
+    percent = ("{0:." + str(decimals) + "f}").format(100 * (current / float(total)))
+    # filledLength = int(length * current // total)
+    style = globals.style
+    # bar = fill * filledLength + '-' * (length - filledLength)
+
+    dt = f"{e.year}-{e.month:02d}-{e.day:02d} {e.hour:02d}:{e.minute:02d}.{e.second:02d}"
+    message_type = f'[{style["cy"][0]}{prefix}{style["b"][1]}]'
+    # deco_prefix = f'{style["b"][0]}{prefix}{style["b"][1]}'
+    # statusBar = f'|{bar}| {percent}% [{current}MB out of {total}MB] {suffix}'
+    status = f"{percent}% [{current} / {total} messages] {remaining_time}s remaining"
+
+    print(f"\r{dt} {message_type} {status}", end=print_End)
+
+
+def progressMessage(msg, print_End="\r"):
+    """
+    Shows a message as progress. Useful since it make take awhile to save large bags
+    even after all messages have been processed.
+
+    Parameters:
+        msg (String): A message to show as progress
+    """
+    e = datetime.datetime.now()
+    style = globals.style
+    dt = f"{e.year}-{e.month:02d}-{e.day:02d} {e.hour:02d}:{e.minute:02d}.{e.second:02d}"
+    message_type = f'[{style["cy"][0]}{"Progress "}{style["b"][1]}]'
+    print(f"\r{dt} {message_type} {msg}", end=print_End)
+
+
+def writeAttachmentsToDisk(dry_run, attachments_dir, message):
+    """
+    Takes an email message object and writes any attachments in the model
+    to the attachments subdirectory according to the mailbag spec.
+    Also creates attachments.csv according to the mailbag spec.
+
+    Parameters:
+        dry_run (Boolean): Option to do a test run without writing changes
+        attachments_dir (Path): Path to the attachments subdirectory
+        message (Email): A full email message object desribed in models.py
+    """
+
+    message_attachments_dir = os.path.join(attachments_dir, str(message.Mailbag_Message_ID))
+    if not dry_run:
+        os.mkdir(message_attachments_dir)
+
+    # Set up attachments.csv
+    attachments_csv = os.path.join(message_attachments_dir, "attachments.csv")
+    attachments_headers = ["Original-Filename", "Mailbag-Filename", "MimeType", "Content-ID"]
+    attachment_data = [attachments_headers]
+
+    for i, attachment in enumerate(message.Attachments):
+        if attachment.Name:
+            # Need to handle filename conflicts with attachments.csv
+            # The format parsers raise a warning about this
+            if attachment.Name.lower() == "attachments.csv":
+                writtenName = str(i) + os.path.splitext(attachment.Name)[1]
+                desc = ""
+                errors = common.handle_error(errors, None, desc, "warn")
+            else:
+                writtenName = attachment.Name
+            attachment_row = [attachment.Name, writtenName, attachment.MimeType, attachment.Content_ID]
+        else:
+            # If there is no filename available, just use and integer
+            # The format parsers raise an error about this
+            writtenName = str(i)
+            attachment_row = ["", writtenName, attachment.MimeType, attachment.Content_ID]
+        attachment_data.append(attachment_row)
+
+        log.debug("Saving Attachment:" + str(attachment.Name))
+        log.debug("Type:" + str(attachment.MimeType))
+        if not dry_run:
+            attachment_path = os.path.join(message_attachments_dir, writtenName)
+            f = open(attachment_path, "wb")
+            f.write(attachment.File)
+            f.close()
+
+    # Write attachments.csv
+    if not dry_run:
+        with open(attachments_csv, "w", encoding="utf-8", newline="") as csv_file:
+            writer = csv.writer(csv_file)
+            writer.writerows(attachment_data)
+            csv_file.close()
diff --git a/mailbagit/helper/derivative.py b/mailbagit/helper/derivative.py
new file mode 100644
index 0000000..5f80a0b
--- /dev/null
+++ b/mailbagit/helper/derivative.py
@@ -0,0 +1,250 @@
+import os
+import base64
+import codecs
+from bs4 import BeautifulSoup, Doctype
+
+import http.server
+import socketserver
+
+from structlog import get_logger
+
+log = get_logger()
+
+
+def startServer(dry_run, httpdShared, port=5000):
+    class QuietHandler(http.server.SimpleHTTPRequestHandler):
+        def log_message(self, format, *args):
+            pass
+
+    log.debug("Starting Server")
+    if not dry_run:
+        Handler = http.server.SimpleHTTPRequestHandler
+        with socketserver.TCPServer(("127.0.0.1", port), QuietHandler) as httpd:
+            httpdShared.append(httpd)
+            httpd.serve_forever()
+
+
+def stopServer(dry_run, httpd):
+    log.debug("Stopping Server")
+    if not dry_run:
+        httpd.shutdown()
+        httpd.server_close()
+
+
+def deleteFile(filePath):
+    if os.path.exists(filePath):
+        os.remove(filePath)
+
+
+def inlineAttachments(html_string, encoding):
+    """
+    Reads an HTML body and looks for images that might rely on inline
+    attachments. This loops through all the <img> tags and returns
+    filenames to check against attached files. Used for deciding the
+    Content-Disposition header for MBOX and EML derivatives
+
+    Parameters:
+        HTML_Body(str):
+
+    Returns:
+        Dict: A dict with filenames as keys and full contentIDs (<filename@identifier>) as values
+    """
+
+    inline_files = {}
+
+    # Formatting HTML with beautiful soup
+    soup = BeautifulSoup(html_string.encode(encoding), "html.parser", from_encoding=encoding)
+    tags = (tag for tag in soup.findAll("img") if tag.get("src") and tag.get("src").startswith("cid:"))
+    for tag in tags:
+        # Iterate through the attachments until we get the right one.
+        cid = tag["src"][4:]
+        if "@" in cid:
+            filename, contentID = cid.split("@", 1)
+            inline_files[filename] = cid
+        else:
+            inline_files[cid] = cid
+
+    return inline_files
+
+
+def htmlFormatting(message, external_css, headers=True):
+    """
+    Creates a formatted html file using message text or html body
+    inserts any additional styling given by user
+    Useful for html and pdf derivative
+
+    Parameters:
+        message:message object
+        external_css(string): path of css file to customize derivative
+        headers(boolean):option to add table of headers
+
+    Returns:
+        String: formatted html for pdf, html derivatives
+    """
+
+    # check to see which body to use
+    html_content = False
+    encoding = False
+    if message.HTML_Body:
+        html_content = message.HTML_Body
+        encoding = message.HTML_Encoding
+    elif message.Text_Body:
+        #  converting text body to html body
+        html_content = (
+            """
+            <!DOCTYPE html>
+            <html>
+                <head>
+                    <style>
+                        #mailbagPlainText{white-space: pre;}
+                    </style>
+                </head>
+                <body>
+                    <section id='mailbagPlainText'>"""
+            + message.Text_Body
+            + """<section>
+                </body>
+            </html>
+        """
+        )
+        # using utf-8 for ascii plain text bodies as we're adding non-ascii whitespace
+        if codecs.lookup(message.Text_Encoding).name.lower() == "ascii":
+            log.debug("Using utf-8 to better handle whitespace for plain text ascii body message " + str(message.Mailbag_Message_ID))
+            encoding = "utf-8"
+        else:
+            encoding = message.Text_Encoding
+    else:
+        log.warn("Unable to format HTML, no message body found for " + str(message.Mailbag_Message_ID))
+
+    if html_content:
+
+        # Formatting HTML with beautiful soup
+        soup = BeautifulSoup(html_content.encode(encoding), "html.parser", from_encoding=encoding)
+
+        # Checking if message contains partial html
+        if not soup.html:
+            html_content = "<html>" + html_content + "</html>"
+            soup = BeautifulSoup(html_content.encode(encoding), "html.parser", from_encoding=encoding)
+        if not soup.head:
+            head = soup.new_tag("head")
+            soup.html.insert(0, head)
+        if not soup.body:
+            body = soup.new_tag("body")
+            soup.head.next_element.wrap(body)
+
+        # Check Doctype
+        doctype = False
+        for item in soup.contents:
+            if isinstance(item, Doctype):
+                doctype = True
+        if doctype is False:
+            tag = Doctype("html")
+            soup.insert(0, tag)
+
+        # optionally adds headers table to html
+        if headers:
+            # Make headers table
+            table = "<table id='mailbagHeadersTable'>"
+            if message.Subject:
+                h2 = "<h2>" + message.Subject + "</h2>"
+            else:
+                h2 = ""
+            # Headers to display
+            headerFields = [
+                "Mailbag_Message_ID",
+                "Message_ID",
+                "From",
+                "Date",
+                "To",
+                "Cc",
+                "Bcc",
+                "Subject",
+            ]
+            # Getting the values of the attrbutes and appending to HTML string
+            for headerField in headerFields:
+                value = getattr(message, headerField)
+                if not value is None and value != []:
+                    table += "<tr>"
+                    table += "<td class='header'>" + str(headerField) + "</td>"
+                    table += "<td>" + str(getattr(message, headerField)).replace("<", "&lt;").replace(">", "&gt;") + "</td>"
+                    table += "</tr>"
+            if len(message.Attachments) > 0:
+                attachmentNumber = len(message.Attachments)
+                table += "<tr>"
+                table += "<td class='header'>Attachments</td><td>"
+                for i, attachment in enumerate(message.Attachments):
+                    table += attachment.Name
+                    if i + 1 < attachmentNumber:
+                        table += "<br/>"
+                table += "</td>"
+                table += "</tr>"
+            tableSection = "<section id='mailbagHeaders'>" + h2 + table + "</table></section>"
+            soupTable = BeautifulSoup(tableSection, "html.parser")
+
+            # Add headers table to HTML body
+            soup.body.insert(0, soupTable)
+
+        # Embedding Encoding with meta
+        meta = soup.new_tag("meta")
+        meta["charset"] = "utf-8"
+        soup.head.insert(0, meta)
+
+        # Embedding default styling
+        default_css = """
+            @media print {
+                /* for Chrome margins */
+                @page { margin: 10; }
+            }
+            section#mailbagHeaders table#mailbagHeadersTable {
+                width: 100%;
+                margin-bottom: 35px;
+                text-align: left;
+                border-top: 4px solid #000000;
+                padding-top: 8px;
+                border-collapse: separate;
+                border-spacing: 0 1px;
+            }
+            section#mailbagHeaders table#mailbagHeadersTable > tbody > tr > td {
+                font: 16px Arial sans-serif;
+                color:  #000000;
+                padding: 2px 5px;
+            }
+            section#mailbagHeaders table#mailbagHeadersTable tbody tr:nth-of-type(even) {
+                background-color: #f3f3f3;
+            }
+            section#mailbagHeaders h2 {
+                margin-bottom: 2px;
+                font-size: 24px;
+                font-family: Arial sans-serif;
+            }
+        """
+        style = soup.new_tag("style")
+        style.string = default_css
+        # Adding external_css
+        if external_css:
+            with open(external_css) as css_file:
+                style.string = style.string + "\n" + css_file.read()
+                css_file.close()
+        soup.head.append(style)
+
+        # Embedding Images
+        # HT to extract_msg for this approach
+        # https://github.com/TeamMsgExtractor/msg-extractor/blob/6bed8213de1a7a41739fcf5c9363322508711fce/extract_msg/message_base.py#L403-L414
+        tags = (tag for tag in soup.findAll("img") if tag.get("src") and tag.get("src").startswith("cid:"))
+        data = None
+        for tag in tags:
+            # Iterate through the attachments until we get the right one.
+            cid = tag["src"][4:]
+
+            for attachment in message.Attachments:
+                if attachment.Name:
+                    if attachment.Name in cid:
+                        data = attachment.File
+
+            # If we found anything, inject it.
+            if data:
+                tag["src"] = (b"data:image;base64," + base64.b64encode(data)).decode(encoding)
+
+        html_content = soup.prettify(encoding).decode(encoding)
+
+    return html_content, encoding
diff --git a/mailbagit/helper/format.py b/mailbagit/helper/format.py
new file mode 100644
index 0000000..1fda9c7
--- /dev/null
+++ b/mailbagit/helper/format.py
@@ -0,0 +1,429 @@
+import os, shutil, glob
+from pathlib import Path
+import mimetypes
+import urllib.parse
+import chardet, codecs
+from email.header import Header, decode_header, make_header
+from mailbagit.models import Attachment
+import mailbagit.helper.common as common
+import html
+import uuid
+
+from structlog import get_logger
+
+log = get_logger()
+
+
+def relativePath(mainPath, file):
+    """
+    Gets the relative path of an input file within the input directory structure
+    Useful for getting the path of messages within an email account
+
+    Parameters:
+        mainPath (String): Parent or provided directory path
+        file (String): Email file path
+
+    Returns:
+        String: emailFolder
+    """
+
+    fullPath = Path(mainPath).resolve()
+    fullFilePath = Path(file).resolve()
+    relPath = str(fullFilePath.relative_to(fullPath))
+    if relPath == ".":
+        return ""
+    else:
+        return relPath
+
+
+def safely_decode(body_type, binary_text, encodings, errors):
+    """
+    Tries to safely decode text for message bodies using an encodings dict.
+    Goes through encodings by priority and uses the first successful one.
+    Tries to detect encoding if all else fails.
+    Fully documents the encoding and label for each one that fails
+
+    Parameters:
+        body_type (str): a description of the body (i.e. HTML or Text)
+        binary_text (binary): an encoded string
+        encodings (dict):
+            Integer key denoting priority (dict):
+                "name" (str):
+                "label" (str): a description for the encoding, such as its source
+        errors (dict):
+            "msg" contains a list of human readable error messages
+            "stack_trace" contains a list of full stack traces
+
+    Returns:
+        test (str): a decoded unicode string
+        used (str): The encoding used to decode the text
+        errors (dict):
+            "msg" contains a list of human readable error messages
+            "stack_trace" contains a list of full stack traces
+    """
+    errorObj = None
+    used = None
+    success = False
+    failed = []
+    valid = []
+    for priority in encodings.keys():
+        # for EML/MBOX could be None
+        if encodings[priority]["name"]:
+            try:
+                valid_encoding = codecs.lookup(encodings[priority]["name"]).name.lower()
+                valid.append(valid_encoding)
+                text = binary_text.decode(valid_encoding)
+                used = encodings[priority]["name"]
+                success = True
+                continue
+            except Exception as e:
+                errorObj = e
+                failed.append(encodings[priority]["name"])
+
+    if success == False:
+        try:
+            detected = chardet.detect(binary_text)["encoding"]
+            text = binary_text.decode(detected)
+            used = detected
+            if len(valid) < 1:
+                # desc = "No valid listed encodings, but successfully decoded " + body_type + " body with detected encoding " + detected
+                # Don't actually raise this warning, not sure if user cares that much?
+                pass
+            else:
+                desc = (
+                    "Failed to decode "
+                    + body_type
+                    + " message body with listed encoding(s) "
+                    + ", ".join(failed)
+                    + " (lies!), but successfully decoded with detected encoding "
+                    + detected
+                )
+                errors = common.handle_error(errors, errorObj, desc, "warn")
+        except UnicodeDecodeError as e:
+            if len(valid) < 1:
+                desc = "No valid listed encodings. Failed to decode " + body_type + " message body with detected encoding " + detected
+                # just replace the errors
+                text = binary_text.decode(detected, errors="replace")
+                used = detected
+            else:
+                desc = "Failed to decode " + body_type + "message body with listed encoding(s) " + ", ".join(failed)
+                # just replace the errors
+                text = binary_text.decode(valid[0], errors="replace")
+                used = valid[0]
+            errors = common.handle_error(errors, e, desc, "error")
+
+    return text, used, errors
+
+
+def parse_part(part, bodies, attachments, errors):
+    """
+    Used for EML and MBOX parsers
+    Parses a part of an email message for multipart messages or a full message with a single part
+
+    Parameters:
+        part (email.Message.message part):
+            "msg" contains a list of human readable error messages
+            "stack_trace" contains a list of full stack traces
+        bodies (dict):
+            "msg" contains a list of human readable error messages
+            "stack_trace" contains a list of full stack traces
+        attachments (list): a list of attachment object as defined in models.py
+        errors (dict):
+            "msg" contains a list of human readable error messages
+            "stack_trace" contains a list of full stack traces
+    Returns:
+        bodies (dict):
+            "msg" contains a list of human readable error messages
+            "stack_trace" contains a list of full stack traces
+        attachments (list): a list of attachment object as defined in models.py
+        errors (dict):
+            "msg" contains a list of human readable error messages
+            "stack_trace" contains a list of full stack traces
+    """
+    content_type = part.get_content_type()
+    content_disposition = part.get_content_disposition()
+    content_id = part["Content-ID"]
+
+    # Extract body
+    try:
+        if content_disposition != "attachment" and content_disposition != "inline":
+            if content_type == "text/html" or content_type == "text/plain":
+                encodings = {}
+                encodings[1] = {"name": part.get_content_charset(), "label": "listed charset"}
+                message_body, part_encoding, errors = safely_decode(content_type, part.get_payload(decode=True), encodings, errors)
+                if content_type == "text/html":
+                    bodies["html_encoding"] = part_encoding
+                    bodies["html_body"] = message_body
+                elif content_type == "text/plain":
+                    bodies["text_encoding"] = part_encoding
+                    bodies["text_body"] = message_body
+    except Exception as e:
+        desc = "Error parsing message body"
+        errors = common.handle_error(errors, e, desc)
+
+    # Extract attachments
+    if part.get_content_maintype() == "multipart":
+        pass
+    elif content_disposition is None and content_id is None:
+        pass
+    else:
+        try:
+            if not part.get_payload(decode=True):
+                if part.get_filename():
+                    desc = "Missing attachment content, failed to read attachment " + part.get_filename()
+                else:
+                    desc = "Missing attachment content and filename, failed to read attachment"
+                errors = common.handle_error(errors, None, desc)
+            else:
+                # Generate a Content-ID if none is available
+                if content_id is None:
+                    content_id = uuid.uuid4().hex
+
+                attachmentFile = part.get_payload(decode=True)
+                if part.get_filename():
+                    attachmentName = part.get_filename()
+                else:
+                    attachmentName = None
+                    desc = "No filename found for attachment, integer will be used instead"
+                    errors = common.handle_error(errors, None, desc)
+
+                # Handle attachments.csv conflict
+                # helper.controller.writeAttachmentsToDisk() handles this
+                if attachmentName:
+                    if attachmentName.lower() == "attachments.csv":
+                        desc = "attachment " + attachmentName + " will be renamed to avoid filename conflict with mailbag spec"
+                        errors = common.handle_error(errors, None, desc, "warn")
+
+                attachment = Attachment(
+                    Name=attachmentName,
+                    File=attachmentFile,
+                    MimeType=content_type,
+                    Content_ID=content_id,
+                )
+                attachments.append(attachment)
+        except Exception as e:
+            desc = "Error parsing attachments"
+            errors = common.handle_error(errors, e, desc)
+
+    return bodies, attachments, errors
+
+
+def decode_header_part(header):
+    """
+    Used for to decode strings according to RFC 1342.
+    If the string is not encoded, just return it.
+    For encoded strings it tries to decode it with email.header.decode_header().
+    If we don't get a real encoding, it tries it best to detect it.
+    Errors are logged and documented in the error report.
+
+    Parameters:
+        header (email.header.Header or string or None):
+    Returns:
+        header_string (str): A as-best-as-we-can-do decoded string
+    """
+    # headerObj, encoding = decode_header(header)[0]
+    header_string = []
+    for part in decode_header(header):
+        headerObj, encoding = part
+        if encoding:
+            # Did we get a real encoding?
+            try:
+                encoding = codecs.lookup(encoding).name.lower()
+            except:
+                # If not, might as well try to detect it.
+                encoding = chardet.detect(headerObj)["encoding"]
+            try:
+                header_string.append(headerObj.decode(encoding))
+            except UnicodeDecodeError as e:
+                # Document that the header isn't valid
+                desc = "Error decoding header with " + encoding
+                errors = common.handle_error(errors, e, desc)
+                header_string.append(headerObj.decode(encoding, errors="replace"))
+        else:
+            if isinstance(headerObj, str):
+                # Not encoded
+                header_string.append(headerObj)
+            else:
+                # Oddly, there are weird cases where quotes (") return a binary with no encoding.
+                # Dunno what to do here so just hopefully safely decode it?
+                header_string.append(headerObj.decode(errors="replace"))
+
+    return "".join(header_string)
+
+
+def parse_header(header):
+    """
+    Used to handle headers that have RFC 1342 encoding.
+    Sometimes the whole header is encoded, while
+    sometimes only part of the header string is encoded.
+    In some cases we also get a Header object that
+    decode_header_part() also handles.
+
+    Parameters:
+        header (email.header.Header or string or None):
+    Returns:
+        header_string (str or None): A as-best-as-we-can-do decoded string
+    """
+    if header is None:
+        header_string = None
+    else:
+        if isinstance(header, str):
+            header_list = []
+            for header_part in header.split(" "):
+                header_list.append(decode_header_part(header_part))
+            header_string = " ".join(header_list)
+        else:
+            header_string = html.unescape(decode_header_part(header))
+
+    return header_string
+
+
+def messagePath(headers):
+    """
+    Tries to read any email folder arragement from headers
+    Useful for getting the path of messages within an email account
+
+    Parameters:
+        headers (email.message.Message): Can be used as a dict of email headers
+
+    Returns:
+        String: messagePath (must at least return an empty string)
+    """
+
+    if headers["X-Folder"]:
+        messagePath = Path(headers["X-Folder"]).as_posix()
+    else:
+        messagePath = ""
+    return messagePath
+
+
+def normalizePath(path):
+    # this is not sufficent yet
+    if os.name == "nt":
+        specials = ["<", ">", ":", '"', "/", "|", "?", "*"]
+        special_names = [
+            "CON",
+            "PRN",
+            "AUX",
+            "NUL",
+            "COM1",
+            "COM2",
+            "COM3",
+            "COM4",
+            "COM5",
+            "COM6",
+            "COM",
+            "COM8",
+            "COM9",
+            "LPT1",
+            "LPT2",
+            "LPT3",
+            "LPT4",
+            "LPT5",
+            "LPT6",
+            "LPT7",
+            "LPT8",
+            "LPT9",
+        ]
+        new_path = []
+        for name in os.path.normpath(path).split(os.sep):
+            illegal = False
+            for char in specials:
+                if char in name:
+                    illegal = True
+            if illegal:
+                new_path.append(urllib.parse.quote_plus(name))
+            else:
+                new_path.append(name)
+        out_path = Path(os.path.join(*new_path)).as_posix()
+    else:
+        out_path = path
+
+    if out_path == ".":
+        return ""
+    else:
+        return out_path
+
+
+def moveFile(dry_run, oldPath, newPath):
+    os.makedirs(os.path.dirname(newPath), exist_ok=True)
+    try:
+        log.debug("from: " + str(oldPath))
+        log.debug("to: " + str(newPath))
+        shutil.move(oldPath, newPath)
+    except IOError as e:
+        log.error("Unable to move file. %s" % e)
+
+
+def getFileBeforeAfterPath(mainPath, mailbag_name, input, file):
+    """
+    Creates file paths for input mail files and new paths
+
+    Parameters:
+        mainPath (String) :
+        mailbag_name (String) :
+        input (String) :
+        file (String) :
+    """
+    fullPath = Path(mainPath).resolve()
+    fullFilePath = Path(file).resolve()
+    relPath = fullFilePath.relative_to(fullPath).parents[0]
+    filename = fullFilePath.name
+    folder_new = os.path.join(fullPath, mailbag_name, "data", input)
+    file_new_path = os.path.join(folder_new, relPath, filename)
+
+    return fullPath, fullFilePath, file_new_path, relPath
+
+
+def moveWithDirectoryStructure(dry_run, mainPath, mailbag_name, input, file):
+    """
+    Create new mailbag directory structure while maintaining the input data's directory structure.
+    Uses for both email files matching the input file extension and companion files if that option is selected
+
+    Parameters:
+        dry_run (Boolean):
+        mainPath (String): Parent or provided directory path
+        mailbag_name (String): Mailbag name
+        input (String): email file format to be packaged into a mailbag
+        emailFolder (String): Path of the email export file relative to mainPath
+        file (String): Email file path
+
+    Returns:
+        file_new_path (Path): The path where the file was moved
+    """
+
+    fullPath, fullFilePath, file_new_path, relPath = getFileBeforeAfterPath(mainPath, mailbag_name, input, file)
+    if file.lower().endswith("." + input.lower()):
+        log.debug("Moving: " + str(fullFilePath) + " to: " + str(file_new_path) + " SubFolder: " + str(relPath))
+    else:
+        log.debug("Moving companion file: " + str(fullFilePath) + " to: " + str(file_new_path) + " SubFolder: " + str(relPath))
+
+    if not dry_run:
+        moveFile(dry_run, fullFilePath, file_new_path)
+        # clean up old directory structure
+        p = fullFilePath.parents[0]
+        while p != p.root and p != fullPath:
+            if not os.listdir(p):
+                log.debug("Cleaning: " + str(p))
+                os.rmdir(p)
+                # dirty hack since rmdir is not synchronous on Windows
+                if os.name == "nt":
+                    import time
+
+                    time.sleep(0.01)
+            p = p.parent
+
+    return file_new_path
+
+
+def guessMimeType(filename):
+    """
+    Takes an file name and uses mimetypes to guess the mime type
+
+    Parameters:
+        filename (String): Attachment filename
+
+    Returns:
+        Mimetype (String)
+    """
+    return mimetypes.guess_type(filename)[0]
diff --git a/mailbagit/models.py b/mailbagit/models.py
index e0b8600..36bf45d 100644
--- a/mailbagit/models.py
+++ b/mailbagit/models.py
@@ -7,6 +7,7 @@ class Attachment(models.Base):
     Name = fields.StringField()
     File = fields.EmbeddedField(bytes)
     MimeType = fields.StringField()
+    Content_ID = fields.StringField()
 
 
 class Email(models.Base):


Test patch:
diff --git a/tests/test_formats.py b/tests/test_formats.py
index ffd9246..ee87b2b 100644
--- a/tests/test_formats.py
+++ b/tests/test_formats.py
@@ -10,7 +10,7 @@
 # This is a mock object representing the args returned from argparse/Gooey
 @pytest.fixture
 def cli_args():
-    return Namespace(dry_run=True, mailbag_name="New_Mailbag")
+    return Namespace(dry_run=True, mailbag_name="New_Mailbag", companion_files=False)
 
 
 def test_Mbox(cli_args):
@@ -36,7 +36,13 @@ def test_Mbox(cli_args):
                     for exp_attach in expected.Attachments:
                         if attachment.Name == exp_attach.Name:
                             match = True
-                            assert attachment == exp_attach
+                            for attachmentField in attachment:
+                                dump = getattr(exp_attach, attachmentField[0])
+                                compare = getattr(attachment, attachmentField[0])
+                                if attachmentField[0] == "Content_ID":
+                                    assert len(compare) > 0
+                                else:
+                                    assert dump == compare
                     assert match == True
             else:
                 assert getattr(message, field[0]) == getattr(expected, field[0])
@@ -65,7 +71,13 @@ def test_MSG(cli_args):
                     for exp_attach in expected.Attachments:
                         if attachment.Name == exp_attach.Name:
                             match = True
-                            assert attachment == exp_attach
+                            for attachmentField in attachment:
+                                dump = getattr(exp_attach, attachmentField[0])
+                                compare = getattr(attachment, attachmentField[0])
+                                if attachmentField[0] == "Content_ID":
+                                    assert len(compare) > 0
+                                else:
+                                    assert dump == compare
                     assert match == True
             else:
                 assert getattr(message, field[0]) == getattr(expected, field[0])
@@ -95,7 +107,13 @@ def test_EML(cli_args):
                     for exp_attach in expected.Attachments:
                         if attachment.Name == exp_attach.Name:
                             match = True
-                            assert attachment == exp_attach
+                            for attachmentField in attachment:
+                                dump = getattr(exp_attach, attachmentField[0])
+                                compare = getattr(attachment, attachmentField[0])
+                                if attachmentField[0] == "Content_ID":
+                                    assert len(compare) > 0
+                                else:
+                                    assert dump == compare
                     assert match == True
             else:
                 assert getattr(message, field[0]) == getattr(expected, field[0])
@@ -127,7 +145,13 @@ def test_PST(cli_args):
                     for exp_attach in expected.Attachments:
                         if attachment.Name == exp_attach.Name:
                             match = True
-                            assert attachment == exp_attach
+                            for attachmentField in attachment:
+                                dump = getattr(exp_attach, attachmentField[0])
+                                compare = getattr(attachment, attachmentField[0])
+                                if attachmentField[0] == "Content_ID":
+                                    assert len(compare) > 0
+                                else:
+                                    assert dump == compare
                     assert match == True
             else:
                 assert getattr(message, field[0]) == getattr(expected, field[0])