Repository: AlexsLemonade/refinebio

Issue:
Don't fail when detecting column header for illumina sample
### Context

We have `4287` samples that failed because we weren't able to find an id column in their headers. For example for sample [GSM2677584](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM2677584) we failed with the error:

`Could not find any ID column in headers ['ID_REF ', 'SAMPLE 1', 'Detection Pval', 'SAMPLE 2', 'Detection Pval', 'SAMPLE 3', 'Detection Pval', 'SAMPLE 4', 'Detection Pval', 'SAMPLE 5', 'Detection Pval', 'SAMPLE 6', 'Detection Pval', 'SAMPLE 7', 'Detection Pval', 'SAMPLE 8', 'Detection Pval', 'SAMPLE 9', 'Detection Pval', 'SAMPLE 10', 'Detection Pval', 'SAMPLE 11', 'Detection Pval', 'SAMPLE 12', 'Detection Pval', 'SAMPLE 13', 'Detection Pval', 'SAMPLE 14', 'Detection Pval', 'SAMPLE 15', 'Detection Pval', 'SAMPLE 16', 'Detection Pval', 'SAMPLE 17', 'Detection Pval', 'SAMPLE 18', 'Detection Pval'] for file /home/user/data_store/processor_job_1310904/GSE100301_non-normalized.txt.sanitized`

https://github.com/AlexsLemonade/refinebio/issues/86#issuecomment-379308817

### Problem or idea

Seems like we are not parsing the first column correctly, there's trailing whitespace on `'ID_REF '`, and that's causing the job to fail in 

https://github.com/AlexsLemonade/refinebio/blob/dev/workers/data_refinery_workers/processors/illumina.py#L115-L121

There're more samples with the same error (~14k), however, the fix doesn't seem to be as easy for the others.

<details><summary>queries</summary>

```sql
select count(*)
from (
    select samples.accession_code, (
        select processor_jobs.failure_reason
        from original_file_sample_associations, processorjob_originalfile_associations, processor_jobs
        where samples.id=original_file_sample_associations.sample_id
            and original_file_sample_associations.original_file_id=processorjob_originalfile_associations.original_file_id
            and processorjob_originalfile_associations.processor_job_id=processor_jobs.id
        order by processor_jobs.start_time desc           
        limit 1
      ) as job_failure_reason
    from samples
    where samples.is_processed='f'
  ) tt 
where job_failure_reason like 'Could not find any ID column in headers [''ID_REF '',%';
```

```sql
select count(*)
from (
    select samples.accession_code, (
        select processor_jobs.failure_reason
        from original_file_sample_associations, processorjob_originalfile_associations, processor_jobs
        where samples.id=original_file_sample_associations.sample_id
            and original_file_sample_associations.original_file_id=processorjob_originalfile_associations.original_file_id
            and processorjob_originalfile_associations.processor_job_id=processor_jobs.id
        order by processor_jobs.start_time desc           
        limit 1
      ) as job_failure_reason
    from samples
    where samples.is_processed='f'
  ) tt 
where job_failure_reason like 'Could not find any ID column in headers%';
 count 
-------
 14389
(1 row)
```

</details>

### Solution or next step

This looks like an easy fix, I'll ensure trailing whitespaces are removed from columns names.



Discussion:
Good news on these! They should be fast enough to process that if you processed with trailing whitespace and then just took the samples that failed there, it'd be quick to keep iterating through the common failure modes.
#1561 should address this. Once that's deployed we can retry those samples with:

```
retry_samples --failure-regex="Could\snot\sfind\sany\sID\scolumn\sin\sheaders\s\[\'ID_REF\s+\'"
``` 

Resolving patch:
diff --git a/foreman/data_refinery_foreman/surveyor/geo.py b/foreman/data_refinery_foreman/surveyor/geo.py
--- a/foreman/data_refinery_foreman/surveyor/geo.py
+++ b/foreman/data_refinery_foreman/surveyor/geo.py
@@ -166,7 +166,7 @@ def set_platform_properties(
         return sample_object
 
     def get_miniml_url(self, experiment_accession_code):
-        """ Build the URL for the MINiML files for this accession code.
+        """Build the URL for the MINiML files for this accession code.
         ex:
         'GSE68061' -> 'ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE68nnn/GSE68061/miniml/GSE68061_family.xml.tgz'
 
@@ -210,7 +210,7 @@ def _apply_harmonized_metadata_to_sample(sample: Sample, harmonized_metadata: di
 
     @staticmethod
     def _apply_metadata_to_experiment(experiment: Experiment, gse):
-        """ Gets the metadata out of gse and applies it to the experiment"""
+        """Gets the metadata out of gse and applies it to the experiment"""
         experiment.source_url = (
             "https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=" + experiment.accession_code
         )
@@ -237,7 +237,7 @@ def _apply_metadata_to_experiment(experiment: Experiment, gse):
     def create_experiment_and_samples_from_api(
         self, experiment_accession_code
     ) -> (Experiment, List[Sample]):
-        """ The main surveyor - find the Experiment and Samples from NCBI GEO.
+        """The main surveyor - find the Experiment and Samples from NCBI GEO.
 
         Uses the GEOParse library, for which docs can be found here: https://geoparse.readthedocs.io/en/latest/usage.html#working-with-geo-objects
 
@@ -337,24 +337,29 @@ def create_experiment_and_samples_from_api(
                     if supplementary_file_url == "NONE":
                         break
 
+                    lower_file_url = supplementary_file_url.lower()
+
                     # We never want these!
-                    if "idat.gz" in supplementary_file_url.lower():
-                        continue
-                    if ".chp" in supplementary_file_url.lower():
-                        continue
-                    if "chp.gz" in supplementary_file_url.lower():
-                        continue
-                    if "ndf.gz" in supplementary_file_url.lower():
-                        continue
-                    if "pos.gz" in supplementary_file_url.lower():
-                        continue
-                    if "pair.gz" in supplementary_file_url.lower():
-                        continue
-                    if "gff.gz" in supplementary_file_url.lower():
+                    if (
+                        "idat.gz" in lower_file_url
+                        or ".chp" in lower_file_url
+                        or "chp.gz" in lower_file_url
+                        or "ndf.gz" in lower_file_url
+                        or "pos.gz" in lower_file_url
+                        or "pair.gz" in lower_file_url
+                        or "gff.gz" in lower_file_url
+                        or "sdf.gz" in lower_file_url
+                        or "tif.gz" in lower_file_url
+                        or "locs.gz" in lower_file_url
+                        or "grn.xml.gz" in lower_file_url
+                        or "red.xml.gz" in lower_file_url
+                        # As far as I have seen, real Illumina files never come in
+                        # .csv, but plenty of irrelevant data does.
+                        or (sample_object.manufacturer == "ILLUMINA" and "csv.gz" in lower_file_url)
+                    ):
                         continue
 
                     # Sometimes, we are lied to about the data processing step.
-                    lower_file_url = supplementary_file_url.lower()
                     if (
                         ".cel" in lower_file_url
                         or ("_non_normalized.txt" in lower_file_url)
@@ -479,7 +484,7 @@ def create_experiment_and_samples_from_api(
         return experiment_object, created_samples
 
     def discover_experiment_and_samples(self) -> (Experiment, List[Sample]):
-        """ Dispatches the surveyor, returns the results """
+        """Dispatches the surveyor, returns the results"""
 
         experiment_accession_code = SurveyJobKeyValue.objects.get(
             survey_job_id=self.survey_job.id, key__exact="experiment_accession_code"
diff --git a/scripts/run_full_pipeline.py b/scripts/run_full_pipeline.py
old mode 100644
new mode 100755
--- a/scripts/run_full_pipeline.py
+++ b/scripts/run_full_pipeline.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 import argparse
 import json
 import subprocess
@@ -49,7 +51,7 @@ def run_job(job):
         elif job_name == "SALMON":
             image_name = "salmon"
         elif job_name == "ILLUMINA_TO_PCL":
-            image_name == "illumina"
+            image_name = "illumina"
         elif job_name in ["TRANSCRIPTOME_INDEX_LONG", "TRANSCRIPTOME_INDEX_SHORT"]:
             image_name = "transcriptome"
         elif job_name == "NO_OP":
diff --git a/workers/data_refinery_workers/processors/illumina.py b/workers/data_refinery_workers/processors/illumina.py
--- a/workers/data_refinery_workers/processors/illumina.py
+++ b/workers/data_refinery_workers/processors/illumina.py
@@ -3,6 +3,7 @@
 import os
 import re
 import subprocess
+import tempfile
 from typing import Dict
 
 from django.utils import timezone
@@ -33,7 +34,7 @@
 
 
 def _prepare_files(job_context: Dict) -> Dict:
-    """Adds the keys "input_file_path" and "output_file_path" to
+    """Adds the keys "sanitized_file_path" and "output_file_path" to
     job_context so everything is prepared for processing.
     """
     # All files for the job are in the same directory.
@@ -43,31 +44,140 @@ def _prepare_files(job_context: Dict) -> Dict:
     os.makedirs(job_context["work_dir"], exist_ok=True)
 
     original_file = job_context["original_files"][0]
-    sanitized_filename = original_file.absolute_file_path.split("/")[-1] + ".sanitized"
-    job_context["input_file_path"] = job_context["work_dir"] + sanitized_filename
+    job_context["input_file_path"] = original_file.absolute_file_path
+
+    # This should not happen, but if it does I would rather know about it here,
+    # whereas before we would get random failures later down the pipeline
+    if not job_context["input_file_path"].endswith(".txt"):
+        logger.error(
+            "Input file doesn't have a suffix we recognize, probably an invalid format",
+            input_file=original_file.absolute_file_path,
+        )
+        job_context["job"].failure_reason = "Couldn't recognize the input file format"
+        job_context["success"] = False
+        job_context["job"].no_retry = True
+        return job_context
+
+    sanitized_filename = job_context["input_file_path"].split("/")[-1] + ".sanitized"
+    job_context["sanitized_file_path"] = job_context["work_dir"] + sanitized_filename
 
     new_filename = original_file.absolute_file_path.split("/")[-1].replace(".txt", ".PCL")
     job_context["output_file_path"] = job_context["work_dir"] + new_filename
 
-    # Sanitize this file so R doesn't choke.
-    # Some have comments, some have non-comment-comments.
-    with open(original_file.absolute_file_path, "r") as file_input:
-        with open(job_context["input_file_path"], "w") as file_output:
+    return job_context
+
+
+def _detect_encoding(job_context: Dict) -> Dict:
+    """Some Illumina files are not encoded using utf-8, so we need to use
+    `file` to detect their encoding"""
+
+    try:
+        encoding = subprocess.check_output(
+            ["file", "--brief", "--mime-encoding", job_context["input_file_path"],],
+            encoding="utf-8",
+        ).strip()
+    except subprocess.CalledProcessError as e:
+        logger.exception(
+            "Failed to detect the input file's encoding",
+            processor_job=job_context["job_id"],
+            input_file=job_context["input_file_path"],
+        )
+        job_context["job"].failure_reason = "Failed to detect the input file's encoding"
+        job_context["success"] = False
+        job_context["job"].no_retry = True
+
+    if encoding not in ["us-ascii", "utf-8", "iso-8859-1"]:
+        logger.error(
+            "Input file has unrecognized encoding",
+            input_file=job_context["input_file_path"],
+            encoding=encoding,
+        )
+        job_context["job"].failure_reason = f"Input file has unrecognized encoding {encoding}"
+        job_context["success"] = False
+        job_context["job"].no_retry = True
+
+    job_context["encoding"] = encoding
+
+    return job_context
+
+
+def _sanitize_input_file(job_context: Dict) -> Dict:
+    """Remove all of the SOFT-specific extensions in the original file (see
+    https://www.ncbi.nlm.nih.gov/geo/info/soft.html) plus some extra things we
+    found that make R choke.  Also, some files aren't utf-8 encoded, so we will
+    re-encode them into utf-8."""
+
+    with open(job_context["input_file_path"], "r", encoding=job_context["encoding"]) as file_input:
+        with open(job_context["sanitized_file_path"], "w", encoding="utf-8") as file_output:
             for line in file_input:
-                if (
-                    "#" not in line
-                    and line.strip() != ""
-                    and line != "\n"
-                    and "\t" in line
-                    and line[0:3].upper() != "GSM"
-                    and line[0] != "'"
-                    and line[0] != '"'
-                    and line[0] != "!"
-                    and line[0] != "/"
-                    and line[0] != "<"
-                    and line[0] != "\t"
-                ):
+                HEADER_CHARS = ["#", "!", "^"]
+
+                stripped_line = line.strip()
+
+                if stripped_line == "":
+                    continue
+
+                # Sometimes we have a quoted header, so we need to check both
+                is_header = stripped_line[0] in HEADER_CHARS or (
+                    len(stripped_line) > 1
+                    and stripped_line[0] in ["'", '"']
+                    and stripped_line[1] in HEADER_CHARS
+                )
+
+                # There are some weird lines that start with accession
+                # codes that don't hold gene measurements
+                is_accession_line = stripped_line[0:3].upper() == "GSM"
+
+                if not is_header and not is_accession_line:
                     file_output.write(line)
+                    wrote_a_line = True
+
+    if not wrote_a_line:
+        logger.error(
+            "Filtered every line out of the input file", input_file=job_context["input_file_path"]
+        )
+        job_context["job"].failure_reason = "No valid rows detected in the input file"
+        job_context["success"] = False
+        job_context["job"].no_retry = True
+
+    return job_context
+
+
+def _convert_sanitized_to_tsv(job_context: Dict) -> Dict:
+    """Now that we have removed all of the SOFT-specific extensions, we are left
+    with some kind of csv/tsv/ssv that may or may not have quoted strings. To
+    make things easier to parse in R, we will try to sniff these features and
+    output a uniform format for the R code to read"""
+
+    _, tmpfile = tempfile.mkstemp(
+        suffix=".txt", dir=os.path.dirname(job_context["sanitized_file_path"])
+    )
+    with open(job_context["sanitized_file_path"], "r") as file_input:
+        with open(tmpfile, "w") as file_output:
+            dialect = csv.Sniffer().sniff(file_input.read(16384), delimiters="\t ,")
+            file_input.seek(0)
+            reader = csv.reader(file_input, dialect=dialect)
+            writer = csv.writer(file_output, delimiter="\t")
+
+            reader_iter = iter(reader)
+
+            headers = next(reader)
+            first_content = next(reader)
+
+            # Sometimes input files have one less header than they have rows,
+            # which means that we should interpret the first column as ID_REF.
+            # The rest of our code expects this explicit header, though, so we
+            # will insert it here.
+            if len(headers) == len(first_content) - 1:
+                headers = ["ID_REF", *headers]
+
+            writer.writerow(headers)
+            writer.writerow(first_content)
+
+            for row in reader_iter:
+                writer.writerow(row)
+
+    os.rename(tmpfile, job_context["sanitized_file_path"])
 
     return job_context
 
@@ -99,7 +209,7 @@ def _detect_columns(job_context: Dict) -> Dict:
         detectionPval: a string which identifies Pvalue columns
     """
     try:
-        input_file = job_context["input_file_path"]
+        input_file = job_context["sanitized_file_path"]
         headers = None
         with open(input_file, "r") as tsv_in:
             tsv_in = csv.reader(tsv_in, delimiter="\t")
@@ -109,11 +219,25 @@ def _detect_columns(job_context: Dict) -> Dict:
 
         # Ex GSE45331_non-normalized.txt
         predicted_header = 0
-        if headers[0].upper() in ["TARGETID", "TARGET_ID"]:
-            predicted_header = 1
+
+        # Some files start with blank columns, so let's skip past those
+        while headers[predicted_header] == "":
+            predicted_header += 1
+
+        if headers[predicted_header].upper() in ["TARGETID", "TARGET_ID"]:
+            predicted_header += 1
 
         # First the probe ID column
-        if headers[predicted_header].upper() not in [
+        if headers[predicted_header].upper().strip() == "ILLUMICODE":
+            logger.error(
+                "Tried to process a beadTypeFile.txt, which we don't support",
+                input_file=job_context["sanitized_file_path"],
+            )
+            job_context["job"].failure_reason = "Unsupported filetype 'beadTypeFile.txt'"
+            job_context["success"] = False
+            job_context["job"].no_retry = True
+            return job_context
+        elif headers[predicted_header].upper().strip() not in [
             "ID_REF",
             "PROBE_ID",
             "IDREF",
@@ -127,7 +251,7 @@ def _detect_columns(job_context: Dict) -> Dict:
                 "Could not find any ID column in headers "
                 + str(headers)
                 + " for file "
-                + job_context["input_file_path"]
+                + job_context["sanitized_file_path"]
             )
             job_context["success"] = False
             return job_context
@@ -137,8 +261,8 @@ def _detect_columns(job_context: Dict) -> Dict:
         # Then check to make sure a detection pvalue exists, which is always(?) some form of
         # 'Detection Pval'
         for header in headers:
-            # check if header contains something like "detection pval"
-            pvalue_header = re.search(r"(detection)(\W?)(pval\w*)", header, re.IGNORECASE)
+            # check if header contains something like "detection pval" or "detection_pval"
+            pvalue_header = re.search(r"(detection)([\W_]?)(pval\w*)", header, re.IGNORECASE)
             if pvalue_header:
                 break
         else:
@@ -182,7 +306,11 @@ def _detect_columns(job_context: Dict) -> Dict:
                     continue
 
                 if (
-                    sample.title.upper() in header.upper()
+                    header != ""
+                    and (
+                        sample.title.upper() in header.upper()
+                        or sample.title.upper().endswith("_" + header.upper())
+                    )
                     and "BEAD" not in header.upper()
                     and "NARRAYS" not in header.upper()
                     and "ARRAY_STDEV" not in header.upper()
@@ -196,12 +324,25 @@ def _detect_columns(job_context: Dict) -> Dict:
                 column_ids.add(offset)
                 continue
 
+        if len(column_ids) == 0:
+            job_context[
+                "job"
+            ].failure_reason = f"could not find columns ids in {job_context['sanitized_file_path']}"
+            job_context["success"] = False
+            logger.error("Could not find columns ids in " + job_context["sanitized_file_path"])
+            job_context["job"].no_retry = True
+            return job_context
+
         job_context["columnIds"] = ",".join(map(lambda id: str(id), column_ids))
     except Exception as e:
-        job_context["job"].failure_reason = str(e)
+        job_context[
+            "job"
+        ].failure_reason = (
+            f"failure to extract columns in {job_context['sanitized_file_path']}: {e}"
+        )
         job_context["success"] = False
         logger.exception(
-            "Failed to extract columns in " + job_context["input_file_path"], exception=str(e)
+            "Failed to extract columns in " + job_context["sanitized_file_path"], exception=str(e)
         )
         job_context["job"].no_retry = True
         return job_context
@@ -245,9 +386,10 @@ def _detect_platform(job_context: Dict) -> Dict:
                     "--platform",
                     platform,
                     "--inputFile",
-                    job_context["input_file_path"],
+                    job_context["sanitized_file_path"],
                     "--column",
-                    job_context["probeId"],
+                    # R strips column names, so we have to too
+                    job_context["probeId"].strip(),
                 ]
             )
 
@@ -320,35 +462,45 @@ def _run_illumina(job_context: Dict) -> Dict:
             "--vanilla",
             "/home/user/data_refinery_workers/processors/illumina.R",
             "--probeId",
-            job_context["probeId"],
+            # R strips column names, so we have to too
+            job_context["probeId"].strip(),
             "--expression",
             job_context["columnIds"],
             "--platform",
             job_context["platform"],
             "--inputFile",
-            job_context["input_file_path"],
+            job_context["sanitized_file_path"],
             "--outputFile",
             job_context["output_file_path"],
             "--cores",
             str(multiprocessing.cpu_count()),
         ]
 
-        subprocess.check_output(formatted_command)
+        output = subprocess.check_output(formatted_command).decode()
+        logger.debug(f"illumina.R ran successfully with output '{output}'")
 
         job_context["formatted_command"] = " ".join(formatted_command)
 
         job_context["time_end"] = timezone.now()
 
-    except Exception as e:
+    except subprocess.CalledProcessError as e:
         error_template = (
             "Encountered error in R code while running illumina.R"
             " pipeline during processing of {0}: {1}"
         )
-        error_message = error_template.format(job_context["input_file_path"], str(e))
+        error_message = error_template.format(job_context["sanitized_file_path"], str(e))
         logger.error(error_message, processor_job=job_context["job_id"])
+        logger.debug(f"illumina.R failed with output '{e.output.decode()}'")
         job_context["job"].failure_reason = error_message
         job_context["success"] = False
 
+    except Exception as e:
+        logger.exception(
+            "Exception raised while running illumina.R", processor_job=job_context["job_id"]
+        )
+        job_context["job"].failure_reason = f"Exception raised while running illumina.R: '{e}'"
+        job_context["success"] = False
+
     return job_context
 
 
@@ -358,6 +510,11 @@ def _get_sample_for_column(column: str, job_context: Dict) -> Sample:
         return job_context["samples"].get(title=column)
     except Sample.DoesNotExist:
         pass
+    # Or maybe they named their samples with a common prefix
+    try:
+        return job_context["samples"].get(title__iendswith="_" + column)
+    except Sample.DoesNotExist:
+        pass
 
     # If the column name is not the title, maybe they used the convention
     # <SAMPLE_TITLE>(.AVG)?_Signal
@@ -455,13 +612,23 @@ def _create_result_objects(job_context: Dict) -> Dict:
     return job_context
 
 
-def illumina_to_pcl(job_id: int) -> None:
+def illumina_to_pcl(job_id: int, cleanup=None) -> None:
     pipeline = Pipeline(name=PipelineEnum.ILLUMINA.value)
+    initial_job_context = {"job_id": job_id, "pipeline": pipeline}
+
+    # When running the tests, don't clean up original files so we don't have to
+    # keep downloading them.
+    if cleanup is not None:
+        initial_job_context["cleanup"] = cleanup
+
     return utils.run_pipeline(
-        {"job_id": job_id, "pipeline": pipeline},
+        initial_job_context,
         [
             utils.start_job,
             _prepare_files,
+            _detect_encoding,
+            _sanitize_input_file,
+            _convert_sanitized_to_tsv,
             _detect_columns,
             _detect_platform,
             _run_illumina,


Test patch:
diff --git a/workers/data_refinery_workers/processors/test_illumina.py b/workers/data_refinery_workers/processors/test_illumina.py
--- a/workers/data_refinery_workers/processors/test_illumina.py
+++ b/workers/data_refinery_workers/processors/test_illumina.py
@@ -1,31 +1,35 @@
+import csv
 import os
+import os.path
 import shutil
+import tempfile
+from typing import Dict
 
 from django.test import TestCase, tag
 
+from data_refinery_common.enums import PipelineEnum
 from data_refinery_common.models import (
     Organism,
     OriginalFile,
     OriginalFileSampleAssociation,
+    Pipeline,
     ProcessorJob,
     ProcessorJobOriginalFileAssociation,
     Sample,
     SampleAnnotation,
 )
-from data_refinery_workers.processors import illumina
+from data_refinery_workers.processors import illumina, testing_utils, utils
 
 
-def prepare_illumina_job(organism):
+def prepare_illumina_job(job_info: Dict) -> ProcessorJob:
     pj = ProcessorJob()
     pj.pipeline_applied = "ILLUMINA_TO_PCL"
     pj.save()
 
     og_file = OriginalFile()
-    og_file.source_filename = "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE22nnn/GSE22427/suppl/GSE22427%5Fnon%2Dnormalized%2Etxt.gz"
-    og_file.filename = "GSE22427_non-normalized.txt"
-    og_file.absolute_file_path = (
-        "/home/user/data_store/raw/TEST/ILLUMINA/GSE22427_non-normalized.txt"
-    )
+    og_file.source_filename = job_info["source_filename"]
+    og_file.filename = job_info["filename"]
+    og_file.absolute_file_path = job_info["absolute_file_path"]
     og_file.is_downloaded = True
     og_file.save()
 
@@ -34,31 +38,29 @@ def prepare_illumina_job(organism):
     assoc1.processor_job = pj
     assoc1.save()
 
-    sample_names = [
-        "LV-C&si-Control-1",
-        "LV-C&si-Control-2",
-        "LV-C&si-Control-3",
-        "LV-C&si-EZH2-1",
-        "LV-C&si-EZH2-2",
-        "LV-C&si-EZH2-3",
-        "LV-EZH2&si-EZH2-1",
-        "LV-EZH2&si-EZH2-2",
-        "LV-EZH2&si-EZH2-3",
-        "LV-T350A&si-EZH2-1",
-        "LV-T350A&si-EZH2-2",
-        "LV-T350A&si-EZH2-3",
-    ]
+    for s in job_info["samples"]:
+        # For convenience, if you give a list of strings we'll just use the
+        # strings as both titles and accessions.
+        annotation = None
+        if type(s) == str:
+            accession_code = s
+            title = s
+        elif type(s) == tuple and list(map(type, s)) == [str, str]:
+            accession_code, title = s
+        elif type(s) == tuple and list(map(type, s)) == [str, str, dict]:
+            accession_code, title, annotation = s
+        else:
+            raise ValueError(f"Invalid sample type for sample {s}")
 
-    for name in sample_names:
         sample = Sample()
-        sample.accession_code = name
-        sample.title = name
-        sample.organism = organism
+        sample.accession_code = accession_code
+        sample.title = title
+        sample.organism = job_info["organism"]
         sample.save()
 
         sa = SampleAnnotation()
         sa.sample = sample
-        sa.data = {"description": [name]}
+        sa.data = annotation if annotation is not None else {"description": [title]}
         sa.is_ccdl = False
         sa.save()
 
@@ -67,15 +69,87 @@ def prepare_illumina_job(organism):
         sample_assoc.sample = sample
         sample_assoc.save()
 
-    sample = Sample.objects.get(title="LV-T350A&si-EZH2-3")
-    sample.title = "ignoreme_for_description"
-    sample.accession_code = "ignoreme_for_description"
-    sample.save()
-
     return pj
 
 
-class IlluminaToPCLTestCase(TestCase):
+def _make_original_file_with_contents(contents: str) -> OriginalFile:
+    _, path = tempfile.mkstemp(suffix=".txt")
+    with open(path, "w") as f:
+        f.write(contents)
+
+    og_file = OriginalFile()
+    og_file.source_filename = path
+    og_file.filename = os.path.basename(path)
+    og_file.absolute_file_path = os.path.realpath(path)
+    og_file.is_downloaded = True
+    og_file.save()
+
+    return og_file
+
+
+def _try_sanitizing_file(file: str) -> str:
+    pj = ProcessorJob()
+    pj.pipeline_applied = "ILLUMINA_TO_PCL"
+    pj.save()
+
+    og_file = _make_original_file_with_contents(file)
+
+    job_context = illumina._prepare_files({"job_id": pj.pk, "original_files": [og_file], "job": pj})
+    job_context = illumina._detect_encoding(job_context)
+    return illumina._sanitize_input_file(job_context)
+
+
+# Save this experiment separately (sans organism) because we need it for multiple tests
+GSE22427 = {
+    "source_filename": "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE22nnn/GSE22427/suppl/GSE22427%5Fnon%2Dnormalized%2Etxt.gz",
+    "filename": "GSE22427_non-normalized.txt",
+    "absolute_file_path": "/home/user/data_store/raw/TEST/ILLUMINA/GSE22427_non-normalized.txt",
+    "samples": [
+        "LV-C&si-Control-1",
+        "LV-C&si-Control-2",
+        "LV-C&si-Control-3",
+        "LV-C&si-EZH2-1",
+        "LV-C&si-EZH2-2",
+        "LV-C&si-EZH2-3",
+        "LV-EZH2&si-EZH2-1",
+        "LV-EZH2&si-EZH2-2",
+        "LV-EZH2&si-EZH2-3",
+        "LV-T350A&si-EZH2-1",
+        "LV-T350A&si-EZH2-2",
+        "LV-T350A&si-EZH2-3",
+    ],
+}
+
+GSE22427_HEADER = [
+    "ID_REF",
+    "LV-C&si-Control-1",
+    "Detection Pval",
+    "LV-C&si-Control-2",
+    "Detection Pval",
+    "LV-C&si-Control-3",
+    "Detection Pval",
+    "LV-C&si-EZH2-1",
+    "Detection Pval",
+    "LV-C&si-EZH2-2",
+    "Detection Pval",
+    "LV-C&si-EZH2-3",
+    "Detection Pval",
+    "LV-EZH2&si-EZH2-1",
+    "Detection Pval",
+    "LV-EZH2&si-EZH2-2",
+    "Detection Pval",
+    "LV-EZH2&si- EZH2-3",
+    "Detection Pval",
+    "LV-T350A&si-EZH2-1",
+    "Detection Pval",
+    "LV- T350A&si-EZH2-2",
+    "Detection Pval",
+    "LV-T350A&si-EZH2-3",
+    "Detection Pval",
+]
+
+
+class IlluminaToPCLTestCase(TestCase, testing_utils.ProcessorJobTestCaseMixin):
     @tag("illumina")
     def test_illumina_to_pcl(self):
         """Most basic Illumina to PCL test"""
@@ -83,9 +157,17 @@ def test_illumina_to_pcl(self):
         organism = Organism(name="HOMO_SAPIENS", taxonomy_id=9606, is_scientific_name=True)
         organism.save()
 
-        job = prepare_illumina_job(organism)
-        final_context = illumina.illumina_to_pcl(job.pk)
-        self.assertTrue(final_context["success"])
+        job = prepare_illumina_job({**GSE22427, "organism": organism})
+
+        # Remove the title of one of the samples to make sure that we can still
+        # find its detection column using the description given as an annotation
+        sample = Sample.objects.get(title="LV-T350A&si-EZH2-3")
+        sample.title = "ignoreme_for_description"
+        sample.accession_code = "ignoreme_for_description"
+        sample.save()
+
+        final_context = illumina.illumina_to_pcl(job.pk, cleanup=False)
+        self.assertSucceeded(job)
 
         for sample in final_context["samples"]:
             smashme = sample.get_most_recent_smashable_result_file()
@@ -102,8 +184,8 @@ def test_bad_illumina_detection(self):
         organism = Organism(name="RATTUS_NORVEGICUS", taxonomy_id=9606, is_scientific_name=True)
         organism.save()
 
-        job = prepare_illumina_job(organism)
-        final_context = illumina.illumina_to_pcl(job.pk)
+        job = prepare_illumina_job({**GSE22427, "organism": organism})
+        final_context = illumina.illumina_to_pcl(job.pk, cleanup=False)
         self.assertTrue(final_context["abort"])
 
         # Cleanup after the job since it won't since we aren't running in cloud.
@@ -113,51 +195,21 @@ def test_bad_illumina_detection(self):
     def test_good_detection(self):
         """GSE54661 appears to be mislabled (illuminaHumanv4) on GEO. Shows our detector works."""
 
-        pj = ProcessorJob()
-        pj.pipeline_applied = "ILLUMINA_TO_PCL"
-        pj.save()
-
-        og_file = OriginalFile()
-        og_file.source_filename = "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE54nnn/GSE54661/suppl/GSE54661%5Fnon%5Fnormalized%2Etxt%2Egz"
-        og_file.filename = "GSE54661_non_normalized.txt"
-        og_file.absolute_file_path = (
-            "/home/user/data_store/raw/TEST/ILLUMINA/GSE54661_non_normalized.txt"
-        )
-        og_file.is_downloaded = True
-        og_file.save()
-
-        assoc1 = ProcessorJobOriginalFileAssociation()
-        assoc1.original_file = og_file
-        assoc1.processor_job = pj
-        assoc1.save()
-
         organism = Organism(name="HOMO_SAPIENS", taxonomy_id=9606, is_scientific_name=True)
         organism.save()
 
-        sample = Sample()
-        sample.accession_code = "ABCD-1234"
-        sample.title = "CB CD34+ hypoxia"
-        sample.organism = organism
-        sample.save()
-
-        sample_assoc = OriginalFileSampleAssociation()
-        sample_assoc.original_file = og_file
-        sample_assoc.sample = sample
-        sample_assoc.save()
-
-        sample2 = Sample()
-        sample2.accession_code = "ABCD-1235"
-        sample2.title = "CB CD34+ normoxia"
-        sample2.organism = organism
-        sample2.save()
-
-        sample_assoc2 = OriginalFileSampleAssociation()
-        sample_assoc2.original_file = og_file
-        sample_assoc2.sample = sample2
-        sample_assoc2.save()
+        pj = prepare_illumina_job(
+            {
+                "source_filename": "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE54nnn/GSE54661/suppl/GSE54661%5Fnon%5Fnormalized%2Etxt%2Egz",
+                "filename": "GSE54661_non_normalized.txt",
+                "absolute_file_path": "/home/user/data_store/raw/TEST/ILLUMINA/GSE54661_non_normalized.txt",
+                "organism": organism,
+                "samples": [("ABCD-1234", "CB CD34+ hypoxia"), ("ABCD-1235", "CB CD34+ normoxia")],
+            }
+        )
 
-        final_context = illumina.illumina_to_pcl(pj.pk)
-        self.assertTrue(final_context["success"])
+        final_context = illumina.illumina_to_pcl(pj.pk, cleanup=False)
+        self.assertSucceeded(pj)
         self.assertEqual(final_context["platform"], "illuminaHumanv3")
 
         for key in final_context["samples"][0].sampleannotation_set.all()[0].data.keys():
@@ -165,5 +217,336 @@ def test_good_detection(self):
                 key in ["detected_platform", "detection_percentage", "mapped_percentage"]
             )
 
+        for sample in final_context["samples"]:
+            smashme = sample.get_most_recent_smashable_result_file()
+            self.assertTrue(os.path.exists(smashme.absolute_file_path))
+
         # Cleanup after the job since it won't since we aren't running in cloud.
         shutil.rmtree(final_context["work_dir"], ignore_errors=True)
+
+    @tag("illumina")
+    def test_illumina_latin1_input(self):
+        """Test a latin1-encoded Illumina file.
+
+        GSE106321 is encoded in latin1 and uses μ in the title of some
+        columns, so preparing the file would cause a UnicodeParseError. Make
+        sure that doesn't happen any more.
+        """
+
+        organism = Organism(name="HOMO_SAPIENS", taxonomy_id=9606, is_scientific_name=True)
+        organism.save()
+
+        pj = prepare_illumina_job(
+            {
+                "source_filename": "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE106nnn/GSE106321/suppl/GSE106321_non-normalized.txt.gz",
+                "filename": "GSE106321_non-normalized.txt",
+                "absolute_file_path": "/home/user/data_store/raw/TEST/ILLUMINA/GSE106321_non-normalized.txt",
+                "organism": organism,
+                "samples": [
+                    (
+                        "GSM2835938",
+                        "A375 + 24h vem (3µM) 2",
+                        {"description": ["A375 + 24h vem (3µM) 2"]},
+                    ),
+                    (
+                        "GSM2835937",
+                        "A375 + 24h vem (3µM) 1",
+                        {"description": ["A375 + 24h vem (3µM) 1"]},
+                    ),
+                    (
+                        "GSM2835936",
+                        "A375 + 24h vem (3µM)",
+                        {"description": ["A375 + 24h vem (3µM)"]},
+                    ),
+                    ("GSM2835935", "A375 + 24h DMSO 2", {"description": ["A375 + 24h DMSO 2"]}),
+                    ("GSM2835934", "A375+ 24h DMSO 1", {"description": ["A375+ 24h DMSO 1"]}),
+                    ("GSM2835933", "A375 + 24h DMSO", {"description": ["A375 + 24h DMSO"]}),
+                ],
+            }
+        )
+
+        final_context = illumina.illumina_to_pcl(pj.pk, cleanup=False)
+        # XXX: For now, this processor job fails, but we want to make sure that it fails in the right place
+        # See https://github.com/AlexsLemonade/refinebio/issues/2870 for why it is failing.
+        self.assertFailed(
+            pj, "Encountered error in R code while running illumina.R pipeline during processing"
+        )
+
+        # Make sure that the input is now utf-8 encoded and has the right headers.
+
+        # Trying to open a latin1 file as utf-8 would cause an
+        # exception to be thrown, so if opening succeeds we can assume the encoding succeeded.
+        with open(final_context["sanitized_file_path"], "r", encoding="utf-8") as f:
+            reader = csv.reader(f, delimiter="\t")
+
+            # Check the headers to make sure that the mu was correctly re-encoded
+            headers = next(reader)
+            self.assertEqual(
+                headers,
+                [
+                    "ID_REF",
+                    "A375 + 24h DMSO",
+                    "Detection Pval",
+                    "A375+ 24h DMSO 1",
+                    "Detection Pval",
+                    "A375 + 24h DMSO 2",
+                    "Detection Pval",
+                    "A375 + 24h vem (3µM)",
+                    "Detection Pval",
+                    "A375 + 24h vem (3µM) 1",
+                    "Detection Pval",
+                    "A375 + 24h vem (3µM) 2",
+                    "Detection Pval",
+                    "",
+                    "",
+                    "",
+                    "",
+                    "",
+                    "",
+                ],
+            )
+
+    @tag("illumina")
+    def test_illumina_quoted_row_names(self):
+        organism = Organism(name="HOMO_SAPIENS", taxonomy_id=9606, is_scientific_name=True)
+        organism.save()
+
+        pj = prepare_illumina_job(
+            {
+                "source_filename": "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE33nnn/GSE33814/suppl/GSE33814%5Fnon%2Dnormalized%2Etxt%2Egz",
+                # Some of the columns are trimmed to save space and time
+                "filename": "GSE33814_trimmed_non-normalized.txt",
+                "absolute_file_path": "/home/user/data_store/raw/TEST/ILLUMINA/GSE33814_trimmed_non-normalized.txt",
+                "organism": organism,
+                "samples": [("GSM836222", "IMGUS_32"), ("GSM836223", "IMGUS_33"),],
+            }
+        )
+
+        final_context = illumina.illumina_to_pcl(pj.pk, cleanup=False)
+        self.assertSucceeded(pj)
+
+        # Make sure that the row names are no longer quoted after sanitizing the file
+        def assertNotQuoted(string: str):
+            string = string.strip()
+            self.assertNotEqual(string[0], '"')
+            self.assertNotEqual(string[-1], '"')
+
+        with open(final_context["sanitized_file_path"], "r") as f:
+            reader = csv.reader(f, delimiter="\t")
+
+            headers = next(reader)
+            for header in headers:
+                assertNotQuoted(header)
+
+            # Also make sure the probe IDs aren't qutoed
+            first_row = next(reader)
+            assertNotQuoted(first_row[0])
+
+    @tag("illumina")
+    def test_illumina_rows_starting_with_whitespace(self):
+        organism = Organism(name="HOMO_SAPIENS", taxonomy_id=9606, is_scientific_name=True)
+        organism.save()
+
+        pj = prepare_illumina_job(
+            {
+                "source_filename": "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE112nnn/GSE112517/suppl/GSE112517_non-normalized.txt.gz",
+                "filename": "GSE112517_non-normalized.txt",
+                "absolute_file_path": "/home/user/data_store/raw/TEST/ILLUMINA/GSE112517_non-normalized.txt",
+                "organism": organism,
+                "samples": [
+                    (
+                        "GSM3071991",
+                        "MCF-7 KLHDC7B siRNA knockdown control",
+                        {"description": ["SAMPLE 1"],},
+                    ),
+                    (
+                        "GSM3071992",
+                        "MCF-7 KLHDC7B siRNA knockdown",
+                        {"description": ["SAMPLE 2"],},
+                    ),
+                ],
+            }
+        )
+
+        final_context = illumina.illumina_to_pcl(pj.pk, cleanup=False)
+        self.assertSucceeded(pj)
+
+    @tag("illumina")
+    def test_illumina_space_separated(self):
+        organism = Organism(name="HOMO_SAPIENS", taxonomy_id=9606, is_scientific_name=True)
+        organism.save()
+
+        pj = prepare_illumina_job(
+            {
+                "source_filename": "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE48nnn/GSE48023/suppl/GSE48023%5Fnon%2Dnormalized%2Etxt%2Egz",
+                # Some of the columns are trimmed to save space and time
+                "filename": "GSE48023_trimmed_non-normalized.txt",
+                "absolute_file_path": "/home/user/data_store/raw/TEST/ILLUMINA/GSE48023_trimmed_non-normalized.txt",
+                "organism": organism,
+                "samples": [
+                    ("GSM1165512", "WholeBloodRNA_IN0242_Day0"),
+                    ("GSM1165513", "WholeBloodRNA_IN0242_Day1"),
+                    ("GSM1165514", "WholeBloodRNA_IN0242_Day14"),
+                    ("GSM1165515", "WholeBloodRNA_IN0242_Day3"),
+                    ("GSM1165516", "WholeBloodRNA_IN0243_Day0"),
+                ],
+            }
+        )
+
+        final_context = illumina.illumina_to_pcl(pj.pk, cleanup=False)
+        self.assertSucceeded(pj)
+
+        # Assert that the sanitized file is tab-separated (by reading it as a
+        # TSV and making sure it has 11 headers) and has an extra ID_REF header
+        with open(final_context["sanitized_file_path"], "r") as f:
+            reader = csv.reader(f, delimiter="\t")
+
+            headers = next(reader)
+
+            # ID_REF + 5 observations + 5 p-values
+            self.assertEqual(len(headers), 11)
+            self.assertEqual(headers[0], "ID_REF")
+
+    @tag("illumina")
+    def test_illumina_no_pvalue(self):
+        """This experiment should fail because it has no p-value columns, so
+        make sure it fails at that stage of the processing"""
+        organism = Organism(name="HOMO_SAPIENS", taxonomy_id=9606, is_scientific_name=True)
+        organism.save()
+
+        pj = prepare_illumina_job(
+            {
+                "source_filename": "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE41nnn/GSE41355/suppl/GSE41355%5Fnon%2Dnormalized%2Etxt%2Egz",
+                "filename": "GSE41355_non-normalized.txt",
+                "absolute_file_path": "/home/user/data_store/raw/TEST/ILLUMINA/GSE41355_non-normalized.txt",
+                "organism": organism,
+                "samples": [("GSM1015436", "IRF3/7 DKO 2"),],
+            }
+        )
+
+        final_context = illumina.illumina_to_pcl(pj.pk, cleanup=False)
+
+        self.assertFailed(pj, "Could not detect PValue column!")
+
+    @tag("illumina")
+    def test_illumina_id_ref_column_with_whitespace(self):
+        """This test case tests the issue brought up in
+        https://github.com/alexslemonade/refinebio/issues/1560
+        where an ID_REF column would not be detected because the column name had a trailing space
+        """
+
+        organism = Organism(name="HOMO_SAPIENS", taxonomy_id=9606, is_scientific_name=True)
+        organism.save()
+
+        pj = prepare_illumina_job(
+            {
+                "source_filename": "ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE100nnn/GSE100301/suppl/GSE100301%5Fnon%2Dnormalized%2Etxt%2Egz",
+                "filename": "GSE100301_non-normalized.txt",
+                "absolute_file_path": "/home/user/data_store/raw/TEST/ILLUMINA/GSE100301_non-normalized.txt",
+                "organism": organism,
+                "samples": [
+                    ("GSM2677583", "22Rv1-tetO-Gal4, replicate 1", {"description": ["SAMPLE 1"],},),
+                ],
+            }
+        )
+
+        final_context = illumina.illumina_to_pcl(pj.pk, cleanup=False)
+        self.assertSucceeded(pj)
+
+    @tag("illumina")
+    def test_sanitize_file_hash_header(self):
+        contents = """
+# The identifiers in the ID_REF column must match the identifiers in the ID column of the referenced platform (GPLxxxx).
+# The Matrix table should include normalized (scaled) signal count data and detection p-value
+"# Values that should be disregarded may either be left blank or labeled as ""null""."
+Index	Data
+"""
+        results = _try_sanitizing_file(contents)
+
+        with open(results["sanitized_file_path"], "r") as f:
+            self.assertEqual(f.read(), "Index	Data\n")
+
+        os.remove(results["sanitized_file_path"])
+
+    @tag("illumina")
+    def test_sanitize_file_GSM_footer(self):
+        """Some samples have this weird footer that we want to filter out"""
+        contents = """
+Index	Data
+GSM674408	1523532074_A
+GSM674409	1523532074_B
+GSM674410	1523532074_C
+GSM674411	1523532074_D
+GSM674412	1523532074_E
+GSM674413	1523532074_F
+"""
+        results = _try_sanitizing_file(contents)
+
+        with open(results["sanitized_file_path"], "r") as f:
+            self.assertEqual(f.read(), "Index	Data\n")
+
+        os.remove(results["sanitized_file_path"])
+
+    @tag("illumina")
+    def test_sanitize_file_soft_example(self):
+        # Example file from https://www.ncbi.nlm.nih.gov/geo/info/soft.html#examples
+        contents = """
+^SAMPLE = Control Embyronic Stem Cell Replicate 1
+!Sample_label_protocol_ch1 = 10 痢 of total RNA were primed with 2 痞 of 100 然 T16N2 DNA primer at 70蚓 for 10 min, then reversed transcribed at 42蚓 for 1 h in the presence of 400 U SuperScript II RTase (Invitrogen), and 100 然 each dATP, dTTP, dGTP, with 25 然 dCTP, 25 然 Cy5-labeled dCTP (NEN Life Science, Boston, MA), and RNase inhibitor (Invitrogen). RNA was then degraded with RNase A, and labeled cDNAs were purified using QIAquick PCR columns (Qiagen).
+!Sample_organism_ch2 = Mus musculus
+!Sample_platform_id = GPL3759
+#ID_REF =
+#VALUE = log2(REDsignal/GREENsignal) per feature (processed signals used).
+#LogRatioError = error of the log ratio calculated according to the error model chosen.
+!sample_table_begin
+ID_REF	VALUE	LogRatioError	PValueLogRatio	gProcessedSignal	rProcessedSignal
+1	-1.6274758	1.36E-01	6.41E-33	9.13E+03	2.15E+02
+2	0.1412248	1.34E+00	1.00E+00	4.14E+01	5.72E+01
+"""
+
+        results = _try_sanitizing_file(contents)
+
+        with open(results["sanitized_file_path"], "r") as f:
+            self.assertEqual(f.read().strip(), "\n".join(contents.splitlines()[-3:]))
+
+        os.remove(results["sanitized_file_path"])
+
+    @tag("illumina")
+    def test_detect_columns(self):
+        organism = Organism(name="HOMO_SAPIENS", taxonomy_id=9606, is_scientific_name=True)
+        organism.save()
+
+        job = prepare_illumina_job({**GSE22427, "organism": organism})
+
+        pipeline = Pipeline(name=PipelineEnum.ILLUMINA.value)
+
+        final_context = utils.run_pipeline(
+            {"job_id": job.id, "pipeline": pipeline},
+            [
+                utils.start_job,
+                illumina._prepare_files,
+                illumina._detect_encoding,
+                illumina._sanitize_input_file,
+                illumina._convert_sanitized_to_tsv,
+                illumina._detect_columns,
+            ],
+        )
+
+        self.assertNotEqual(final_context.get("success"), False)
+
+        # For this experiment, the probe ID is the first column
+        self.assertEqual(final_context.get("probeId"), GSE22427_HEADER[0])
+
+        expected_column_ids = ",".join(
+            map(
+                lambda t: str(t[0]),
+                filter(
+                    # For this header file, the samples all have the prefix LV-
+                    lambda t: t[1].startswith("LV-"),
+                    # We use start=1 here because the column IDs are formatted
+                    # for R code so they treat the header as a 1-indexed list
+                    enumerate(GSE22427_HEADER, start=1),
+                ),
+            )
+        )
+        self.assertEqual(final_context.get("columnIds"), expected_column_ids)
diff --git a/workers/data_refinery_workers/processors/testing_utils.py b/workers/data_refinery_workers/processors/testing_utils.py
--- a/workers/data_refinery_workers/processors/testing_utils.py
+++ b/workers/data_refinery_workers/processors/testing_utils.py
@@ -27,3 +27,32 @@ def assertMostlyAgrees(test_case: TestCase, expected_data: pd.Series, actual_dat
 
     (rho, _) = scipy.stats.spearmanr(expected_df.join(actual_df, how="inner"))
     test_case.assertGreater(rho, 0.99)
+
+
+class ProcessorJobTestCaseMixin:
+    def assertSucceeded(self, pj):
+        pj.refresh_from_db()
+
+        if pj.success:
+            return
+
+        if pj.failure_reason is not None:
+            msg = f"Processor job failed with reason '{pj.failure_reason}'"
+        else:
+            msg = f"Processor job failed without a given reason"
+
+        raise self.failureException(msg)
+
+    def assertFailed(self, pj, expected_reason=""):
+        pj.refresh_from_db()
+
+        if not pj.success and expected_reason in pj.failure_reason:
+            return
+
+        if pj.success:
+            raise self.failureException("Expected processor job to fail, but it succeeded")
+
+        raise self.failureException(
+            "Processor job failed, but for an incorrect reason.\n"
+            + f"We expected the reason to contain '{expected_reason}' but the actual reason was '{pj.failure_reason}'"
+        )
diff --git a/workers/run_tests.sh b/workers/run_tests.sh
--- a/workers/run_tests.sh
+++ b/workers/run_tests.sh
@@ -193,21 +193,32 @@ if [ -z "$tag" ] || [ "$tag" = "transcriptome" ]; then
 fi
 
 if [ -z "$tag" ] || [ "$tag" = "illumina" ]; then
-    # Illumina test file
-    ilu_file="GSE22427_non-normalized.txt"
     ilu_test_raw_dir="$volume_directory/raw/TEST/ILLUMINA"
-    if [ ! -e "$ilu_test_raw_dir/$ilu_file" ]; then
-        mkdir -p "$ilu_test_raw_dir"
-        echo "Downloading Illumina file for Illumina tests."
-        wget -q -O "$ilu_test_raw_dir/$ilu_file" \
-             "$test_data_repo/$ilu_file"
-    fi
-    ilu_file2="GSE54661_non_normalized.txt"
-    if [ ! -e "$ilu_test_raw_dir/$ilu_file2" ]; then
-        mkdir -p "$ilu_test_raw_dir"
-        echo "Downloading Illumina file 2 for Illumina tests."
-        wget -q -O "$ilu_test_raw_dir/$ilu_file2" \
-             "$test_data_repo/$ilu_file2"
+    ilu_files='GSE22427_non-normalized.txt GSE54661_non_normalized.txt GSE106321_non-normalized.txt
+GSE33814_trimmed_non-normalized.txt GSE112517_non-normalized.txt GSE48023_trimmed_non-normalized.txt
+GSE41355_non-normalized.txt GSE100301_non-normalized.txt'
+    mkdir -p "$ilu_test_raw_dir"
+
+    i=1
+    for ilu_file in $ilu_files; do
+        if [ ! -e "$ilu_test_raw_dir/$ilu_file" ]; then
+            echo "Downloading Illumina file $i for Illumina tests."
+            wget -q -O "$ilu_test_raw_dir/$ilu_file" \
+                "$test_data_repo/$ilu_file"
+        fi
+
+        i=$(( i + 1 ))
+    done
+    unset i
+
+
+    ilu_test_ref_dir="$volume_directory/raw/TEST/ILLUMINA/reference"
+    ilu_ref_file="Ad-Cre-2.AVG_Signal.tsv"
+    if [ ! -e "$ilu_test_ref_dir/$ilu_ref_file" ]; then
+        mkdir -p "$ilu_test_ref_dir"
+        echo "Downloading Illumin reference file for Illumina tests."
+        wget -q -O "$ilu_test_ref_dir/$ilu_ref_file" \
+             "$test_data_repo/$ilu_ref_file"
     fi
 fi