TITLE: SCRMshaw: supervised cis-regulatory module prediction for insect genomes
AUTHORS: Hasiba Asma, Luna Liu, Marc S. Halfon

[DESCRIPTION]
As the number of sequenced insect genomes continues to grow, there is a pressing need for rapid and accurate annotation of their regulatory component. SCRMshaw is a computational tool designed to predict cis-regulatory modules (CRMs) in the genomes of various insect species (Kantorovitz 2009; Kazemian 2011; Asma and Halfon 2019; Asma 2024).
One of the key advantages of SCRMshaw is its accessibility. It requires minimal resources—just a genome sequence and training data from known Drosophila CRMs, which are readily available for download. Even users with modest computational skills can run SCRMshaw on a desktop computer for basic applications, although an HPC cluster is recommended for optimal results. SCRMshaw can be tailored to specific needs. Users can employ a single set of training data to predict CRMs associated with a particular gene expression pattern, or utilize multiple sets to provide a rough regulatory annotation for a newly-sequenced genome.
This protocol provides an update to the protocol in Kazemian and Halfon (2019) and incorporates modifications introduced in Asma and Halfon (2019) and Asma et al. (2024).

[BEFORE_START]
Software and Dependencies:
Ensure that the following software is installed :

Python:
Python3
Pybedtools
Statistics
Pandas
Numpy
Biopython (for “extract_unannoatatedScaffolds.py”)

Perl:
Perl
Bioperl

Other:
MACS2
Download - https://pypi.org/project/MACS2/

NCBI Download Utility (optional)
Download - https://www.ncbi.nlm.nih.gov/home/tools/

Optional (for Orthology assignment steps):
Docker
Download - https://www.docker.com/products/docker-desktop/

Orthologer Software
Requirements: Docker on Mac or Linux
Download - https://hub.docker.com/r/ezlabgva/orthologer (ver2.4.3)
Documentation - https://hub.docker.com/r/ezlabgva/orthologer 

Optional (for final merging of the SCRMshaw results)
BEDTools (https://github.com/arq5x/bedtools2/releases)


[GUIDELINES]
Computational considerations:
The SCRMshaw_HD pipeline can analyze an average-sized insect genome using several training sets in a matter of hours; we are able to run all but the largest or most fragmented genomes with our full default set of 48 training sets in under 72 hours. The bulk of the computational time is spent in the SCRMshaw step. To decrease run times, chromosomes and/or training sets can be split out and run as separate instances on additional sets of 25 nodes, if available, as a simple parallelization strategy. Storage space increases with genome size, mostly due to the larger number of kmers that must be stored, and can grow to several TB with larger genomes. However, the majority of this space can be released upon completion of the pipeline by deleting intermediate and temporary files, using “cleanup” scripts we have made available. If sufficient temporary storage space is not available, it is advisable to run the original lightweight SCRMshaw (Kazemian and Halfon, 2019) rather than SCRMshaw_HD, which will keep storage requirements below 100GB for most genomes.

Running Orthologer using Singularity:
Orthologer is currently distributed as a Docker container. Many HPC environments do not allow deployment of Docker files, but can run Singularity (Aptainer) images. To run Orthologer using Singularity, you must first create a Singularity image from the Orthologer Docker image. You may need to do this on a local machine and then transfer the Singularity image to your HPC cluster. 
 
From the Ortholger GitLab site (https://gitlab.com/ezlab/orthologer_container):
 
## Note on singularity
 
The latest docker2singularity (v4.1.0) seems to produce errors when converting the docker image (error relocating /usr/local/singularity/bin/singularity: ...).
It has been tested successfully on v3.9.9.
 
The command used to convert orthologer:3.2.2 to a singularity image:
```
docker run -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/test:/output --privileged -t --rm quay.io/singularity/docker2singularity:v3.9.9 orthologer:3.2.2
```
 
The output singularity file will be created at `/tmp/test/orthologer_3.2.2-*.sif .
 
To run the image one can e.g do:
```
singularity shell /tmp/test/orthologer_3.2.2-<date and hash>.sif
```
and then at the prompt type `run_tests`.
 
In the current directory, this will create a directory `test` from where all tests are run.
##

On our cluster, we find these additional changes are also necessary:
(1) Replace the command “docker run” with “singularity exec”
(2) Omit the “-u $(id -u)” flag
(3) Substitute “-H” for “-v” in all of the Orthologer commands, e.g.  “-H $(pwd):/obdwork”
(4) Make sure to substitute the Singularity .sif image for the Docker image in all commands, e.g. “/orthologer_3.2.2-<date and hash>.sif” instead of “/ezlabgva/orthologer:v3.2.2”
 
So for example, the command at protocol step 28 would be:

 

 
Making these changes as necessary, steps 28-37 can otherwise be followed as  provided.


[STEPS]
SECTION: Create Project directory
1. Create a Project Directory and subdirectories
 

SECTION: Create Project directory
2. Under the project directory create subdirectories for saving input and output files respectively, for training data, and for scripts
 

 

 

SECTION: Install SCRMshaw
3. From the project directory, clone the SCRMshaw_HD software from 
https://github.com/HalfonLab .
 

 

SECTION: Install SCRMshaw
4. If necessary, decompress the tar archive (e.g., tar xvf SCRMshaw_HD.tar). This will create a single “SCRMshaw_HD” directory with the following subdirectories:

 

SECTION: Install SCRMshaw
5. Navigate to the SCRMshaw directory
 

SECTION: Install SCRMshaw
6. Run “make.” Detailed information on how to install and run the software is provided in the “README.txt” file located within the software package.
 

SECTION: Obtain training set sequences
7. Obtain up-to-date training data from
https://github.com/HalfonLab/Training_sets. 
Find details on how to construct custom training sets in Kazemian and Halfon (2019). Place training sets into the “TrainingSets” directory.

SECTION: Download required genome files
8. Download the genome files from NCBI Datasets 
(https://www.ncbi.nlm.nih.gov/datasets/) and place them in the GenomeFiles directory. Necessary files are:

Genome sequences (FASTA)
Annotation features (GFF)
Protein (FASTA)

We recommend using the RefSeq builds if available.

 

 

 

 

SECTION: Download required genome files
9. Rename the “genomic.gff” and “protein.faa” files to “SpeciesX.gff” and “SpeciesX_protein.fs”, respectively (where “X” is the name of your species). This avoids future confusion due to the generic “genome” and “protein” designations, and provides compatibility for downstream steps.
 

 

The GenomeFiles directory should now look similar to this:

 

SECTION: Assessing genome files using preflight
10. Assess genome and annotation files for proper content and formatting using our preflight script, which can be downloaded from 
https://github.com/HalfonLab/UtilityPrograms. We recommend placing this into the “Scripts” directory.
 

SECTION: Assessing genome files using preflight
10.1. Preflight validates the formats of these files and produces a comprehensive log file that highlights any issues along with basic information such as the number of chromosomes/scaffolds and their sizes, data types present in the annotation (e.g., ‘gene’, ‘exon’, ‘ncRNA’, etc.), and average intergenic distances. 

Preflight also provides a sample output of the SCRMshaw-generated ‘gene’ and ‘exon’ files. This feature allows users to identify any discrepancies or errors stemming from the input files and to reformat these files as needed before running SCRMshaw. Be sure to check the following:

In the first section, no GFF3 errors are reported
Gene and Exon data appear in the following sections
The ‘FASTA’ section reports “all sequences have proper characters”
The following section reports that “All seqids in GFF are also in FASTA”

If any of these conditions is not met, correct the input files before moving on to run SCRMshaw, or SCRMshaw will fail.

SECTION: Removing unannotated genes
11. Preflight will also indicate whether there are sequence scaffolds that do not contain any annotated genes (reported as “seqids not in the GFF file.”) Any such sequence scaffolds should be removed before passing the genome sequence to the main SCRMshaw program. Perform this using the “extract_unannotatedScaffolds.py” script (from 
https://github.com/HalfonLab/UtilityPrograms) that takes the preflight output file and FASTA sequence file as input and generates a fasta file, “mappedOnly_GCF_xxxxx.fna”, that contains only the desired sequences.
 

 

SECTION: Masking Tandem Repeats
12. 
 


Download the current version of TRF for your computer architecture from 
https://tandem.bu.edu/trf/home (e.g., trf409.linux64).
 

SECTION: Masking Tandem Repeats
13. Change the downloaded TRF file to an executable form:
 

SECTION: Masking Tandem Repeats
14. Run TRF on your FASTA files (e.g., mappedOnly_GCF_xxxxx.fna) using parameters as shown:
 

The resulting output file will be named 
“mappedOnly_GCF_xxxxx.fna.2.7.7.80.10.50.500.mask”

 


SECTION: Running SCRMshaw
15. SCRMshaw requires the following files and directories:

A genome file (e.g., “genome.fa”), a multi-FASTA file including all of the genomic sequences needing to be scored. This is typically a downloaded genome file that has been masked for tandem repeats, and is passed to the program using the “--genome" option.
One or more dataset directories each containing two files, “crms.fasta” and “neg.fasta.” These files are respectively the positive and negative training sequences, both in FASTA file format. We highly recommend tandem repeat masking of genome.fa, crms.fasta, and neg.fasta.
A text file containing a list of the dataset directories (e.g., “trainingSet.lst”; see below step 17). This is specified using the “--traindirlst” option.

SECTION: Running SCRMshaw
16. Navigate back to the project directory.
 

SECTION: Running SCRMshaw
17. Under the project directory, create a list file “trainingSet.lst” that contains the path to the training datasets you wish to use (e.g., project1/TrainingSets/dataset1/), one line per dataset. You can create this file using a simple text editor or using the following “echo” commands (adjust path as necessary):
 

 

 

 

At the conclusion of these steps, your directory should resemble Figure 3 below.

 

SECTION: Running SCRMshaw
18. Run SCRMshaw by providing the full paths to the genome file, gene annotation file, and training set list after the --genome, --gff, and --traindirlst flags, respectively (e.g. GenomeFiles/SpeciesX.gff), and providing appropriate values for --lb and --outdir (see Note).
 

SECTION: Running SCRMshaw
18.1. We recommend that the base directory for --outdir be set to ~/project1/HD_output for consistency with the examples in this protocol.

 


 

SECTION: Running SCRMshaw
19. The output of SCRMshaw for the 25 offset instances will be saved in subdirectories, such as 'task_offset_0_1', 'task_offset_10_2', 'task_offset_20_3', and so on up to 'task_offset_240_25', as specified by the '--outdir' parameter in the Slurm script. These directories will be created automatically if they don't already exist. If Slurm is not utilized, it is crucial to ensure that the output directory and file names adhere to the specified format (e.g., 'task_offset_0_1' to 'task_offset_240_25') to prevent any downstream disruptions. 

SECTION: Running SCRMshaw
20. The SCRMshaw command line options include:
SECTION: Running SCRMshaw
20.1. Use the name of scoring method(s) for prediction (“--imm” for IMM, “--hexmcd” for HexMCD, and “--pac” for PAC): required. Select any individual or combination of methods to run (e.g. “--imm --pac” to run IMM and PAC). For details about the individual scoring methods see (Kantorovitz et al. 2009, Kazemian et al. 2011).

SECTION: Running SCRMshaw
20.2. The gene annotation file of the genomic regions to be scored in GFF3 format (e.g., “--gff SpeciesX.gff”): optional but strongly recommended. 
Alternatively, separate “gene” and “exon” files can be used. It is also possible to use SCRMshaw without providing any annotation, but this will not allow for exclusion of coding sequences or calculation of “local rank” (see below).

 

SECTION: Running SCRMshaw
20.3. The number of top scoring regions to be reported as CRM predictions (--thitw N): optional, default = 2000.

 

SECTION: Running SCRMshaw
21. The offset parameter --lb: optional, default = 0. The --lb tells the algorithm to ignore the base pairs at the beginning of a chromosome/scaffold before this point, i.e., to start creating the analysis windows from this point in the genome. The Slurm script described above sets --lb from 0 to 240 with a step size of 10 (i.e 10, 20, 30,..., 240) for the 25 instances being run for each training set.

SECTION: Running SCRMshaw
22. Which steps of the SCRMshaw pipeline to run (--step 123): optional, default = 123. Step 1 processes the genome and genome annotation, step 2 processes the training data, and step 3 scores the genome.

 

SECTION: Running SCRMshaw
23. Detailed information on how to run SCRMshaw from the command line and additional available options can be obtained by the following command: “perl SCRMshaw_HD/code/scrm.pl” or from the README document accompanying the SCRMshaw distribution. 

An example benchmark data set along with how to run and interpret its results is provided in the example directory within the software package.

SECTION: Post-processing
24. The post-processing steps merge the data from the 25 SCRMshaw instances and select the top-scoring regions as CRM predictions. To run the post-processing steps, download the following scripts from GitHub and place in the 
“Scripts” directory:

SECTION: Post-processing
24.1. post_processing_complete.sh (https://github.com/HalfonLab/UtilityPrograms)

SECTION: Post-processing
24.2. Generate_top_N_SCRMhits.pl (https://github.com/HalfonLab/UtilityPrograms)
postProcessingScrmshawPipeline.py
(https://github.com/HalfonLab/post_processing_SCRMshaw_pipeline) 

SECTION: Post-processing
24.3. Execute the shell script post_processing_complete.sh from the project directory. Use the path to the GFF annotation file for SCRMshaw and enter as a command line argument:
 

SECTION: Post-processing
24.4. The post-processing procedure will create the following files:
scrmshawOutput_offset_0to240.bed: this is the concatenated results of the top 5000 results from each of the individual SCRMshaw instances

SECTION: Post-processing
24.5. scrmshawOutput_peaksCalled_[hexmcd/imm/pac]: BED-formatted files with the MACS-generated peaks with the results for each training set and SCRMshaw method

SECTION: Post-processing
24.6. peaks_AllSets.bed: the concatenated final results from the three separate peaks files in BED format

SECTION: Post-processing
24.7. log_flankingMoreThanOneGenesFromAllSets.txt: a log file documenting cases of SCRMshaw predictions where there is more than a single flanking gene either up- or downstream of the prediction (e.g., where genes overlap or are nested).

 

 

 

SECTION: Post-processing
24.8. It is possible to finish your SCRMshaw processing at this step, and use the information from the peaks_AllSets.bed file for any desired downstream analysis. 
However, we recommend continuing with the Orthology Mapping steps below, as it makes the output more understandable by providing recognizable gene names to what otherwise would be arbitrary ID numbers. 
If you choose to end your SCRMshaw processing here, we recommend you still follow the “cleanup” procedures in the final section of this protocol, “Cleaning Up.”

 

SECTION: Running Orthologer
25. 
 


 



Before running Orthologer, create a directory titled Project_OM.
 

SECTION: Running Orthologer
29. Create a subdirectory of Project_OM titled “protein_files”.

 

SECTION: Running Orthologer
30. Copy the protein FASTA files for the current species as well as for Drosophila melanogaster into the protein_files directory.

 

 



SECTION: Running Orthologer
26. Navigate to the Project_OM directory

 

 

SECTION: Running Orthologer
27. Obtain the Orthologer Docker image (see note above about Docker and HPC platforms).

 

 

SECTION: Running Orthologer
28. Set a working environment in directory $(pwd) as user $(id -u):
 


 


SECTION: Running Orthologer
31. Create a text file specifying the FASTA files to be used, as follows:
 

SECTION: Running Orthologer
32. Import the specified proteomes:
 

SECTION: Running Orthologer
33. Run the Orthologer container
 

SECTION: Saving Orthologer outputs for orthology mapping
34. 
 


Navigate back to the main project directory
 

SECTION: Saving Orthologer outputs for orthology mapping
35. Create a directory “orthologer_output”
 

SECTION: Saving Orthologer outputs for orthology mapping
36. Copy the desired Orthologer output files to the orthologer_output directory
 

SECTION: Saving Orthologer outputs for orthology mapping
37. If desired, at this point the Project_OM directory and its remaining files can be deleted.
 

 

SECTION: Mapping DMEL orthologs to SpeciesX SCRMshaw predictions
38. 
 



Generate a gene ID ⇔ protein ID mapping file
 

 

 


SECTION: Mapping DMEL orthologs to SpeciesX SCRMshaw predictions
39. Inspect the speciesX_A.map file and check that the first column of this file has the gene IDs formatted the same as that in column 6 of the SCRMshaw output file, and that the protein IDs in the second column match those in the Orthologer-generated “mydata.og_map” file. If necessary, edit these files for appropriate formatting. The following figures provide an example:


 



SECTION: Mapping DMEL orthologs to SpeciesX SCRMshaw predictions
39.1. Looking at this file we can see that protein ID (2nd) column matches the format of the Orthologer output, but the gene ID (1st) column is formatted slightly differently than SCRMshaw output (the latter have ‘gene-’ at the beginning of each ID). We therefore edit the file as follows:
 

SECTION: Mapping DMEL orthologs to SpeciesX SCRMshaw predictions
39.2. The resulting file should look like this:


 



SECTION: Mapping DMEL orthologs to SpeciesX SCRMshaw predictions
40. Obtain the necessary scripts and files from the Halfon Lab GitHub 
 

SECTION: Mapping DMEL orthologs to SpeciesX SCRMshaw predictions
41. Run the script “OM_mappingFlyOrthologsToSCRMshawPredictions.py”. 
You will need to supply the following command line options:

–ft: A table of D. melanogaster annotation features (included in the Mapping-D.mel-Orthologs repository)
–mD: D. melanogaster Orthologer “maptxt” output file (included in the Mapping-D.mel-Orthologs repository or from your output in the Orthologer step)
–mX: The SpeciesX Orthologer “maptxt” file generated in the Orthologer step
–og: The “mydata.og_map Orthologer output file generated in the Orthologer step
–sp1id: The appropriately edited SpeciesX.map file from step 40
–so: The SCRMshaw final output “peaks_Allsets.bed” file

 

SECTION: Interpreting the output
42. The output file will have the same file name as SCRMshaw’s output file with an additional “SO_” prefix. As noted above, this file may have duplicate and/or overlapping predictions, e.g. predict the same or similar sequences by more than one training set, or more than one method. This can be resolved using the “sort” and “merge” commands in BEDTools (Quinlan and Hall, 2010) as follows:

 

 

 

SECTION: Cleaning up
43. 
 


Download the “cleanup_fastaAndScoredirectory_SCRMshawHD.sh” from 
(https://github.com/HalfonLab/UtilityPrograms) and place into the “Scripts” directory.

SECTION: Cleaning up
44. From the Project1 directory, run the cleanup script.