Package: methylpy
methylpyWelcome to the home page of methylpy, a pyhton-based analysis pipeline
for(single-cell) (whole-genome) bisulfite sequencing data(single-cell) NOMe-seq datadifferential methylation analysismethylpy is available atgithubandPyPI.NoteVersion 1.3 has major changes on options related to mapping. A new
aligner, minimap2, is supported starting in this version. To
accommodate this new features,--bowtie2option is replaced with--aligner, which specifies the aligner to use. The parameters of--build-referencefunction are modified as well.methylpy only considers cytosines that are in uppercase in the genome
fasta file (i.e. not masked)methylpy was initiated by and built on the work ofMattew D.
Schultzbeta version oftutorialis released!What can methylpy do?Processing bisulfite sequencing data and NOMe-seq datafast and flexible pipeline for both single-end and paired-end dataall the way from raw reads (fastq) to methylation state and/or open
chromatin readoutsalso support getting readouts from alignment (BAM file)including options for read trimming, quality filter and PCR duplicate
removalaccept compressed input and generate compressed outputsupport post-bisulfite adaptor tagging (PBAT) dataCalling differentially methylated regions (DMRs)DMR calling at single cytosine levelsupport comparison across 2 or more samples/groupsconservative and accurateuseful feature for dealing with low-coverage data by combining data
of adjacent cytosinesWhat you want to doUse methylpy without
installationInstall methylpyTest methylpyProcess dataCall DMRsAdditional functions for data
processingCite methylpyrunmethylpy-hto get a list of functions.Use methylpy without installationMethylpy can be used within docker container with all dependencies
resolved. The docker image for methylpy can be built from theDockerfileundermethylpy/directory using the below command. It
will take ~3g space.git clone https://github.com/yupenghe/methylpy.git
cd methylpy/
docker build -t methylpy:latest ./Then, you can start a docker container by runningdocker run -it methylpy:latestmethylpy can be run with full functionality within the container. You
can mount your working directory to the container by adding-voption to the docker command and store methylpy output there.docker run -it -v /YOUR/WORKING/PATH/:/output methylpy:latestSeeherefor details.Install methylpyStep 1 - Download methylpyThe easiest way of installing methylpy will be through PyPI by runningpip install methylpy. The commandpip install--upgrademethylpyupdates methylpy to latest version.Methylpy can also be installed throughanacondaor [miniconda]
(https://docs.conda.io/en/latest/miniconda.html).conda env create --name methylpy_env
conda activate methylpy_env
conda install -y -c bioconda -c conda-forge methylpyAlternatively, methylpy can be installed through github: enter the
directory where you would like to install methylpy and rungit clone https://github.com/yupenghe/methylpy.git
cd methylpy/
python setup.py installIf you would like to install methylpy in path of your choice, runpython setup.py install--prefix=/USER/PATH/. Then, trymethylpyand if no error pops out, the setup is likely successful. SeeTest
methylpyfor more rigorious test. Last, processing
large dataset will require large spare space for temporary files.
Usually, the default directory for temporary files will not meet the
need. You may want to set theTMPDIRenvironmental variable to the
(absolute) path of a directory on hard drive with sufficient space (e.g./YOUR/TMP/DIR/). This can be done by adding the below command to~/.bashrcfile:exportTMPDIR=/YOUR/TMP/DIR/and runsource~/.bashrc.Step 2 - Install dependenciespython is required for running methylpy. Both python2 (>=2.7.9) and
python3 (>=3.6.2) will work. methylpy also depends on two python
modules,numpyandscipy. The easiest way to get these
dependencies is to installanaconda.In addition, some features of methylpy depend on several publicly
available tools (not all of them are required if you only use a subset
of methylpy functions). *cutadapt(>=1.9) for raw read trimming *bowtieand/orbowtie2for
alignment *samtools(>=1.3)
for alignment result manipulation. Samtools can also be installed using
condaconda install-cbioconda samtools*Picard(>=2.10.8) for PCR duplicate removal * java for running Picard (its
path needs to be included inPATHenvironment variable) . *wigToBigWigfor converting methylpy output to bigwig formatLastly, if paths to cutadapt, bowtie/bowtie2, samtools and wigToBigWig
are included inPATHvariable, methylpy can run these tools
directly. Otherwise, the paths have to be passed to methylpy as
augments. Path to Picard needs to be passed to methylpy as a parameter
to run PCR duplicate removal.Optional step - Compile rms.cppDMR finding requires an executablemethylpy/methylpy/run_rms_tests.out, which was compiled from C++
codemethylpy/methylpy/rms.cpp. In most cases, the precompiled file
can be used directly. To test this, simply run executemethylpy/methylpy/run_rms_tests.out. If help page shows, recompiling
is not required. If error turns up, the executable needs to be
regenerated by compilingrms.cppand this step requiresGSLinstalled correctly. In most
linux operating system, the below commands will do the jobcd methylpy/methylpy/
g++ -O3 -l gsl -l gslcblas -o run_rms_tests.out rms.cppIn Ubuntu (>=16.04), please try the below commands first.cd methylpy/methylpy/
g++ -o run_rms_tests.out rms.cpp `gsl-config --cflags --libs`Lastly, the compiled filerun_rms_tests.outneeds to be copied to
the directory where methylpy is installed. You can get the directory by
running the blow commands in python console (pythonto open a python
console):import methylpy
print(methylpy.__file__[:methylpy.__file__.rfind("/")]+"/")Test methylpyTo test whether methylpy and the dependencies are installed and set up
correctly, runwget http://neomorph.salk.edu/yupeng/share/methylpy_test.tar.gz
tar -xf methylpy_test.tar.gz
cd methylpy_test/
python run_test.pyThe test should take around 3 minutes, and progress will be printed on
screen. After the test is started, two filestest_output_msg.txtandtest_error_msg.txtwill be generated. The former contains more
details about each test and the later stores error message (if any) as
well as additional information.If test fails, please checktest_error_msg.txtfor the error
message. If you decide to submit an issue regarding test failure to
methylpy github page, please include the error message in this file.Process dataPlease seetutorial.
for more details.Step 1 - Build converted genome referenceBuild bowtie/bowtie2 index for converted genome. Runmethylpybuild-reference-hto get more information. An example of
building mm10 mouse reference index:methylpy build-reference \
    --input-files mm10_bt2/mm10.fa \
    --output-prefix mm10_bt2/mm10 \
    --bowtie2 TrueStep 2 - Process bisulfite sequencing and NOMe-seq dataFunctionsingle-end-pipelineis For processing single-end data. Runmethylpysingle-end-pipeline-hto get help information. Below code
is an example of using methylpy to process single-end bisulfite
sequencing data. For processing NOMe-seq data, please usenum_upstr_bases=1to include one base upstream cytosine as part of
cytosine sequence context, which can be used to tease out GC sites.methylpy single-end-pipeline \
    --read-files raw/mESC_R1.fastq.gz \
    --sample mESC \
    --forward-ref mm10_bt2/mm10_f \
    --reverse-ref mm10_bt2/mm10_r \
    --ref-fasta mm10_bt2/mm10.fa \
    --num-procs 8 \
    --remove-clonal True \
    --path-to-picard="picard/"An command example for processing paired-end data. Runmethylpypaired-end-pipeline-hto get more information.methylpy paired-end-pipeline \
    --read1-files raw/mESC_R1.fastq.gz \
    --read2-files raw/mESC_R2.fastq.gz \
    --sample mESC \
    --forward-ref mm10_bt2/mm10_f \
    --reverse-ref mm10_bt2/mm10_r \
    --ref-fasta mm10_bt2/mm10.fa \
    --num-procs 8 \
    --remove-clonal True \
    --path-to-picard="picard/"If you would like methylpy to perform binomial test for teasing out
sites that show methylation above noise level (which is mainly due to
sodium bisulfite non-conversion), please check options--binom-testand--unmethylated-control.Output formatOutput file(s) are (compressed) tab-separated text file(s) in allc
format. “allc” stands for all cytosine (C). Each row in an allc file
corresponds to one cytosine in the genome. An allc file contain 7
mandatory columns and no header. Two additional columns may be added
with--add-snp-infooption when usingsingle-end-pipeline,paired-end-pipelineorcall-methylation-statemethods.indexcolumn
nameexamplenote1chromoso
me12with
no
“chr”2position182833421-base
d3strandeither
+ or -4sequence
contextCGTcan be
more
than 3
bases5mc18count
of
reads
suppor
ting
methyl
ation6cov21read
covera
ge7methylat
ed1indica
tor
of
signif
icant
methyl
ation
(1 if
no
test
is
perfor
med)8(optiona
l)
num_mat
ches3,2,3number
of
match
baseca
lls
at
contex
t
nucleo
tides9(optiona
l)
num_mis
matches0,1,0number
of
mismat
ches
at
contex
t
nucleo
tidesCall DMRsThis function will take a list of compressed/uncompressed allc files
(output files from methylpy pipeline) as input and look for DMRs. Help
information of this function is available via runningmethylpy DMRfind-h.Below is the code of an example of calling DMRs for CG methylation
between two samples,AD_HTandAD_ITon chromosome 1 through 5
using 8 processors.methylpy DMRfind \
    --allc-files allc/allc_AD_HT.tsv.gz allc/allc_AD_IT.tsv.gz \
    --samples AD_HT AD_IT \
    --mc-type "CGN" \
    --chroms 1 2 3 4 5 \
    --num-procs 8 \
    --output-prefix DMR_HT_ITPlease seetutorialfor details.Additional functions for data processingExtract cytosine methylation state from BAM fileThecall-methylation-statefunction allows users to get cytosine
methylation state (allc file) from alignment file (BAM file). It is part
of the data processing pipeline which is especially useful for getting
the allc file from alignment file from other methylation data pipelines
like bismark. Runmethylpycall-methylation-state-hto get help
information. Below is an example of running this function. Please make
sure to remove--paired-endTrueor use--paired-endFalsefor
BAM file from single-end data.methylpy call-methylation-state \
    --input-file mESC_processed_reads_no_clonal.bam \
    --paired-end True \
    --sample mESC \
    --ref-fasta mm10_bt2/mm10.fa \
    --num-procs 8Get methylation level for genomic regionsCalculating methylation level of certain genomic regions can give an
estimate of the methylation abundance of these loci. This can be
achieved using theadd-methylation-levelfunction. Seemethylpyadd-methylation-level-hfor more details about the input
format and available options.methylpy add-methylation-level \
    --input-tsv-file DMR_AD_IT.tsv \
    --output-file DMR_AD_IT_with_level.tsv \
    --allc-files allc/allc_AD_HT_1.tsv.gz allc/allc_AD_HT_2.tsv.gz \
        allc/allc_AD_IT_1.tsv.gz allc/allc_AD_IT_2.tsv.gz \
    --samples AD_HT_1 AD_HT_2 AD_IT_1 AD_IT_2 \
    --mc-type CGN \
    --num-procs 4Merge allc filesThemerge-allcfunction can merge multiple allc files into a single
allc file. It is useful when separate allc files are generated for
replicates of a tissue or cell type, and one wants to get a single allc
file for that tissue/cell type. Seemethylpymerge-allc-hfor more
information.methylpy merge-allc \
    --allc-files allc/allc_AD_HT_1.tsv.gz allc/allc_AD_HT_2.tsv.gz \
    --output-file allc/allc_AD_HT.tsv.gz \
    --num-procs 1 \
    --compress-output TrueFilter allc filesThefilter-allcfunction is for filtering sites by cytosine context,
coverage etc. Seemethylpyfilter-allc-hfor more information.methylpy filter-allc \
    --allc-file allc/allc_AD_HT_1.tsv.gz \
    --output-file allc/allCG_AD_HT_1.tsv.gz \
    --mc-type CGN \
    --min-cov 2 \
    --compress-output TrueIndex allc filesTheindex-allcfunction allows creating index file for each allc
file. The index file can be used for speeding up allc file reading
similar to the .fai file for .fasta file. Seemethylpyindex-allc-hfor more information.methylpy index-allc \
    --allc-files allc/allc_AD_HT_1.tsv.gz allc/allc_AD_HT_2.tsv.gz \
    --num-procs 2 \
    --no-reindex FalseConvert allc file to bigwig formatTheallc-to-bigwigfunction generates bigwig file from allc file.
Methylation level will be calculated in equally divided non-overlapping
genomic bins and the output will be stored in a bigwig file. Seemethylpyallc-to-bigwig-hfor more information.methylpy allc-to-bigwig \
    --allc-file results/allc_mESC.tsv.gz \
    --output-file results/allc_mESC.bw \
    --ref-fasta mm10_bt2/mm10.fa \
    --mc-type CGN \
    --bin-size 100Quality filter for bisulfite sequencing readsSometimes, we want to filter out reads that cannot be mapped confidently
or are likely from under-converted DNA fragments. This can be done using
thebam-quality-filterfunction. Seemethylpybam-quality-filter-hfor parameter inforamtion.For example, below command can be used to filter out reads with less
than 30 MAPQ score (poor alignment) and with mCH level greater than 0.7
(under-conversion) if the reads contain enough (at least 3) CH sites.methylpy bam-quality-filter \
    --input-file mESC_processed_reads_no_clonal.bam \
    --output-file mESC_processed_reads_no_clonal.filtered.bam \
    --ref-fasta mm10_bt2/mm10.fa \
    --min-mapq 30 \
    --min-num-ch 3 \
    --max-mch-level 0.7 \
    --buffer-line-number 100Reidentify DMRs from existing resultmethylpy is able to reidentify-DMR based on the result of previous
DMRfind run. This function is especially useful in picking out DMRs
across a subset of categories and/or with different filters. Seemethylpyreidentify-DMR-hfor details about the options.methylpy reidentify-DMR \
    --input-rms-file results/DMR_P0_FBvsHT_rms_results.tsv.gz \
    --output-file results/DMR_P0_FBvsHT_rms_results_recollapsed.tsv \
    --collapse-samples P0_FB_1 P0_FB_2 P0_HT_1 P0_HT_2 \
    --sample-category P0_FB P0_FB P0_HT P0_HT \
    --min-cluster 2Cite methylpyIf you use methylpy, please cite >Matthew D. Schultz, Yupeng He, John
W.Whitaker, Manoj Hariharan, Eran A. Mukamel, Danny Leung, Nisha
Rajagopal, Joseph R. Nery, Mark A. Urich, Huaming Chen, Shin Lin, Yiing
Lin, Bing Ren, Terrence J. Sejnowski, Wei Wang, Joseph R. Ecker. Human
Body Epigenome Maps Reveal Noncanonical DNA Methylation Variation.
Nature. 523(7559):212-216, 2015 Jul.