Repository: pepkit/geofetch (language: python)

Issue:
Sanitize sample names
Sample names coming from GEO can be messy. This breaks some functionality within other PEP tools. Sample names should not have any odd characters like slashes (`/`) or spaces.

It would be nice to have geofetch sanitize (or have an option to sanitize) sample names.

**Example:**
`ATAC-seq from Luminal1/BMP k.o CAF-FOXA1 o.e tumor` becomes `ATAC-seq_from_Luminal1-BMP_k_o_CAF-FOXA1_o_e tumor`

Maybe the flag name could be `--sanitize-sample-names`

See https://github.com/pepkit/pephub/issues/40 for further discussion
Large, constant sample attributes should be moved to project config
Sometimes methods are stored as sample attributes in the SOFT files. This will be more efficiently stored at the project level in a PEP.
Line parsing flexibility
We should be able to overcome some SOFT file problems; for now we can skip over such lines but should consider populating, e.g., missing `sample_name` column in such cases, if that's an/the underlying issue.

> From a collaborator:
> 
> I followed a tip and run geofetch under 2.7 - it returns all files, great :)
>
> Unfortunately, when I tried run it exactly with GSE12417 I received:

```
  Found sample GSM311598Traceback (most recent call last):

  File "./Library/Frameworks/Python.framework/Versions/2.7/bin/geofetch", line 11, in <module>

    sys.exit(main())

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/geofetch/geofetch.py", line 776, in main

    run_geofetch(sys.argv[1:])

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/geofetch/geofetch.py", line 471, in run_geofetch

    pl = parse_SOFT_line(line)

  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/geofetch/geofetch.py", line 169, in parse_SOFT_line

    return {elems[0].rstrip(): elems[1].lstrip()}

IndexError: list index out of range
```
> And script was not able to produce annotation file. 
> 
> I will be grateful for any idea!


Discussion / hints:


Reported by collaborator via @j-lawson 

Fix patch:
diff --git a/README.md b/README.md
index 5c2b575..deb3b09 100644
--- a/README.md
+++ b/README.md
@@ -1,8 +1,9 @@
 # <img src="docs/img/geofetch_logo.svg" alt="geofetch logo" height="70">
 
+![Run pytests](https://github.com/pepkit/geofetch/workflows/Run%20pytests/badge.svg)
 [![PEP compatible](http://pepkit.github.io/img/PEP-compatible-green.svg)](http://pepkit.github.io)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
-`geofetch` is a command-line tool that downloads sequencing data and metadata from GEO and SRA and creates [standard PEPs](http://pepkit.github.io). `geofetch` is hosted at [pypi](https://pypi.org/project/geofetch/) and documentation is hosted at [geofetch.databio.org](http://geofetch.databio.org) (source in the [/docs](/docs) folder).
+`geofetch` is a command-line tool that downloads sequencing data and metadata from GEO and SRA and creates [standard PEPs](http://pep.databio.org/). `geofetch` is hosted at [pypi](https://pypi.org/project/geofetch/) and documentation is hosted at [geofetch.databio.org](http://geofetch.databio.org) (source in the [/docs](/docs) folder).
 
 You can convert the result of geofetch into unmapped `bam` or `fastq` files with the included `sraconvert` command.
diff --git a/docs/README.md b/docs/README.md
index 3184d07..de6d549 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1,5 +1,6 @@
 # <img src="img/geofetch_logo.svg" class="img-header">
 
+![Run pytests](https://github.com/pepkit/geofetch/workflows/Run%20pytests/badge.svg)
 [![PEP compatible](http://pepkit.github.io/img/PEP-compatible-green.svg)](http://pepkit.github.io)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
@@ -24,11 +25,16 @@ You can add `--processed` if you want to download processed files from the given
 geofetch -i GSE95654 --processed
 ```
 
+
 You can add `--just-metadata` if you want to download metadata without the raw SRA files or processed GEO files.
 
 ```console
 geofetch -i GSE95654 --just-metadata
 ```
 
+```console
+geofetch -i GSE95654 --processed --just-metadata
+```
+
 For more details, check out the [usage](usage.md) reference, [installation instructions](install.md), or head on over to the [tutorial for raw data](raw-data-downloading.md) and [tutorial for processed data](processed-data-downloading.md) for a detailed walkthrough.
 
diff --git a/docs/changelog.md b/docs/changelog.md
index bdfd79a..95d2b09 100644
--- a/docs/changelog.md
+++ b/docs/changelog.md
@@ -1,5 +1,19 @@
 # Changelog
 
+## [0.9.0] -- 2022-06-20
+- Updated `--pipeline-interface` argument that adds it in for looper. `--pipeline-interface` argument was divided into: 
+`--pipeline-samples` and `--pipeline-project`.
+- Fixed empty sample_name error while creating PEP.
+- Added `--discard-soft` argument.
+- Added `--const-limit-project` argument.
+- Added `--const-limit-discard` argument.
+- Added `--attr-limit-truncate` argument.
+- Added `"--add-dotfile"` argument.
+- Disabled creating combined pep when flag `--acc-anno` is set.
+- Improved finding and separating metadata keys and genome assembly information.
+- Added standardization of column names by replacing characters to lowercase and spaces by underscore.
+
+
 ## [0.8.0] -- 2022-03-10
 - Added `--filter-size` argument.
 - Added `--data-source` argument.
diff --git a/docs/usage.md b/docs/usage.md
index cc7c3d3..89b3fad 100644
--- a/docs/usage.md
+++ b/docs/usage.md
@@ -1,4 +1,6 @@
-# <img src="./img/geofetch_logo.svg" class="img-header">  Usage reference
+# <img src="img/geofetch_logo.svg" class="img-header">
+
+Usage reference
 
 `geofetch` command-line usage instructions:
 
@@ -6,8 +8,12 @@
 
 `geofetch --help`
 ```{console}
-usage: geofetch [-h] [-V] -i INPUT [-n NAME] [-m METADATA_ROOT] [-u METADATA_FOLDER] [--just-metadata] [-r] [--config-template CONFIG_TEMPLATE] [-P PIPELINE_INTERFACES] [-k SKIP] [--acc-anno] [--use-key-subset] [-p]
-                [--data-source {all,samples,series}] [--filter FILTER] [--filter-size FILTER_SIZE] [-g GEO_FOLDER] [-x] [-b BAM_FOLDER] [-f FQ_FOLDER] [--silent] [--verbosity V] [--logdev]
+usage: geofetch [-h] [-V] -i INPUT [-n NAME] [-m METADATA_ROOT] [-u METADATA_FOLDER] [--just-metadata] [-r] [--config-template CONFIG_TEMPLATE]
+                [--pipeline_samples PIPELINE_SAMPLES] [--pipeline_project PIPELINE_PROJECT] [-k SKIP] [--acc-anno] [--discard-soft]
+                [--const-limit-project CONST_LIMIT_PROJECT] [--const-limit-discard CONST_LIMIT_DISCARD]
+                [--attr-limit-truncate ATTR_LIMIT_TRUNCATE] [-p] [--data-source {all,samples,series}] [--filter FILTER]
+                [--filter-size FILTER_SIZE] [-g GEO_FOLDER] [-x] [-b BAM_FOLDER] [-f FQ_FOLDER] [--use-key-subset] [--silent] [--verbosity V]
+                [--logdev]
 
 Automatic GEO and SRA data downloader
 
@@ -20,17 +26,29 @@ optional arguments:
   -m METADATA_ROOT, --metadata-root METADATA_ROOT
                         Specify a parent folder location to store metadata. The project name will be added as a subfolder [Default: $SRAMETA:]
   -u METADATA_FOLDER, --metadata-folder METADATA_FOLDER
-                        Specify an absolute folder location to store metadata. No subfolder will be added. Overrides value of --metadata-root [Default: Not used (--metadata-root is used by default)]
+                        Specify an absolute folder location to store metadata. No subfolder will be added. Overrides value of --metadata-root
+                        [Default: Not used (--metadata-root is used by default)]
   --just-metadata       If set, don't actually run downloads, just create metadata
   -r, --refresh-metadata
                         If set, re-download metadata even if it exists.
   --config-template CONFIG_TEMPLATE
                         Project config yaml file template.
-  -P PIPELINE_INTERFACES, --pipeline_interfaces PIPELINE_INTERFACES
-                        Optional: Specify one or more filepaths to pipeline interface yaml files. These will be added to the project config file to make it immediately compatible with looper. [Default: null]
+  --pipeline_samples PIPELINE_SAMPLES
+                        Optional: Specify one or more filepaths to SAMPLES pipeline interface yaml files. These will be added to the project
+                        config file to make it immediately compatible with looper. [Default: null]
+  --pipeline_project PIPELINE_PROJECT
+                        Optional: Specify one or more filepaths to PROJECT pipeline interface yaml files. These will be added to the project
+                        config file to make it immediately compatible with looper. [Default: null]
   -k SKIP, --skip SKIP  Skip some accessions. [Default: no skip].
-  --acc-anno            Also produce annotation sheets for each accession, not just for the whole project combined
-  --use-key-subset      Use just the keys defined in this module when writing out metadata.
+  --acc-anno            Optional: Produce annotation sheets for each accession. Project combined PEP for the whole project won't be produced.
+  --discard-soft        Optional: After creation of PEP files, all soft and additional files will be deleted
+  --const-limit-project CONST_LIMIT_PROJECT
+                        Optional: Limit of the number of the constant sample characters that should not be in project yaml. [Default: 50]
+  --const-limit-discard CONST_LIMIT_DISCARD
+                        Optional: Limit of the number of the constant sample characters that should not be discarded [Default: 250]
+  --attr-limit-truncate ATTR_LIMIT_TRUNCATE
+                        Optional: Limit of the number of sample characters.Any attribute with more than X characters will truncate to the first
+                        X, where X is a number of characters [Default: 500]
   --silent              Silence logging. Overrides verbosity.
   --verbosity V         Set logging level (1-5 or logging module level name)
   --logdev              Expand content of logging message format.
@@ -38,23 +56,27 @@ optional arguments:
 processed:
   -p, --processed       Download processed data [Default: download raw data].
   --data-source {all,samples,series}
-                        Optional: Specifies the source of data on the GEO record to retrieve processed data, which may be attached to the collective series entity, or to individual samples. Allowable values are: samples,
-                        series or both (all). Ignored unless 'processed' flag is set. [Default: all]
+                        Optional: Specifies the source of data on the GEO record to retrieve processed data, which may be attached to the
+                        collective series entity, or to individual samples. Allowable values are: samples, series or both (all). Ignored unless
+                        'processed' flag is set. [Default: samples]
   --filter FILTER       Optional: Filter regex for processed filenames [Default: None].Ignored unless 'processed' flag is set.
   --filter-size FILTER_SIZE
-                        Optional: Filter size for processed files that are stored as sample repository [Default: None]. Supported input formats : 12B, 12KB, 12MB, 12GB. Ignored unless 'processed' flag is set.
+                        Optional: Filter size for processed files that are stored as sample repository [Default: None]. Works only for sample
+                        data. Supported input formats : 12B, 12KB, 12MB, 12GB. Ignored unless 'processed' flag is set.
   -g GEO_FOLDER, --geo-folder GEO_FOLDER
                         Optional: Specify a location to store processed GEO files. Ignored unless 'processed' flag is set.[Default: $GEODATA:]
 
 raw:
   -x, --split-experiments
-                        Split SRR runs into individual samples. By default, SRX experiments with multiple SRR Runs will have a single entry in the annotation table, with each run as a separate row in the subannotation
-                        table. This setting instead treats each run as a separate sample
+                        Split SRR runs into individual samples. By default, SRX experiments with multiple SRR Runs will have a single entry in
+                        the annotation table, with each run as a separate row in the subannotation table. This setting instead treats each run as
+                        a separate sample
   -b BAM_FOLDER, --bam-folder BAM_FOLDER
-                        Optional: Specify folder of bam files. Geofetch will not download sra files when corresponding bam files already exist. [Default: $SRABAM:]
+                        Optional: Specify folder of bam files. Geofetch will not download sra files when corresponding bam files already exist.
+                        [Default: $SRABAM:]
   -f FQ_FOLDER, --fq-folder FQ_FOLDER
-                        Optional: Specify folder of fastq files. Geofetch will not download sra files when corresponding fastq files already exist. [Default: $SRAFQ:]
-
-
+                        Optional: Specify folder of fastq files. Geofetch will not download sra files when corresponding fastq files already
+                        exist. [Default: $SRAFQ:]
+  --use-key-subset      Use just the keys defined in this module when writing out metadata.
 
 ```
diff --git a/docs_jupyter/build/processed-data-downloading.md b/docs_jupyter/build/processed-data-downloading.md
new file mode 100644
index 0000000..d85b32b
--- /dev/null
+++ b/docs_jupyter/build/processed-data-downloading.md
@@ -0,0 +1,477 @@
+jupyter:True
+# geofetch tutorial for processed data
+
+The [GSE185701 data set](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE185701) has about 355 Mb of processed data that contains 57 Supplementary files, so it's a quick download for a test case. Let's take a quick peek at the geofetch version:
+
+
+```bash
+geofetch --version
+```
+
+```.output
+geofetch 0.9.0
+
+```
+
+To see your CLI options, invoke `geofetch -h`:
+
+
+```bash
+geofetch -h
+```
+
+```.output
+usage: geofetch [-h] [-V] -i INPUT [-n NAME] [-m METADATA_ROOT]
+                [-u METADATA_FOLDER] [--just-metadata] [-r]
+                [--config-template CONFIG_TEMPLATE]
+                [--pipeline_samples PIPELINE_SAMPLES]
+                [--pipeline_project PIPELINE_PROJECT] [-k SKIP] [--acc-anno]
+                [--discard-soft] [--const-limit-project CONST_LIMIT_PROJECT]
+                [--const-limit-discard CONST_LIMIT_DISCARD]
+                [--attr-limit-truncate ATTR_LIMIT_TRUNCATE] [-p]
+                [--data-source {all,samples,series}] [--filter FILTER]
+                [--filter-size FILTER_SIZE] [-g GEO_FOLDER] [-x]
+                [-b BAM_FOLDER] [-f FQ_FOLDER] [--use-key-subset] [--silent]
+                [--verbosity V] [--logdev]
+
+Automatic GEO and SRA data downloader
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -V, --version         show program's version number and exit
+  -i INPUT, --input INPUT
+                        required: a GEO (GSE) accession, or a file with a list
+                        of GSE numbers
+  -n NAME, --name NAME  Specify a project name. Defaults to GSE number
+  -m METADATA_ROOT, --metadata-root METADATA_ROOT
+                        Specify a parent folder location to store metadata.
+                        The project name will be added as a subfolder
+                        [Default: $SRAMETA:]
+  -u METADATA_FOLDER, --metadata-folder METADATA_FOLDER
+                        Specify an absolute folder location to store metadata.
+                        No subfolder will be added. Overrides value of
+                        --metadata-root [Default: Not used (--metadata-root is
+                        used by default)]
+  --just-metadata       If set, don't actually run downloads, just create
+                        metadata
+  -r, --refresh-metadata
+                        If set, re-download metadata even if it exists.
+  --config-template CONFIG_TEMPLATE
+                        Project config yaml file template.
+  --pipeline_samples PIPELINE_SAMPLES
+                        Optional: Specify one or more filepaths to SAMPLES
+                        pipeline interface yaml files. These will be added to
+                        the project config file to make it immediately
+                        compatible with looper. [Default: null]
+  --pipeline_project PIPELINE_PROJECT
+                        Optional: Specify one or more filepaths to PROJECT
+                        pipeline interface yaml files. These will be added to
+                        the project config file to make it immediately
+                        compatible with looper. [Default: null]
+  -k SKIP, --skip SKIP  Skip some accessions. [Default: no skip].
+  --acc-anno            Optional: Produce annotation sheets for each
+                        accession. Project combined PEP for the whole project
+                        won't be produced.
+  --discard-soft        Optional: After creation of PEP files, all soft and
+                        additional files will be deleted
+  --const-limit-project CONST_LIMIT_PROJECT
+                        Optional: Limit of the number of the constant sample
+                        characters that should not be in project yaml.
+                        [Default: 50]
+  --const-limit-discard CONST_LIMIT_DISCARD
+                        Optional: Limit of the number of the constant sample
+                        characters that should not be discarded [Default: 250]
+  --attr-limit-truncate ATTR_LIMIT_TRUNCATE
+                        Optional: Limit of the number of sample characters.Any
+                        attribute with more than X characters will truncate to
+                        the first X, where X is a number of characters
+                        [Default: 500]
+  --silent              Silence logging. Overrides verbosity.
+  --verbosity V         Set logging level (1-5 or logging module level name)
+  --logdev              Expand content of logging message format.
+
+processed:
+  -p, --processed       Download processed data [Default: download raw data].
+  --data-source {all,samples,series}
+                        Optional: Specifies the source of data on the GEO
+                        record to retrieve processed data, which may be
+                        attached to the collective series entity, or to
+                        individual samples. Allowable values are: samples,
+                        series or both (all). Ignored unless 'processed' flag
+                        is set. [Default: samples]
+  --filter FILTER       Optional: Filter regex for processed filenames
+                        [Default: None].Ignored unless 'processed' flag is
+                        set.
+  --filter-size FILTER_SIZE
+                        Optional: Filter size for processed files that are
+                        stored as sample repository [Default: None]. Works
+                        only for sample data. Supported input formats : 12B,
+                        12KB, 12MB, 12GB. Ignored unless 'processed' flag is
+                        set.
+  -g GEO_FOLDER, --geo-folder GEO_FOLDER
+                        Optional: Specify a location to store processed GEO
+                        files. Ignored unless 'processed' flag is
+                        set.[Default: $GEODATA:]
+
+raw:
+  -x, --split-experiments
+                        Split SRR runs into individual samples. By default,
+                        SRX experiments with multiple SRR Runs will have a
+                        single entry in the annotation table, with each run as
+                        a separate row in the subannotation table. This
+                        setting instead treats each run as a separate sample
+  -b BAM_FOLDER, --bam-folder BAM_FOLDER
+                        Optional: Specify folder of bam files. Geofetch will
+                        not download sra files when corresponding bam files
+                        already exist. [Default: $SRABAM:]
+  -f FQ_FOLDER, --fq-folder FQ_FOLDER
+                        Optional: Specify folder of fastq files. Geofetch will
+                        not download sra files when corresponding fastq files
+                        already exist. [Default: $SRAFQ:]
+  --use-key-subset      Use just the keys defined in this module when writing
+                        out metadata.
+
+```
+
+Calling geofetch will do 4 tasks: 
+
+1. download all or filtered processed files from `GSE#####` into your geo folder.
+2. download all metadata from GEO and store in your metadata folder.
+2. produce a PEP-compatible sample table, `PROJECT_NAME_sample_processed.csv` and `PROJECT_NAME_series_processed.csv`, in your metadata folder.
+3. produce a PEP-compatible project configuration file, `PROJECT_NAME_sample_processed.yaml` and `PROJECT_NAME_series_processed.yaml`, in your metadata folder.
+
+Complete details about geofetch outputs is cataloged in the [metadata outputs reference](metadata_output.md).
+
+## Download the data
+
+First, create the metadata for processed data (by adding --processed and --just-metadata):
+
+
+```bash
+geofetch -i GSE185701 --processed -n bright_test --just-metadata
+```
+
+```.output
+Metadata folder: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test
+Trying GSE185701 (not a file) as accession...
+Skipped 0 accessions. Starting now.
+Processing accession 1 of 1: 'GSE185701'
+--2022-06-10 14:43:11--  https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?targ=gse&acc=GSE185701&form=text&view=full
+Resolving www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)... 2607:f220:41e:4290::110, 130.14.29.110
+Connecting to www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)|2607:f220:41e:4290::110|:443... connected.
+HTTP request sent, awaiting response... 200 OK
+Length: unspecified [geo/text]
+Saving to: ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_GSE.soft’
+
+/home/bnt4me/Virgin     [ <=>                ]   2.82K  --.-KB/s    in 0s      
+
+2022-06-10 14:43:12 (207 MB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_GSE.soft’ saved [2885]
+
+--2022-06-10 14:43:12--  https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?targ=gsm&acc=GSE185701&form=text&view=full
+Resolving www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)... 2607:f220:41e:4290::110, 130.14.29.110
+Connecting to www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)|2607:f220:41e:4290::110|:443... connected.
+HTTP request sent, awaiting response... 200 OK
+Length: unspecified [geo/text]
+Saving to: ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_GSM.soft’
+
+/home/bnt4me/Virgin     [  <=>               ]  39.51K   110KB/s    in 0.4s    
+
+2022-06-10 14:43:12 (110 KB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_GSM.soft’ saved [40454]
+
+
+--2022-06-10 14:43:12--  ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE185nnn/GSE185701/suppl/filelist.txt
+           => ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_file_list.txt’
+Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41f:250::228, 2607:f220:41f:250::229, 130.14.250.7, ...
+Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41f:250::228|:21... connected.
+Logging in as anonymous ... Logged in!
+==> SYST ... done.    ==> PWD ... done.
+==> TYPE I ... done.  ==> CWD (1) /geo/series/GSE185nnn/GSE185701/suppl ... done.
+==> SIZE filelist.txt ... 794
+==> EPSV ... done.    ==> RETR filelist.txt ... done.
+Length: 794 (unauthoritative)
+
+filelist.txt        100%[===================>]     794  --.-KB/s    in 0.001s  
+
+2022-06-10 14:43:13 (716 KB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_file_list.txt’ saved [794]
+
+0
+
+Total number of processed SAMPLES files found is: 8
+Total number of processed SERIES files found is: 1
+Expanding metadata list...
+Expanding metadata list...
+Finished processing 1 accession(s)
+Unifying and saving of metadata... 
+File /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_samples.csv has been saved successfully
+  Config file: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_samples.yaml
+
+```
+
+
+```bash
+ls bright_test
+```
+
+```.output
+GSE185701_file_list.txt  GSE185701_GSM.soft     GSE185701_samples.yaml
+GSE185701_GSE.soft       GSE185701_samples.csv
+
+```
+
+The `.soft` files are the direct output from GEO, which contain all the metadata as stored by GEO, for both the experiment (`_GSE`) and for the individual samples (`_GSM`). Geofetch also produces a `csv` file with the SRA metadata. The filtered version (ending in `_filt`) would contain only the specified subset of the samples if we didn't request them all, but in this case, since we only gave an accession, it is identical to the complete file. Additionally, file_list.txt is downloaded, that contains information about size, type and creation date of all sample files.
+
+Finally, there are the 2 files that make up the PEP: the `_config.yaml` file and the `_annotation.csv` file (for samples and series). Let's see what's in these files now.
+
+
+```bash
+cat bright_test/GSE185701_samples.yaml
+```
+
+```.output
+# Autogenerated by geofetch
+
+pep_version: 2.1.0
+project_name: GSE185701
+sample_table: GSE185701_samples.csv
+
+sample_modifiers:
+  append:
+    output_file_path: FILES
+    Sample_growth_protocol_ch1: Huh 7 was cultured in Dulbecco’s modified Eagle’s medium (DMEM) (Invitrogen, Carlsbad, CA, USA) containing 10% fetal bovine serum (FBS) (HyClone, Logan, UT, USA) and antibiotics (penicillin and streptomycin, Invitrogen) at 37 °C in 5% CO2.
+    
+  derive:
+    attributes: [output_file_path]
+    sources:
+      FILES: /{GSE}/{file}
+
+
+
+
+```
+
+There are few important things to note in this file:
+
+* First, see in the PEP that `sample_table` points to the csv file produced by geofetch.
+* Second: output_file_path is location of all the files. 
+* Third: sample_modifier Sample_growth_protocol_ch1 is constant sample character and is larger then 50 characters so it is deleted from csv file. For large project it can significantly reduced size of the metadata
+
+Now let's look at the first 100 characters of the csv file:
+
+
+```bash
+cut -c -100 bright_test/GSE185701_samples.csv
+```
+
+```.output
+GSE,Sample_title,Sample_geo_accession,Sample_status,Sample_submission_date,Sample_last_update_date,S
+GSE185701,Huh7_siNC_H3K27ac,GSM5621756,Public on Mar 01 2022,Oct 12 2021,Mar 03 2022,SRA,1,Huh 7,Hom
+GSE185701,Huh7_siNC_H3K27ac,GSM5621756,Public on Mar 01 2022,Oct 12 2021,Mar 03 2022,SRA,1,Huh 7,Hom
+GSE185701,Huh7_siDHX37_H3K27ac,GSM5621758,Public on Mar 01 2022,Oct 12 2021,Mar 03 2022,SRA,1,Huh 7,
+GSE185701,Huh7_siDHX37_H3K27ac,GSM5621758,Public on Mar 01 2022,Oct 12 2021,Mar 03 2022,SRA,1,Huh 7,
+GSE185701,Huh7_DHX37,GSM5621760,Public on Mar 01 2022,Oct 12 2021,Mar 03 2022,SRA,1,Huh 7,Homo sapie
+GSE185701,Huh7_DHX37,GSM5621760,Public on Mar 01 2022,Oct 12 2021,Mar 03 2022,SRA,1,Huh 7,Homo sapie
+GSE185701,Huh7_PLRG1,GSM5621761,Public on Mar 01 2022,Oct 12 2021,Mar 03 2022,SRA,1,Huh 7,Homo sapie
+GSE185701,Huh7_PLRG1,GSM5621761,Public on Mar 01 2022,Oct 12 2021,Mar 03 2022,SRA,1,Huh 7,Homo sapie
+
+```
+
+Now let's download the actual data. This time we will will be downloading data from the [GSE185701 data set](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE185701) .
+
+Let's additionally add few arguments:
+
+* _geo-folder_ (required) - path to the location where processed files have to be saved
+* _filter_ argument, to download only _bed_ files  (--filter ".Bed.gz$")
+* _data-source_ argument, to download files only from sample location (--data-source samples)
+
+
+```bash
+geofetch -i GSE185701 --processed -n bright_test --filter ".bed.gz$" --data-source samples \
+--geo-folder /home/bnt4me/Virginia/for_docs/geo
+```
+
+```.output
+Metadata folder: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter
+Trying GSE185701 (not a file) as accession...
+Skipped 0 accessions. Starting now.
+Processing accession 1 of 1: 'GSE185701'
+--2022-06-10 14:53:15--  https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?targ=gse&acc=GSE185701&form=text&view=full
+Resolving www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)... 2607:f220:41e:4290::110, 130.14.29.110
+Connecting to www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)|2607:f220:41e:4290::110|:443... connected.
+HTTP request sent, awaiting response... 200 OK
+Length: unspecified [geo/text]
+Saving to: ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_GSE.soft’
+
+/home/bnt4me/Virgin     [ <=>                ]   2.82K  --.-KB/s    in 0s      
+
+2022-06-10 14:53:16 (898 MB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_GSE.soft’ saved [2885]
+
+--2022-06-10 14:53:16--  https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?targ=gsm&acc=GSE185701&form=text&view=full
+Resolving www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)... 2607:f220:41e:4290::110, 130.14.29.110
+Connecting to www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)|2607:f220:41e:4290::110|:443... connected.
+HTTP request sent, awaiting response... 200 OK
+Length: unspecified [geo/text]
+Saving to: ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_GSM.soft’
+
+/home/bnt4me/Virgin     [  <=>               ]  39.51K   180KB/s    in 0.2s    
+
+2022-06-10 14:53:16 (180 KB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_GSM.soft’ saved [40454]
+
+
+--2022-06-10 14:53:16--  ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE185nnn/GSE185701/suppl/filelist.txt
+           => ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_file_list.txt’
+Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41f:250::230, 2607:f220:41e:250::12, 165.112.9.228, ...
+Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41f:250::230|:21... connected.
+Logging in as anonymous ... Logged in!
+==> SYST ... done.    ==> PWD ... done.
+==> TYPE I ... done.  ==> CWD (1) /geo/series/GSE185nnn/GSE185701/suppl ... done.
+==> SIZE filelist.txt ... 794
+==> EPSV ... done.    ==> RETR filelist.txt ... done.
+Length: 794 (unauthoritative)
+
+filelist.txt        100%[===================>]     794  --.-KB/s    in 0s      
+
+2022-06-10 14:53:17 (285 MB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_file_list.txt’ saved [794]
+
+0
+
+Total number of processed SAMPLES files found is: 8
+Total number of files after filter is: 4 
+Total number of processed SERIES files found is: 1
+Total number of files after filter is: 0 
+Expanding metadata list...
+Expanding metadata list...
+
+--2022-06-10 14:53:17--  ftp://ftp.ncbi.nlm.nih.gov/geo/samples/GSM5621nnn/GSM5621756/suppl/GSM5621756_ChIPseq_Huh7_siNC_H3K27ac_summits.bed.gz
+           => ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621756_ChIPseq_Huh7_siNC_H3K27ac_summits.bed.gz’
+Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41f:250::230, 2607:f220:41e:250::12, 165.112.9.228, ...
+Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41f:250::230|:21... connected.
+Logging in as anonymous ... Logged in!
+==> SYST ... done.    ==> PWD ... done.
+==> TYPE I ... done.  ==> CWD (1) /geo/samples/GSM5621nnn/GSM5621756/suppl ... done.
+==> SIZE GSM5621756_ChIPseq_Huh7_siNC_H3K27ac_summits.bed.gz ... 785486
+==> EPSV ... done.    ==> RETR GSM5621756_ChIPseq_Huh7_siNC_H3K27ac_summits.bed.gz ... done.
+Length: 785486 (767K) (unauthoritative)
+
+GSM5621756_ChIPseq_ 100%[===================>] 767.08K  2.46MB/s    in 0.3s    
+
+2022-06-10 14:53:19 (2.46 MB/s) - ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621756_ChIPseq_Huh7_siNC_H3K27ac_summits.bed.gz’ saved [785486]
+
+0
+
+File /home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621756_ChIPseq_Huh7_siNC_H3K27ac_summits.bed.gz has been downloaded successfully
+
+--2022-06-10 14:53:20--  ftp://ftp.ncbi.nlm.nih.gov/geo/samples/GSM5621nnn/GSM5621758/suppl/GSM5621758_ChIPseq_Huh7_siDHX37_H3K27ac_summits.bed.gz
+           => ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621758_ChIPseq_Huh7_siDHX37_H3K27ac_summits.bed.gz’
+Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41f:250::230, 2607:f220:41e:250::12, 165.112.9.228, ...
+Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41f:250::230|:21... connected.
+Logging in as anonymous ... Logged in!
+==> SYST ... done.    ==> PWD ... done.
+==> TYPE I ... done.  ==> CWD (1) /geo/samples/GSM5621nnn/GSM5621758/suppl ... done.
+==> SIZE GSM5621758_ChIPseq_Huh7_siDHX37_H3K27ac_summits.bed.gz ... 784432
+==> EPSV ... done.    ==> RETR GSM5621758_ChIPseq_Huh7_siDHX37_H3K27ac_summits.bed.gz ... done.
+Length: 784432 (766K) (unauthoritative)
+
+GSM5621758_ChIPseq_ 100%[===================>] 766.05K  2.99MB/s    in 0.2s    
+
+2022-06-10 14:53:20 (2.99 MB/s) - ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621758_ChIPseq_Huh7_siDHX37_H3K27ac_summits.bed.gz’ saved [784432]
+
+0
+
+File /home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621758_ChIPseq_Huh7_siDHX37_H3K27ac_summits.bed.gz has been downloaded successfully
+
+--2022-06-10 14:53:21--  ftp://ftp.ncbi.nlm.nih.gov/geo/samples/GSM5621nnn/GSM5621760/suppl/GSM5621760_CUTTag_Huh7_DHX37_summits.bed.gz
+           => ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621760_CUTTag_Huh7_DHX37_summits.bed.gz’
+Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41f:250::230, 2607:f220:41e:250::12, 165.112.9.228, ...
+Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41f:250::230|:21... connected.
+Logging in as anonymous ... Logged in!
+==> SYST ... done.    ==> PWD ... done.
+==> TYPE I ... done.  ==> CWD (1) /geo/samples/GSM5621nnn/GSM5621760/suppl ... done.
+==> SIZE GSM5621760_CUTTag_Huh7_DHX37_summits.bed.gz ... 163441
+==> EPSV ... done.    ==> RETR GSM5621760_CUTTag_Huh7_DHX37_summits.bed.gz ... done.
+Length: 163441 (160K) (unauthoritative)
+
+GSM5621760_CUTTag_H 100%[===================>] 159.61K   733KB/s    in 0.2s    
+
+2022-06-10 14:53:22 (733 KB/s) - ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621760_CUTTag_Huh7_DHX37_summits.bed.gz’ saved [163441]
+
+0
+
+File /home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621760_CUTTag_Huh7_DHX37_summits.bed.gz has been downloaded successfully
+
+--2022-06-10 14:53:22--  ftp://ftp.ncbi.nlm.nih.gov/geo/samples/GSM5621nnn/GSM5621761/suppl/GSM5621761_CUTTag_Huh7_PLRG1_summits.bed.gz
+           => ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621761_CUTTag_Huh7_PLRG1_summits.bed.gz’
+Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41f:250::230, 2607:f220:41e:250::12, 165.112.9.228, ...
+Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41f:250::230|:21... connected.
+Logging in as anonymous ... Logged in!
+==> SYST ... done.    ==> PWD ... done.
+==> TYPE I ... done.  ==> CWD (1) /geo/samples/GSM5621nnn/GSM5621761/suppl ... done.
+==> SIZE GSM5621761_CUTTag_Huh7_PLRG1_summits.bed.gz ... 117250
+==> EPSV ... done.    ==> RETR GSM5621761_CUTTag_Huh7_PLRG1_summits.bed.gz ... done.
+Length: 117250 (115K) (unauthoritative)
+
+GSM5621761_CUTTag_H 100%[===================>] 114.50K  --.-KB/s    in 0.1s    
+
+2022-06-10 14:53:23 (937 KB/s) - ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621761_CUTTag_Huh7_PLRG1_summits.bed.gz’ saved [117250]
+
+0
+
+File /home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621761_CUTTag_Huh7_PLRG1_summits.bed.gz has been downloaded successfully
+Finished processing 1 accession(s)
+Unifying and saving of metadata... 
+File /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_samples.csv has been saved successfully
+  Config file: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_samples.yaml
+
+```
+
+Now lets list the folder to see what data is there. And let's see what's in pep files now.
+
+
+```bash
+ls /home/bnt4me/Virginia/for_docs/geo/GSE185701
+```
+
+```.output
+GSM5621756_ChIPseq_Huh7_siNC_H3K27ac_summits.bed.gz
+GSM5621758_ChIPseq_Huh7_siDHX37_H3K27ac_summits.bed.gz
+GSM5621760_CUTTag_Huh7_DHX37_summits.bed.gz
+GSM5621761_CUTTag_Huh7_PLRG1_summits.bed.gz
+
+```
+
+
+```bash
+cut -c -100 bright_test/bright_test_annotation_sample_processed.csv
+```
+
+```.output
+GSE,Sample_title,Sample_geo_accession,Sample_status,Sample_submission_date,Sample_last_update_date,S
+GSE185701,Huh7_siNC_H3K27ac,GSM5621756,Public on Mar 01 2022,Oct 12 2021,Mar 03 2022,SRA,1,Huh 7,Hom
+GSE185701,Huh7_siDHX37_H3K27ac,GSM5621758,Public on Mar 01 2022,Oct 12 2021,Mar 03 2022,SRA,1,Huh 7,
+GSE185701,Huh7_DHX37,GSM5621760,Public on Mar 01 2022,Oct 12 2021,Mar 03 2022,SRA,1,Huh 7,Homo sapie
+GSE185701,Huh7_PLRG1,GSM5621761,Public on Mar 01 2022,Oct 12 2021,Mar 03 2022,SRA,1,Huh 7,Homo sapie
+
+```
+
+
+```bash
+cat bright_test/bright_test_annotation_sample_processed.yaml
+```
+
+```.output
+pep_version: 2.0.0
+project_name: bright_test
+sample_table: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/bright_test_annotation_sample_processed.csv
+
+sample_modifiers:
+  append:
+    output_file_path: FILES
+  derive:
+    attributes: [output_file_path]
+    sources:
+      FILES: /home/bnt4me/Virginia/for_docs/geo/{SRA}/{sample_name}
+
+
+
+```
+
+Now we have easy access to this data by using [peppy](http://peppy.databio.org/en/latest/) package in python or [pepr](https://code.databio.org/pepr/) in r in further analysis 
diff --git a/docs_jupyter/build/raw-data-downloading.md b/docs_jupyter/build/raw-data-downloading.md
new file mode 100644
index 0000000..ca329d2
--- /dev/null
+++ b/docs_jupyter/build/raw-data-downloading.md
@@ -0,0 +1,664 @@
+jupyter:True
+# geofetch tutorial for raw data
+
+The [GSE67303 data set](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE67303) has about 250 mb of data across 4 samples, so it's a quick download for a test case. Let's take a quick peek at the geofetch version:
+
+
+```bash
+geofetch --version
+```
+
+```.output
+geofetch 0.9.0
+
+```
+
+To see your CLI options, invoke `geofetch -h`:
+
+
+```bash
+geofetch -h
+```
+
+```.output
+usage: geofetch [-h] [-V] -i INPUT [-n NAME] [-m METADATA_ROOT]
+                [-u METADATA_FOLDER] [--just-metadata] [-r]
+                [--config-template CONFIG_TEMPLATE]
+                [--pipeline_samples PIPELINE_SAMPLES]
+                [--pipeline_project PIPELINE_PROJECT] [-k SKIP] [--acc-anno]
+                [--discard-soft] [--const-limit-project CONST_LIMIT_PROJECT]
+                [--const-limit-discard CONST_LIMIT_DISCARD]
+                [--attr-limit-truncate ATTR_LIMIT_TRUNCATE] [-p]
+                [--data-source {all,samples,series}] [--filter FILTER]
+                [--filter-size FILTER_SIZE] [-g GEO_FOLDER] [-x]
+                [-b BAM_FOLDER] [-f FQ_FOLDER] [--use-key-subset] [--silent]
+                [--verbosity V] [--logdev]
+
+Automatic GEO and SRA data downloader
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -V, --version         show program's version number and exit
+  -i INPUT, --input INPUT
+                        required: a GEO (GSE) accession, or a file with a list
+                        of GSE numbers
+  -n NAME, --name NAME  Specify a project name. Defaults to GSE number
+  -m METADATA_ROOT, --metadata-root METADATA_ROOT
+                        Specify a parent folder location to store metadata.
+                        The project name will be added as a subfolder
+                        [Default: $SRAMETA:]
+  -u METADATA_FOLDER, --metadata-folder METADATA_FOLDER
+                        Specify an absolute folder location to store metadata.
+                        No subfolder will be added. Overrides value of
+                        --metadata-root [Default: Not used (--metadata-root is
+                        used by default)]
+  --just-metadata       If set, don't actually run downloads, just create
+                        metadata
+  -r, --refresh-metadata
+                        If set, re-download metadata even if it exists.
+  --config-template CONFIG_TEMPLATE
+                        Project config yaml file template.
+  --pipeline_samples PIPELINE_SAMPLES
+                        Optional: Specify one or more filepaths to SAMPLES
+                        pipeline interface yaml files. These will be added to
+                        the project config file to make it immediately
+                        compatible with looper. [Default: null]
+  --pipeline_project PIPELINE_PROJECT
+                        Optional: Specify one or more filepaths to PROJECT
+                        pipeline interface yaml files. These will be added to
+                        the project config file to make it immediately
+                        compatible with looper. [Default: null]
+  -k SKIP, --skip SKIP  Skip some accessions. [Default: no skip].
+  --acc-anno            Optional: Produce annotation sheets for each
+                        accession. Project combined PEP for the whole project
+                        won't be produced.
+  --discard-soft        Optional: After creation of PEP files, all soft and
+                        additional files will be deleted
+  --const-limit-project CONST_LIMIT_PROJECT
+                        Optional: Limit of the number of the constant sample
+                        characters that should not be in project yaml.
+                        [Default: 50]
+  --const-limit-discard CONST_LIMIT_DISCARD
+                        Optional: Limit of the number of the constant sample
+                        characters that should not be discarded [Default: 250]
+  --attr-limit-truncate ATTR_LIMIT_TRUNCATE
+                        Optional: Limit of the number of sample characters.Any
+                        attribute with more than X characters will truncate to
+                        the first X, where X is a number of characters
+                        [Default: 500]
+  --silent              Silence logging. Overrides verbosity.
+  --verbosity V         Set logging level (1-5 or logging module level name)
+  --logdev              Expand content of logging message format.
+
+processed:
+  -p, --processed       Download processed data [Default: download raw data].
+  --data-source {all,samples,series}
+                        Optional: Specifies the source of data on the GEO
+                        record to retrieve processed data, which may be
+                        attached to the collective series entity, or to
+                        individual samples. Allowable values are: samples,
+                        series or both (all). Ignored unless 'processed' flag
+                        is set. [Default: samples]
+  --filter FILTER       Optional: Filter regex for processed filenames
+                        [Default: None].Ignored unless 'processed' flag is
+                        set.
+  --filter-size FILTER_SIZE
+                        Optional: Filter size for processed files that are
+                        stored as sample repository [Default: None]. Works
+                        only for sample data. Supported input formats : 12B,
+                        12KB, 12MB, 12GB. Ignored unless 'processed' flag is
+                        set.
+  -g GEO_FOLDER, --geo-folder GEO_FOLDER
+                        Optional: Specify a location to store processed GEO
+                        files. Ignored unless 'processed' flag is
+                        set.[Default: $GEODATA:]
+
+raw:
+  -x, --split-experiments
+                        Split SRR runs into individual samples. By default,
+                        SRX experiments with multiple SRR Runs will have a
+                        single entry in the annotation table, with each run as
+                        a separate row in the subannotation table. This
+                        setting instead treats each run as a separate sample
+  -b BAM_FOLDER, --bam-folder BAM_FOLDER
+                        Optional: Specify folder of bam files. Geofetch will
+                        not download sra files when corresponding bam files
+                        already exist. [Default: $SRABAM:]
+  -f FQ_FOLDER, --fq-folder FQ_FOLDER
+                        Optional: Specify folder of fastq files. Geofetch will
+                        not download sra files when corresponding fastq files
+                        already exist. [Default: $SRAFQ:]
+  --use-key-subset      Use just the keys defined in this module when writing
+                        out metadata.
+
+```
+
+Calling geofetch will do 4 tasks: 
+
+1. download all `.sra` files from `GSE#####` into your SRA folder (wherever you have configured `sratools` to stick data).
+2. download all metadata from GEO and SRA and store in your metadata folder.
+2. produce a PEP-compatible sample table, `PROJECT_NAME_annotation.csv`, in your metadata folder.
+3. produce a PEP-compatible project configuration file, `PROJECT_NAME_config.yaml`, in your metadata folder.
+
+Complete details about geofetch outputs is cataloged in the [metadata outputs reference](metadata_output.md).
+
+## Download the data
+
+First, create the metadata:
+
+
+```bash
+geofetch -i GSE67303 -n red_algae -m `pwd` --just-metadata
+```
+
+```.output
+Metadata folder: /home/nsheff/code/geofetch/docs_jupyter/red_algae
+Trying GSE67303 (not a file) as accession...
+Skipped 0 accessions. Starting now.
+Processing accession 1 of 1: 'GSE67303'
+Found previous GSE file: /home/nsheff/code/geofetch/docs_jupyter/red_algae/GSE67303_GSE.soft
+Found previous GSM file: /home/nsheff/code/geofetch/docs_jupyter/red_algae/GSE67303_GSM.soft
+Processed 4 samples.
+Found SRA Project accession: SRP056574
+Found previous SRA file: /home/nsheff/code/geofetch/docs_jupyter/red_algae/GSE67303_SRA.csv
+SRP: SRP056574
+Parsing SRA file to download SRR records
+Get SRR: SRR1930183 (SRX969073)
+Dry run (no data download)
+Get SRR: SRR1930184 (SRX969074)
+Dry run (no data download)
+Get SRR: SRR1930185 (SRX969075)
+Dry run (no data download)
+Get SRR: SRR1930186 (SRX969076)
+Dry run (no data download)
+Finished processing 1 accession(s)
+Creating complete project annotation sheets and config file...
+Sample annotation sheet: /home/nsheff/code/geofetch/docs_jupyter/red_algae/red_algae_annotation.csv
+Writing: /home/nsheff/code/geofetch/docs_jupyter/red_algae/red_algae_annotation.csv
+  Config file: /home/nsheff/code/geofetch/docs_jupyter/red_algae/red_algae_config.yaml
+
+```
+
+The `-m` parameter specifies to use the current directory, storing the data according to the name (`-n`) parameter. So, we'll now have a `red_alga` subfolder, where the results will be saved. Inside that folder you'll see the output of the command:
+
+
+```bash
+ls red_algae
+```
+
+```.output
+GSE67303_GSE.soft  GSE67303_SRA.csv       red_algae_annotation.csv  submission
+GSE67303_GSM.soft  GSE67303_SRA_filt.csv  red_algae_config.yaml
+
+```
+
+The `.soft` files are the direct output from GEO, which contain all the metadata as stored by GEO, for both the experiment (`_GSE`) and for the individual samples (`_GSM`). Geofetch also produces a `csv` file with the SRA metadata. The filtered version (ending in `_filt`) would contain only the specified subset of the samples if we didn't request them all, but in this case, since we only gave an accession, it is identical to the complete file.
+
+Finally, there are the 2 files that make up the PEP: the `_config.yaml` file and the `_annotation.csv` file. Let's see what's in these files now.
+
+
+```bash
+cat red_algae/red_algae_config.yaml
+```
+
+```.output
+# Autogenerated by geofetch
+
+name: red_algae
+pep_version: 2.0.0
+sample_table: red_algae_annotation.csv
+subsample_table: null
+
+looper:
+  output_dir: red_algae
+  pipeline_interfaces: null
+
+sample_modifiers:
+  append:
+    SRR_files: SRA
+    pipeline_interfaces: null
+  derive:
+    attributes: [read1, read2, SRR_files]
+    sources:
+      SRA: "${SRABAM}/{SRR}.bam"
+      FQ: "${SRAFQ}/{SRR}.fastq.gz"
+      FQ1: "${SRAFQ}/{SRR}_1.fastq.gz"
+      FQ2: "${SRAFQ}/{SRR}_2.fastq.gz"      
+  imply:
+    - if: 
+        organism: "Mus musculus"
+      then:
+        genome: mm10
+    - if: 
+        organism: "Homo sapiens"
+      then:
+        genome: hg38          
+    - if: 
+        read_type: "PAIRED"
+      then:
+        read1: FQ1
+        read2: FQ2          
+    - if: 
+        read_type: "SINGLE"
+      then:
+        read1: FQ1
+
+project_modifiers:
+  amend:
+    sra_convert:
+      looper:
+        results_subdir: sra_convert_results      
+      sample_modifiers:
+        append:
+          SRR_files: SRA
+          pipeline_interfaces: ${CODE}/geofetch/pipeline_interface_convert.yaml
+        derive:
+          attributes: [read1, read2, SRR_files]
+          sources:
+            SRA: "${SRARAW}/{SRR}.sra"
+            FQ: "${SRAFQ}/{SRR}.fastq.gz"
+            FQ1: "${SRAFQ}/{SRR}_1.fastq.gz"
+            FQ2: "${SRAFQ}/{SRR}_2.fastq.gz"
+
+
+```
+
+There are two important things to note in his file: First, see in the PEP that `sample_table` points to the csv file produced by geofetch. Second, look at the amendment called `sra_convert`. This adds a pipeline interface to the sra conversion pipeline, and adds derived attributes for SRA files and fastq files that rely on environment variables called `$SRARAW` and `$SRAFQ`. These environment variables should point to folders where you store your raw .sra files and the converted fastq files.
+
+Now let's look at the first 100 characters of the csv file:
+
+
+```bash
+cut -c -100 red_algae/red_algae_annotation.csv
+```
+
+```.output
+sample_name,protocol,organism,read_type,data_source,SRR,SRX,Sample_title,Sample_geo_accession,Sample
+Cm_BlueLight_Rep1,cDNA,Cyanidioschyzon merolae strain 10D,PAIRED,SRA,SRR1930183,SRX969073,Cm_BlueLig
+Cm_BlueLight_Rep2,cDNA,Cyanidioschyzon merolae strain 10D,PAIRED,SRA,SRR1930184,SRX969074,Cm_BlueLig
+Cm_Darkness_Rep1,cDNA,Cyanidioschyzon merolae strain 10D,PAIRED,SRA,SRR1930185,SRX969075,Cm_Darkness
+Cm_Darkness_Rep2,cDNA,Cyanidioschyzon merolae strain 10D,PAIRED,SRA,SRR1930186,SRX969076,Cm_Darkness
+
+```
+
+Now let's download the actual data.
+
+
+```bash
+geofetch -i GSE67303 -n red_algae -m `pwd`
+```
+
+```.output
+Metadata folder: /home/nsheff/code/geofetch/docs_jupyter/red_algae
+Trying GSE67303 (not a file) as accession...
+Skipped 0 accessions. Starting now.
+Processing accession 1 of 1: 'GSE67303'
+Found previous GSE file: /home/nsheff/code/geofetch/docs_jupyter/red_algae/GSE67303_GSE.soft
+Found previous GSM file: /home/nsheff/code/geofetch/docs_jupyter/red_algae/GSE67303_GSM.soft
+Processed 4 samples.
+Found SRA Project accession: SRP056574
+Found previous SRA file: /home/nsheff/code/geofetch/docs_jupyter/red_algae/GSE67303_SRA.csv
+SRP: SRP056574
+Parsing SRA file to download SRR records
+Get SRR: SRR1930183 (SRX969073)
+
+2020-05-21T20:20:24 prefetch.2.10.0: 1) Downloading 'SRR1930183'...
+2020-05-21T20:20:24 prefetch.2.10.0:  Downloading via https...
+2020-05-21T20:24:56 prefetch.2.10.0:  https download succeed
+2020-05-21T20:24:56 prefetch.2.10.0: 1) 'SRR1930183' was downloaded successfully
+2020-05-21T20:24:56 prefetch.2.10.0: 'SRR1930183' has 0 unresolved dependencies
+Get SRR: SRR1930184 (SRX969074)
+
+2020-05-21T20:24:58 prefetch.2.10.0: 1) Downloading 'SRR1930184'...
+2020-05-21T20:24:58 prefetch.2.10.0:  Downloading via https...
+2020-05-21T20:27:20 prefetch.2.10.0:  https download succeed
+2020-05-21T20:27:20 prefetch.2.10.0: 1) 'SRR1930184' was downloaded successfully
+2020-05-21T20:27:20 prefetch.2.10.0: 'SRR1930184' has 0 unresolved dependencies
+Get SRR: SRR1930185 (SRX969075)
+
+2020-05-21T20:27:21 prefetch.2.10.0: 1) Downloading 'SRR1930185'...
+2020-05-21T20:27:21 prefetch.2.10.0:  Downloading via https...
+2020-05-21T20:33:40 prefetch.2.10.0:  https download succeed
+2020-05-21T20:33:40 prefetch.2.10.0: 1) 'SRR1930185' was downloaded successfully
+2020-05-21T20:33:40 prefetch.2.10.0: 'SRR1930185' has 0 unresolved dependencies
+Get SRR: SRR1930186 (SRX969076)
+
+2020-05-21T20:33:42 prefetch.2.10.0: 1) Downloading 'SRR1930186'...
+2020-05-21T20:33:42 prefetch.2.10.0:  Downloading via https...
+2020-05-21T20:38:41 prefetch.2.10.0:  https download succeed
+2020-05-21T20:38:41 prefetch.2.10.0: 1) 'SRR1930186' was downloaded successfully
+2020-05-21T20:38:41 prefetch.2.10.0: 'SRR1930186' has 0 unresolved dependencies
+Finished processing 1 accession(s)
+Creating complete project annotation sheets and config file...
+Sample annotation sheet: /home/nsheff/code/geofetch/docs_jupyter/red_algae/red_algae_annotation.csv
+Writing: /home/nsheff/code/geofetch/docs_jupyter/red_algae/red_algae_annotation.csv
+  Config file: /home/nsheff/code/geofetch/docs_jupyter/red_algae/red_algae_config.yaml
+
+```
+
+## Convert to fastq format
+
+Now the `.sra` files have been downloaded. The project that was automatically created by GEO contained an amendment for sra file conversion. This project expects you to have an environment variable called `SRARAW` that points to the location where `prefetch` stores your `.sra` files. We also should define a `$SRAFQ` variable to point to where we ant the fastq files stored. In this command below, we set these on the fly for this command, but you can also just use globals.
+
+We'll use `-d` first to do a dry run:
+
+
+```bash
+SRARAW=${HOME}/ncbi/public/sra/ SRAFQ=red_algae/fastq \
+  looper run red_algae/red_algae_config.yaml -a sra_convert -p local -d
+```
+
+```.output
+Looper version: 1.2.0-dev
+Command: run
+Using amendments: sra_convert
+Activating compute package 'local'
+## [1 of 4] sample: Cm_BlueLight_Rep1; pipeline: sra_convert
+Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_BlueLight_Rep1.sub
+Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_BlueLight_Rep1.sub
+Dry run, not submitted
+## [2 of 4] sample: Cm_BlueLight_Rep2; pipeline: sra_convert
+Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_BlueLight_Rep2.sub
+Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_BlueLight_Rep2.sub
+Dry run, not submitted
+## [3 of 4] sample: Cm_Darkness_Rep1; pipeline: sra_convert
+Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_Darkness_Rep1.sub
+Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_Darkness_Rep1.sub
+Dry run, not submitted
+## [4 of 4] sample: Cm_Darkness_Rep2; pipeline: sra_convert
+Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_Darkness_Rep2.sub
+Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_Darkness_Rep2.sub
+Dry run, not submitted
+
+Looper finished
+Samples valid for job generation: 4 of 4
+Commands submitted: 4 of 4
+Jobs submitted: 4
+Dry run. No jobs were actually submitted.
+
+```
+
+And now the real thing:
+
+
+```bash
+SRARAW=${HOME}/ncbi/public/sra/ SRAFQ=red_algae/fastq \
+  looper run red_algae/red_algae_config.yaml -a sra_convert -p local \
+  --command-extra=--keep-sra
+```
+
+```.output
+Looper version: 1.2.0-dev
+Command: run
+Using amendments: sra_convert
+Activating compute package 'local'
+## [1 of 4] sample: Cm_BlueLight_Rep1; pipeline: sra_convert
+Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_BlueLight_Rep1.sub
+Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_BlueLight_Rep1.sub
+Compute node: zither
+Start time: 2020-05-21 17:40:56
+Using outfolder: red_algae/results_pipeline/SRX969073
+### Pipeline run code and environment:
+
+*              Command:  `/home/nsheff/.local/bin/sraconvert --srr /home/nsheff/ncbi/public/sra//SRR1930183.sra --sample-name SRX969073 -O red_algae/results_pipeline --keep-sra`
+*         Compute host:  zither
+*          Working dir:  /home/nsheff/code/geofetch/docs_jupyter
+*            Outfolder:  red_algae/results_pipeline/SRX969073/
+*  Pipeline started at:   (05-21 17:40:57) elapsed: 0.0 _TIME_
+
+### Version log:
+
+*       Python version:  3.7.5
+*          Pypiper dir:  `/home/nsheff/.local/lib/python3.7/site-packages/pypiper`
+*      Pypiper version:  0.12.1
+*         Pipeline dir:  `/home/nsheff/.local/bin`
+*     Pipeline version:  None
+
+### Arguments passed to pipeline:
+
+*          `bamfolder`:  ``
+*        `config_file`:  `sraconvert.yaml`
+*             `format`:  `fastq`
+*           `fqfolder`:  `red_algae/fastq`
+*           `keep_sra`:  `True`
+*             `logdev`:  `False`
+*               `mode`:  `convert`
+*      `output_parent`:  `red_algae/results_pipeline`
+*            `recover`:  `False`
+*        `sample_name`:  `['SRX969073']`
+*             `silent`:  `False`
+*          `srafolder`:  `/home/nsheff/ncbi/public/sra/`
+*                `srr`:  `['/home/nsheff/ncbi/public/sra//SRR1930183.sra']`
+*          `verbosity`:  `None`
+
+----------------------------------------
+
+Processing 1 of 1 files: SRR1930183
+Target to produce: `red_algae/fastq/SRR1930183_1.fastq.gz`  
+
+> `fastq-dump /home/nsheff/ncbi/public/sra//SRR1930183.sra --split-files --gzip -O red_algae/fastq` (9436)
+<pre>
+Read 1068319 spots for /home/nsheff/ncbi/public/sra//SRR1930183.sra
+Written 1068319 spots for /home/nsheff/ncbi/public/sra//SRR1930183.sra
+</pre>
+Command completed. Elapsed time: 0:00:38. Running peak memory: 0.067GB.  
+  PID: 9436;	Command: fastq-dump;	Return code: 0;	Memory used: 0.067GB
+
+Already completed files: []
+
+### Pipeline completed. Epilogue
+*        Elapsed time (this run):  0:00:38
+*  Total elapsed time (all runs):  0:00:38
+*         Peak memory (this run):  0.0666 GB
+*        Pipeline completed time: 2020-05-21 17:41:35
+## [2 of 4] sample: Cm_BlueLight_Rep2; pipeline: sra_convert
+Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_BlueLight_Rep2.sub
+Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_BlueLight_Rep2.sub
+Compute node: zither
+Start time: 2020-05-21 17:41:36
+Using outfolder: red_algae/results_pipeline/SRX969074
+### Pipeline run code and environment:
+
+*              Command:  `/home/nsheff/.local/bin/sraconvert --srr /home/nsheff/ncbi/public/sra//SRR1930184.sra --sample-name SRX969074 -O red_algae/results_pipeline --keep-sra`
+*         Compute host:  zither
+*          Working dir:  /home/nsheff/code/geofetch/docs_jupyter
+*            Outfolder:  red_algae/results_pipeline/SRX969074/
+*  Pipeline started at:   (05-21 17:41:36) elapsed: 0.0 _TIME_
+
+### Version log:
+
+*       Python version:  3.7.5
+*          Pypiper dir:  `/home/nsheff/.local/lib/python3.7/site-packages/pypiper`
+*      Pypiper version:  0.12.1
+*         Pipeline dir:  `/home/nsheff/.local/bin`
+*     Pipeline version:  None
+
+### Arguments passed to pipeline:
+
+*          `bamfolder`:  ``
+*        `config_file`:  `sraconvert.yaml`
+*             `format`:  `fastq`
+*           `fqfolder`:  `red_algae/fastq`
+*           `keep_sra`:  `True`
+*             `logdev`:  `False`
+*               `mode`:  `convert`
+*      `output_parent`:  `red_algae/results_pipeline`
+*            `recover`:  `False`
+*        `sample_name`:  `['SRX969074']`
+*             `silent`:  `False`
+*          `srafolder`:  `/home/nsheff/ncbi/public/sra/`
+*                `srr`:  `['/home/nsheff/ncbi/public/sra//SRR1930184.sra']`
+*          `verbosity`:  `None`
+
+----------------------------------------
+
+Processing 1 of 1 files: SRR1930184
+Target exists: `red_algae/fastq/SRR1930184_1.fastq.gz`  
+Already completed files: []
+
+### Pipeline completed. Epilogue
+*        Elapsed time (this run):  0:00:00
+*  Total elapsed time (all runs):  0:00:00
+*         Peak memory (this run):  0 GB
+*        Pipeline completed time: 2020-05-21 17:41:36
+## [3 of 4] sample: Cm_Darkness_Rep1; pipeline: sra_convert
+Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_Darkness_Rep1.sub
+Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_Darkness_Rep1.sub
+Compute node: zither
+Start time: 2020-05-21 17:41:36
+Using outfolder: red_algae/results_pipeline/SRX969075
+### Pipeline run code and environment:
+
+*              Command:  `/home/nsheff/.local/bin/sraconvert --srr /home/nsheff/ncbi/public/sra//SRR1930185.sra --sample-name SRX969075 -O red_algae/results_pipeline --keep-sra`
+*         Compute host:  zither
+*          Working dir:  /home/nsheff/code/geofetch/docs_jupyter
+*            Outfolder:  red_algae/results_pipeline/SRX969075/
+*  Pipeline started at:   (05-21 17:41:36) elapsed: 0.0 _TIME_
+
+### Version log:
+
+*       Python version:  3.7.5
+*          Pypiper dir:  `/home/nsheff/.local/lib/python3.7/site-packages/pypiper`
+*      Pypiper version:  0.12.1
+*         Pipeline dir:  `/home/nsheff/.local/bin`
+*     Pipeline version:  None
+
+### Arguments passed to pipeline:
+
+*          `bamfolder`:  ``
+*        `config_file`:  `sraconvert.yaml`
+*             `format`:  `fastq`
+*           `fqfolder`:  `red_algae/fastq`
+*           `keep_sra`:  `True`
+*             `logdev`:  `False`
+*               `mode`:  `convert`
+*      `output_parent`:  `red_algae/results_pipeline`
+*            `recover`:  `False`
+*        `sample_name`:  `['SRX969075']`
+*             `silent`:  `False`
+*          `srafolder`:  `/home/nsheff/ncbi/public/sra/`
+*                `srr`:  `['/home/nsheff/ncbi/public/sra//SRR1930185.sra']`
+*          `verbosity`:  `None`
+
+----------------------------------------
+
+Processing 1 of 1 files: SRR1930185
+Target to produce: `red_algae/fastq/SRR1930185_1.fastq.gz`  
+
+> `fastq-dump /home/nsheff/ncbi/public/sra//SRR1930185.sra --split-files --gzip -O red_algae/fastq` (9607)
+<pre>
+Read 1707508 spots for /home/nsheff/ncbi/public/sra//SRR1930185.sra
+Written 1707508 spots for /home/nsheff/ncbi/public/sra//SRR1930185.sra
+</pre>
+Command completed. Elapsed time: 0:01:01. Running peak memory: 0.066GB.  
+  PID: 9607;	Command: fastq-dump;	Return code: 0;	Memory used: 0.066GB
+
+Already completed files: []
+
+### Pipeline completed. Epilogue
+*        Elapsed time (this run):  0:01:01
+*  Total elapsed time (all runs):  0:01:01
+*         Peak memory (this run):  0.0656 GB
+*        Pipeline completed time: 2020-05-21 17:42:37
+## [4 of 4] sample: Cm_Darkness_Rep2; pipeline: sra_convert
+Writing script to /home/nsheff/code/geofetch/docs_jupyter/red_algae/submission/sra_convert_Cm_Darkness_Rep2.sub
+Job script (n=1; 0.00Gb): red_algae/submission/sra_convert_Cm_Darkness_Rep2.sub
+Compute node: zither
+Start time: 2020-05-21 17:42:38
+Using outfolder: red_algae/results_pipeline/SRX969076
+### Pipeline run code and environment:
+
+*              Command:  `/home/nsheff/.local/bin/sraconvert --srr /home/nsheff/ncbi/public/sra//SRR1930186.sra --sample-name SRX969076 -O red_algae/results_pipeline --keep-sra`
+*         Compute host:  zither
+*          Working dir:  /home/nsheff/code/geofetch/docs_jupyter
+*            Outfolder:  red_algae/results_pipeline/SRX969076/
+*  Pipeline started at:   (05-21 17:42:38) elapsed: 0.0 _TIME_
+
+### Version log:
+
+*       Python version:  3.7.5
+*          Pypiper dir:  `/home/nsheff/.local/lib/python3.7/site-packages/pypiper`
+*      Pypiper version:  0.12.1
+*         Pipeline dir:  `/home/nsheff/.local/bin`
+*     Pipeline version:  None
+
+### Arguments passed to pipeline:
+
+*          `bamfolder`:  ``
+*        `config_file`:  `sraconvert.yaml`
+*             `format`:  `fastq`
+*           `fqfolder`:  `red_algae/fastq`
+*           `keep_sra`:  `True`
+*             `logdev`:  `False`
+*               `mode`:  `convert`
+*      `output_parent`:  `red_algae/results_pipeline`
+*            `recover`:  `False`
+*        `sample_name`:  `['SRX969076']`
+*             `silent`:  `False`
+*          `srafolder`:  `/home/nsheff/ncbi/public/sra/`
+*                `srr`:  `['/home/nsheff/ncbi/public/sra//SRR1930186.sra']`
+*          `verbosity`:  `None`
+
+----------------------------------------
+
+Processing 1 of 1 files: SRR1930186
+Target to produce: `red_algae/fastq/SRR1930186_1.fastq.gz`  
+
+> `fastq-dump /home/nsheff/ncbi/public/sra//SRR1930186.sra --split-files --gzip -O red_algae/fastq` (9780)
+<pre>
+Read 1224029 spots for /home/nsheff/ncbi/public/sra//SRR1930186.sra
+Written 1224029 spots for /home/nsheff/ncbi/public/sra//SRR1930186.sra
+</pre>
+Command completed. Elapsed time: 0:00:44. Running peak memory: 0.067GB.  
+  PID: 9780;	Command: fastq-dump;	Return code: 0;	Memory used: 0.067GB
+
+Already completed files: []
+
+### Pipeline completed. Epilogue
+*        Elapsed time (this run):  0:00:44
+*  Total elapsed time (all runs):  0:00:44
+*         Peak memory (this run):  0.0673 GB
+*        Pipeline completed time: 2020-05-21 17:43:22
+
+Looper finished
+Samples valid for job generation: 4 of 4
+Commands submitted: 4 of 4
+Jobs submitted: 4
+
+```
+
+Now that's done, let's take a look in the `red_algae/fastq` folder (where we set the `$SRAFQ` variable).
+
+
+```bash
+ls red_algae/fastq
+```
+
+```.output
+SRR1930183_1.fastq.gz  SRR1930184_2.fastq.gz  SRR1930186_1.fastq.gz
+SRR1930183_2.fastq.gz  SRR1930185_1.fastq.gz  SRR1930186_2.fastq.gz
+SRR1930184_1.fastq.gz  SRR1930185_2.fastq.gz
+
+```
+
+By default, the sra conversion script will delete the `.sra` files after they have been converted to fastq. You can keep them if you want by passing `--keep-sra`, which you can do by passing `--command-extra=--keep-sra` to your `looper run` command.
+
+
+## Finalize the project config and sample annotation
+
+That's basically it! `geofetch` will have produced a general-purpose PEP for you, but you'll need to modify it for whatever purpose you have. For example, one common thing is to link to the pipeline you want to use by adding a `pipeline_interface` to the project config file. You may also need to adjust the `sample_annotation` file to make sure you have the right column names and attributes needed by the pipeline you're using. GEO submitters are notoriously bad at getting the metadata correct.
+
+
+## Selecting samples to download.
+
+By default, `geofetch` downloads all the data for one accession of interest. If you need more fine-grained control, either because you have multiple accessions or you need a subset of samples within them, you can use the [file-based sample specification](file-specification.md).
+
+
+## Tips
+
+* Set an environment variable for `$SRABAM` (where `.bam` files will live), and `geofetch` will check to see if you have an already-converted bamfile there before issuing the command to download the `sra` file. In this way, you can delete old `sra` files after conversion and not have to worry about re-downloading them. 
+
+* The config template uses an environment variable `$SRARAW` for where `.sra` files will live. If you set this variable to the same place you instructed `sratoolkit` to download `sra` files, you won't have to tweak the config file. For more information refer to the [`sratools` page](howto-location.md).
+
+You can find a complete example of [using `geofetch` for RNA-seq data](https://github.com/databio/example-projects/tree/master/rna-seq). 
+
diff --git a/docs_jupyter/processed-data-downloading.ipynb b/docs_jupyter/processed-data-downloading.ipynb
index dfcf672..9fba088 100644
--- a/docs_jupyter/processed-data-downloading.ipynb
+++ b/docs_jupyter/processed-data-downloading.ipynb
@@ -11,19 +11,19 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 1,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "geofetch 0.8.0\n"
+      "geofetch 0.9.0\n"
      ]
     }
    ],
    "source": [
-    "geofetch -V"
+    "geofetch --version"
    ]
   },
   {
@@ -35,7 +35,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 2,
    "metadata": {
     "scrolled": true
    },
@@ -46,12 +46,16 @@
      "text": [
       "usage: geofetch [-h] [-V] -i INPUT [-n NAME] [-m METADATA_ROOT]\n",
       "                [-u METADATA_FOLDER] [--just-metadata] [-r]\n",
-      "                [--config-template CONFIG_TEMPLATE] [-P PIPELINE_INTERFACES]\n",
-      "                [-k SKIP] [--acc-anno] [--use-key-subset] [-p]\n",
+      "                [--config-template CONFIG_TEMPLATE]\n",
+      "                [--pipeline_samples PIPELINE_SAMPLES]\n",
+      "                [--pipeline_project PIPELINE_PROJECT] [-k SKIP] [--acc-anno]\n",
+      "                [--discard-soft] [--const-limit-project CONST_LIMIT_PROJECT]\n",
+      "                [--const-limit-discard CONST_LIMIT_DISCARD]\n",
+      "                [--attr-limit-truncate ATTR_LIMIT_TRUNCATE] [-p]\n",
       "                [--data-source {all,samples,series}] [--filter FILTER]\n",
       "                [--filter-size FILTER_SIZE] [-g GEO_FOLDER] [-x]\n",
-      "                [-b BAM_FOLDER] [-f FQ_FOLDER] [--silent] [--verbosity V]\n",
-      "                [--logdev]\n",
+      "                [-b BAM_FOLDER] [-f FQ_FOLDER] [--use-key-subset] [--silent]\n",
+      "                [--verbosity V] [--logdev]\n",
       "\n",
       "Automatic GEO and SRA data downloader\n",
       "\n",
@@ -77,16 +81,34 @@
       "                        If set, re-download metadata even if it exists.\n",
       "  --config-template CONFIG_TEMPLATE\n",
       "                        Project config yaml file template.\n",
-      "  -P PIPELINE_INTERFACES, --pipeline_interfaces PIPELINE_INTERFACES\n",
-      "                        Optional: Specify one or more filepaths to pipeline\n",
-      "                        interface yaml files. These will be added to the\n",
-      "                        project config file to make it immediately compatible\n",
-      "                        with looper. [Default: null]\n",
+      "  --pipeline_samples PIPELINE_SAMPLES\n",
+      "                        Optional: Specify one or more filepaths to SAMPLES\n",
+      "                        pipeline interface yaml files. These will be added to\n",
+      "                        the project config file to make it immediately\n",
+      "                        compatible with looper. [Default: null]\n",
+      "  --pipeline_project PIPELINE_PROJECT\n",
+      "                        Optional: Specify one or more filepaths to PROJECT\n",
+      "                        pipeline interface yaml files. These will be added to\n",
+      "                        the project config file to make it immediately\n",
+      "                        compatible with looper. [Default: null]\n",
       "  -k SKIP, --skip SKIP  Skip some accessions. [Default: no skip].\n",
-      "  --acc-anno            Also produce annotation sheets for each accession, not\n",
-      "                        just for the whole project combined\n",
-      "  --use-key-subset      Use just the keys defined in this module when writing\n",
-      "                        out metadata.\n",
+      "  --acc-anno            Optional: Produce annotation sheets for each\n",
+      "                        accession. Project combined PEP for the whole project\n",
+      "                        won't be produced.\n",
+      "  --discard-soft        Optional: After creation of PEP files, all soft and\n",
+      "                        additional files will be deleted\n",
+      "  --const-limit-project CONST_LIMIT_PROJECT\n",
+      "                        Optional: Limit of the number of the constant sample\n",
+      "                        characters that should not be in project yaml.\n",
+      "                        [Default: 50]\n",
+      "  --const-limit-discard CONST_LIMIT_DISCARD\n",
+      "                        Optional: Limit of the number of the constant sample\n",
+      "                        characters that should not be discarded [Default: 250]\n",
+      "  --attr-limit-truncate ATTR_LIMIT_TRUNCATE\n",
+      "                        Optional: Limit of the number of sample characters.Any\n",
+      "                        attribute with more than X characters will truncate to\n",
+      "                        the first X, where X is a number of characters\n",
+      "                        [Default: 500]\n",
       "  --silent              Silence logging. Overrides verbosity.\n",
       "  --verbosity V         Set logging level (1-5 or logging module level name)\n",
       "  --logdev              Expand content of logging message format.\n",
@@ -99,15 +121,16 @@
       "                        attached to the collective series entity, or to\n",
       "                        individual samples. Allowable values are: samples,\n",
       "                        series or both (all). Ignored unless 'processed' flag\n",
-      "                        is set. [Default: all]\n",
+      "                        is set. [Default: samples]\n",
       "  --filter FILTER       Optional: Filter regex for processed filenames\n",
       "                        [Default: None].Ignored unless 'processed' flag is\n",
       "                        set.\n",
       "  --filter-size FILTER_SIZE\n",
       "                        Optional: Filter size for processed files that are\n",
-      "                        stored as sample repository [Default: None]. Supported\n",
-      "                        input formats : 12B, 12KB, 12MB, 12GB. Ignored unless\n",
-      "                        'processed' flag is set.\n",
+      "                        stored as sample repository [Default: None]. Works\n",
+      "                        only for sample data. Supported input formats : 12B,\n",
+      "                        12KB, 12MB, 12GB. Ignored unless 'processed' flag is\n",
+      "                        set.\n",
       "  -g GEO_FOLDER, --geo-folder GEO_FOLDER\n",
       "                        Optional: Specify a location to store processed GEO\n",
       "                        files. Ignored unless 'processed' flag is\n",
@@ -127,7 +150,9 @@
       "  -f FQ_FOLDER, --fq-folder FQ_FOLDER\n",
       "                        Optional: Specify folder of fastq files. Geofetch will\n",
       "                        not download sra files when corresponding fastq files\n",
-      "                        already exist. [Default: $SRAFQ:]\n"
+      "                        already exist. [Default: $SRAFQ:]\n",
+      "  --use-key-subset      Use just the keys defined in this module when writing\n",
+      "                        out metadata.\n"
      ]
     }
    ],
@@ -160,7 +185,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 3,
    "metadata": {},
    "outputs": [
     {
@@ -170,34 +195,34 @@
       "Metadata folder: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test\n",
       "Trying GSE185701 (not a file) as accession...\n",
       "Skipped 0 accessions. Starting now.\n",
-      "\u001b[38;5;228mProcessing accession 1 of 1: 'GSE185701'\u001b[0m\n",
-      "--2022-03-10 11:32:18--  https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?targ=gse&acc=GSE185701&form=text&view=full\n",
+      "\u001b[38;5;200mProcessing accession 1 of 1: 'GSE185701'\u001b[0m\n",
+      "--2022-06-10 14:43:11--  https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?targ=gse&acc=GSE185701&form=text&view=full\n",
       "Resolving www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)... 2607:f220:41e:4290::110, 130.14.29.110\n",
       "Connecting to www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)|2607:f220:41e:4290::110|:443... connected.\n",
       "HTTP request sent, awaiting response... 200 OK\n",
       "Length: unspecified [geo/text]\n",
       "Saving to: ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_GSE.soft’\n",
       "\n",
-      "/home/bnt4me/Virgin     [ <=>                ]   2.79K  --.-KB/s    in 0s      \n",
+      "/home/bnt4me/Virgin     [ <=>                ]   2.82K  --.-KB/s    in 0s      \n",
       "\n",
-      "2022-03-10 11:32:18 (262 MB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_GSE.soft’ saved [2855]\n",
+      "2022-06-10 14:43:12 (207 MB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_GSE.soft’ saved [2885]\n",
       "\n",
-      "--2022-03-10 11:32:18--  https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?targ=gsm&acc=GSE185701&form=text&view=full\n",
+      "--2022-06-10 14:43:12--  https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?targ=gsm&acc=GSE185701&form=text&view=full\n",
       "Resolving www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)... 2607:f220:41e:4290::110, 130.14.29.110\n",
       "Connecting to www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)|2607:f220:41e:4290::110|:443... connected.\n",
       "HTTP request sent, awaiting response... 200 OK\n",
       "Length: unspecified [geo/text]\n",
       "Saving to: ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_GSM.soft’\n",
       "\n",
-      "/home/bnt4me/Virgin     [  <=>               ]  39.51K   139KB/s    in 0.3s    \n",
+      "/home/bnt4me/Virgin     [  <=>               ]  39.51K   110KB/s    in 0.4s    \n",
       "\n",
-      "2022-03-10 11:32:19 (139 KB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_GSM.soft’ saved [40454]\n",
+      "2022-06-10 14:43:12 (110 KB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_GSM.soft’ saved [40454]\n",
       "\n",
       "\u001b[38;5;242m\n",
-      "--2022-03-10 11:32:19--  ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE185nnn/GSE185701/suppl/filelist.txt\n",
+      "--2022-06-10 14:43:12--  ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE185nnn/GSE185701/suppl/filelist.txt\n",
       "           => ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_file_list.txt’\n",
-      "Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41e:250::10, 2607:f220:41f:250::228, 130.14.250.11, ...\n",
-      "Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41e:250::10|:21... connected.\n",
+      "Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41f:250::228, 2607:f220:41f:250::229, 130.14.250.7, ...\n",
+      "Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41f:250::228|:21... connected.\n",
       "Logging in as anonymous ... Logged in!\n",
       "==> SYST ... done.    ==> PWD ... done.\n",
       "==> TYPE I ... done.  ==> CWD (1) /geo/series/GSE185nnn/GSE185701/suppl ... done.\n",
@@ -205,22 +230,20 @@
       "==> EPSV ... done.    ==> RETR filelist.txt ... done.\n",
       "Length: 794 (unauthoritative)\n",
       "\n",
-      "filelist.txt        100%[===================>]     794  --.-KB/s    in 0.04s   \n",
+      "filelist.txt        100%[===================>]     794  --.-KB/s    in 0.001s  \n",
       "\n",
-      "2022-03-10 11:32:20 (18.3 KB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_file_list.txt’ saved [794]\n",
+      "2022-06-10 14:43:13 (716 KB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_file_list.txt’ saved [794]\n",
       "\n",
       "\u001b[38;5;242m0\u001b[0m\n",
       "\u001b[0m\n",
       "Total number of processed SAMPLES files found is: 8\n",
       "Total number of processed SERIES files found is: 1\n",
-      "Finished processing 1 accession(s)\n",
       "Expanding metadata list...\n",
+      "Expanding metadata list...\n",
+      "Finished processing 1 accession(s)\n",
       "Unifying and saving of metadata... \n",
-      "\u001b[92mFile /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/bright_test_annotation_sample_processed.csv has been saved successfully\u001b[0m\n",
-      "  Config file: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/bright_test_annotation_sample_processed.yaml\n",
-      "Unifying and saving of metadata... \n",
-      "\u001b[92mFile /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/bright_test_annotation_series_processed.csv has been saved successfully\u001b[0m\n",
-      "  Config file: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/bright_test_annotation_series_processed.yaml\n"
+      "\u001b[92mFile /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_samples.csv has been saved successfully\u001b[0m\n",
+      "  Config file: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_samples.yaml\n"
      ]
     }
    ],
@@ -230,17 +253,15 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 4,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "bright_test_annotation_sample_processed.csv   GSE185701_file_list.txt\n",
-      "bright_test_annotation_sample_processed.yaml  GSE185701_GSE.soft\n",
-      "bright_test_annotation_series_processed.csv   GSE185701_GSM.soft\n",
-      "bright_test_annotation_series_processed.yaml\n"
+      "GSE185701_file_list.txt  GSE185701_GSM.soft     GSE185701_samples.yaml\n",
+      "GSE185701_GSE.soft       GSE185701_samples.csv\n"
      ]
     }
    ],
@@ -259,45 +280,54 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 10,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "pep_version: 2.0.0\n",
-      "project_name: bright_test\n",
-      "sample_table: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/bright_test_annotation_sample_processed.csv\n",
+      "# Autogenerated by geofetch\n",
+      "\n",
+      "pep_version: 2.1.0\n",
+      "project_name: GSE185701\n",
+      "sample_table: GSE185701_samples.csv\n",
       "\n",
       "sample_modifiers:\n",
       "  append:\n",
       "    output_file_path: FILES\n",
+      "    Sample_growth_protocol_ch1: Huh 7 was cultured in Dulbecco’s modified Eagle’s medium (DMEM) (Invitrogen, Carlsbad, CA, USA) containing 10% fetal bovine serum (FBS) (HyClone, Logan, UT, USA) and antibiotics (penicillin and streptomycin, Invitrogen) at 37 °C in 5% CO2.\n",
+      "    \n",
       "  derive:\n",
       "    attributes: [output_file_path]\n",
       "    sources:\n",
-      "      FILES: /{SRA}/{sample_name}\n",
+      "      FILES: /{GSE}/{file}\n",
+      "\n",
       "\n",
       "\n"
      ]
     }
    ],
    "source": [
-    "cat bright_test/bright_test_annotation_sample_processed.yaml"
+    "cat bright_test/GSE185701_samples.yaml"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "There are two important things to note in his file: First, see in the PEP that `sample_table` points to the csv file produced by geofetch. Second: output_file_path is location of all the files.\n",
+    "There are few important things to note in this file:\n",
+    "\n",
+    "* First, see in the PEP that `sample_table` points to the csv file produced by geofetch.\n",
+    "* Second: output_file_path is location of all the files. \n",
+    "* Third: sample_modifier Sample_growth_protocol_ch1 is constant sample character and is larger then 50 characters so it is deleted from csv file. For large project it can significantly reduced size of the metadata\n",
     "\n",
     "Now let's look at the first 100 characters of the csv file:"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": 11,
    "metadata": {
     "scrolled": true
    },
@@ -319,7 +349,7 @@
     }
    ],
    "source": [
-    "cut -c -100 bright_test/bright_test_annotation_sample_processed.csv"
+    "cut -c -100 bright_test/GSE185701_samples.csv"
    ]
   },
   {
@@ -327,37 +357,78 @@
    "metadata": {},
    "source": [
     "Now let's download the actual data. This time we will will be downloading data from the [GSE185701 data set](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE185701) .\n",
+    "\n",
     "Let's additionally add few arguments:\n",
-    "- _geo-folder_ (required) - path to the location where processed files have to be saved\n",
-    "- _filter_ argument, to download only _bed_ files  (--filter \".Bed.gz$\") \n",
-    "- _data-source_ argument, to download files only from sample location (--data-source samples)"
+    "\n",
+    "* _geo-folder_ (required) - path to the location where processed files have to be saved\n",
+    "* _filter_ argument, to download only _bed_ files  (--filter \".Bed.gz$\")\n",
+    "* _data-source_ argument, to download files only from sample location (--data-source samples)"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": 12,
    "metadata": {},
    "outputs": [
     {
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Metadata folder: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test\n",
+      "Metadata folder: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter\n",
       "Trying GSE185701 (not a file) as accession...\n",
       "Skipped 0 accessions. Starting now.\n",
-      "\u001b[38;5;228mProcessing accession 1 of 1: 'GSE185701'\u001b[0m\n",
-      "Found previous GSE file: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_GSE.soft\n",
-      "Found previous GSM file: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_GSM.soft\n",
-      "\u001b[38;5;242mFile /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/GSE185701_file_list.txt exists.\u001b[0m\n",
+      "\u001b[38;5;200mProcessing accession 1 of 1: 'GSE185701'\u001b[0m\n",
+      "--2022-06-10 14:53:15--  https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?targ=gse&acc=GSE185701&form=text&view=full\n",
+      "Resolving www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)... 2607:f220:41e:4290::110, 130.14.29.110\n",
+      "Connecting to www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)|2607:f220:41e:4290::110|:443... connected.\n",
+      "HTTP request sent, awaiting response... 200 OK\n",
+      "Length: unspecified [geo/text]\n",
+      "Saving to: ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_GSE.soft’\n",
+      "\n",
+      "/home/bnt4me/Virgin     [ <=>                ]   2.82K  --.-KB/s    in 0s      \n",
+      "\n",
+      "2022-06-10 14:53:16 (898 MB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_GSE.soft’ saved [2885]\n",
+      "\n",
+      "--2022-06-10 14:53:16--  https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?targ=gsm&acc=GSE185701&form=text&view=full\n",
+      "Resolving www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)... 2607:f220:41e:4290::110, 130.14.29.110\n",
+      "Connecting to www.ncbi.nlm.nih.gov (www.ncbi.nlm.nih.gov)|2607:f220:41e:4290::110|:443... connected.\n",
+      "HTTP request sent, awaiting response... 200 OK\n",
+      "Length: unspecified [geo/text]\n",
+      "Saving to: ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_GSM.soft’\n",
+      "\n",
+      "/home/bnt4me/Virgin     [  <=>               ]  39.51K   180KB/s    in 0.2s    \n",
+      "\n",
+      "2022-06-10 14:53:16 (180 KB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_GSM.soft’ saved [40454]\n",
+      "\n",
+      "\u001b[38;5;242m\n",
+      "--2022-06-10 14:53:16--  ftp://ftp.ncbi.nlm.nih.gov/geo/series/GSE185nnn/GSE185701/suppl/filelist.txt\n",
+      "           => ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_file_list.txt’\n",
+      "Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41f:250::230, 2607:f220:41e:250::12, 165.112.9.228, ...\n",
+      "Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41f:250::230|:21... connected.\n",
+      "Logging in as anonymous ... Logged in!\n",
+      "==> SYST ... done.    ==> PWD ... done.\n",
+      "==> TYPE I ... done.  ==> CWD (1) /geo/series/GSE185nnn/GSE185701/suppl ... done.\n",
+      "==> SIZE filelist.txt ... 794\n",
+      "==> EPSV ... done.    ==> RETR filelist.txt ... done.\n",
+      "Length: 794 (unauthoritative)\n",
+      "\n",
+      "filelist.txt        100%[===================>]     794  --.-KB/s    in 0s      \n",
+      "\n",
+      "2022-06-10 14:53:17 (285 MB/s) - ‘/home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_file_list.txt’ saved [794]\n",
+      "\n",
+      "\u001b[38;5;242m0\u001b[0m\n",
+      "\u001b[0m\n",
       "Total number of processed SAMPLES files found is: 8\n",
       "\u001b[32mTotal number of files after filter is: 4 \u001b[0m\n",
       "Total number of processed SERIES files found is: 1\n",
       "\u001b[32mTotal number of files after filter is: 0 \u001b[0m\n",
+      "Expanding metadata list...\n",
+      "Expanding metadata list...\n",
       "\u001b[38;5;242m\n",
-      "--2022-03-10 11:35:36--  ftp://ftp.ncbi.nlm.nih.gov/geo/samples/GSM5621nnn/GSM5621756/suppl/GSM5621756_ChIPseq_Huh7_siNC_H3K27ac_summits.bed.gz\n",
+      "--2022-06-10 14:53:17--  ftp://ftp.ncbi.nlm.nih.gov/geo/samples/GSM5621nnn/GSM5621756/suppl/GSM5621756_ChIPseq_Huh7_siNC_H3K27ac_summits.bed.gz\n",
       "           => ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621756_ChIPseq_Huh7_siNC_H3K27ac_summits.bed.gz’\n",
-      "Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41e:250::12, 2607:f220:41e:250::11, 165.112.9.228, ...\n",
-      "Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41e:250::12|:21... connected.\n",
+      "Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41f:250::230, 2607:f220:41e:250::12, 165.112.9.228, ...\n",
+      "Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41f:250::230|:21... connected.\n",
       "Logging in as anonymous ... Logged in!\n",
       "==> SYST ... done.    ==> PWD ... done.\n",
       "==> TYPE I ... done.  ==> CWD (1) /geo/samples/GSM5621nnn/GSM5621756/suppl ... done.\n",
@@ -365,18 +436,18 @@
       "==> EPSV ... done.    ==> RETR GSM5621756_ChIPseq_Huh7_siNC_H3K27ac_summits.bed.gz ... done.\n",
       "Length: 785486 (767K) (unauthoritative)\n",
       "\n",
-      "GSM5621756_ChIPseq_ 100%[===================>] 767.08K  1.90MB/s    in 0.4s    \n",
+      "GSM5621756_ChIPseq_ 100%[===================>] 767.08K  2.46MB/s    in 0.3s    \n",
       "\n",
-      "2022-03-10 11:35:37 (1.90 MB/s) - ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621756_ChIPseq_Huh7_siNC_H3K27ac_summits.bed.gz’ saved [785486]\n",
+      "2022-06-10 14:53:19 (2.46 MB/s) - ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621756_ChIPseq_Huh7_siNC_H3K27ac_summits.bed.gz’ saved [785486]\n",
       "\n",
       "\u001b[38;5;242m0\u001b[0m\n",
       "\u001b[0m\n",
       "\u001b[92mFile /home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621756_ChIPseq_Huh7_siNC_H3K27ac_summits.bed.gz has been downloaded successfully\u001b[0m\n",
       "\u001b[38;5;242m\n",
-      "--2022-03-10 11:35:38--  ftp://ftp.ncbi.nlm.nih.gov/geo/samples/GSM5621nnn/GSM5621758/suppl/GSM5621758_ChIPseq_Huh7_siDHX37_H3K27ac_summits.bed.gz\n",
+      "--2022-06-10 14:53:20--  ftp://ftp.ncbi.nlm.nih.gov/geo/samples/GSM5621nnn/GSM5621758/suppl/GSM5621758_ChIPseq_Huh7_siDHX37_H3K27ac_summits.bed.gz\n",
       "           => ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621758_ChIPseq_Huh7_siDHX37_H3K27ac_summits.bed.gz’\n",
-      "Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41e:250::11, 2607:f220:41e:250::12, 130.14.250.12, ...\n",
-      "Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41e:250::11|:21... connected.\n",
+      "Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41f:250::230, 2607:f220:41e:250::12, 165.112.9.228, ...\n",
+      "Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41f:250::230|:21... connected.\n",
       "Logging in as anonymous ... Logged in!\n",
       "==> SYST ... done.    ==> PWD ... done.\n",
       "==> TYPE I ... done.  ==> CWD (1) /geo/samples/GSM5621nnn/GSM5621758/suppl ... done.\n",
@@ -384,18 +455,18 @@
       "==> EPSV ... done.    ==> RETR GSM5621758_ChIPseq_Huh7_siDHX37_H3K27ac_summits.bed.gz ... done.\n",
       "Length: 784432 (766K) (unauthoritative)\n",
       "\n",
-      "GSM5621758_ChIPseq_ 100%[===================>] 766.05K  2.48MB/s    in 0.3s    \n",
+      "GSM5621758_ChIPseq_ 100%[===================>] 766.05K  2.99MB/s    in 0.2s    \n",
       "\n",
-      "2022-03-10 11:35:39 (2.48 MB/s) - ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621758_ChIPseq_Huh7_siDHX37_H3K27ac_summits.bed.gz’ saved [784432]\n",
+      "2022-06-10 14:53:20 (2.99 MB/s) - ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621758_ChIPseq_Huh7_siDHX37_H3K27ac_summits.bed.gz’ saved [784432]\n",
       "\n",
       "\u001b[38;5;242m0\u001b[0m\n",
       "\u001b[0m\n",
       "\u001b[92mFile /home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621758_ChIPseq_Huh7_siDHX37_H3K27ac_summits.bed.gz has been downloaded successfully\u001b[0m\n",
       "\u001b[38;5;242m\n",
-      "--2022-03-10 11:35:39--  ftp://ftp.ncbi.nlm.nih.gov/geo/samples/GSM5621nnn/GSM5621760/suppl/GSM5621760_CUTTag_Huh7_DHX37_summits.bed.gz\n",
+      "--2022-06-10 14:53:21--  ftp://ftp.ncbi.nlm.nih.gov/geo/samples/GSM5621nnn/GSM5621760/suppl/GSM5621760_CUTTag_Huh7_DHX37_summits.bed.gz\n",
       "           => ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621760_CUTTag_Huh7_DHX37_summits.bed.gz’\n",
-      "Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41e:250::11, 2607:f220:41e:250::12, 130.14.250.12, ...\n",
-      "Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41e:250::11|:21... connected.\n",
+      "Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41f:250::230, 2607:f220:41e:250::12, 165.112.9.228, ...\n",
+      "Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41f:250::230|:21... connected.\n",
       "Logging in as anonymous ... Logged in!\n",
       "==> SYST ... done.    ==> PWD ... done.\n",
       "==> TYPE I ... done.  ==> CWD (1) /geo/samples/GSM5621nnn/GSM5621760/suppl ... done.\n",
@@ -403,18 +474,18 @@
       "==> EPSV ... done.    ==> RETR GSM5621760_CUTTag_Huh7_DHX37_summits.bed.gz ... done.\n",
       "Length: 163441 (160K) (unauthoritative)\n",
       "\n",
-      "GSM5621760_CUTTag_H 100%[===================>] 159.61K   862KB/s    in 0.2s    \n",
+      "GSM5621760_CUTTag_H 100%[===================>] 159.61K   733KB/s    in 0.2s    \n",
       "\n",
-      "2022-03-10 11:35:40 (862 KB/s) - ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621760_CUTTag_Huh7_DHX37_summits.bed.gz’ saved [163441]\n",
+      "2022-06-10 14:53:22 (733 KB/s) - ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621760_CUTTag_Huh7_DHX37_summits.bed.gz’ saved [163441]\n",
       "\n",
       "\u001b[38;5;242m0\u001b[0m\n",
       "\u001b[0m\n",
       "\u001b[92mFile /home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621760_CUTTag_Huh7_DHX37_summits.bed.gz has been downloaded successfully\u001b[0m\n",
       "\u001b[38;5;242m\n",
-      "--2022-03-10 11:35:41--  ftp://ftp.ncbi.nlm.nih.gov/geo/samples/GSM5621nnn/GSM5621761/suppl/GSM5621761_CUTTag_Huh7_PLRG1_summits.bed.gz\n",
+      "--2022-06-10 14:53:22--  ftp://ftp.ncbi.nlm.nih.gov/geo/samples/GSM5621nnn/GSM5621761/suppl/GSM5621761_CUTTag_Huh7_PLRG1_summits.bed.gz\n",
       "           => ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621761_CUTTag_Huh7_PLRG1_summits.bed.gz’\n",
-      "Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41e:250::11, 2607:f220:41e:250::12, 130.14.250.12, ...\n",
-      "Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41e:250::11|:21... connected.\n",
+      "Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 2607:f220:41f:250::230, 2607:f220:41e:250::12, 165.112.9.228, ...\n",
+      "Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|2607:f220:41f:250::230|:21... connected.\n",
       "Logging in as anonymous ... Logged in!\n",
       "==> SYST ... done.    ==> PWD ... done.\n",
       "==> TYPE I ... done.  ==> CWD (1) /geo/samples/GSM5621nnn/GSM5621761/suppl ... done.\n",
@@ -422,18 +493,17 @@
       "==> EPSV ... done.    ==> RETR GSM5621761_CUTTag_Huh7_PLRG1_summits.bed.gz ... done.\n",
       "Length: 117250 (115K) (unauthoritative)\n",
       "\n",
-      "GSM5621761_CUTTag_H 100%[===================>] 114.50K   644KB/s    in 0.2s    \n",
+      "GSM5621761_CUTTag_H 100%[===================>] 114.50K  --.-KB/s    in 0.1s    \n",
       "\n",
-      "2022-03-10 11:35:41 (644 KB/s) - ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621761_CUTTag_Huh7_PLRG1_summits.bed.gz’ saved [117250]\n",
+      "2022-06-10 14:53:23 (937 KB/s) - ‘/home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621761_CUTTag_Huh7_PLRG1_summits.bed.gz’ saved [117250]\n",
       "\n",
       "\u001b[38;5;242m0\u001b[0m\n",
       "\u001b[0m\n",
       "\u001b[92mFile /home/bnt4me/Virginia/for_docs/geo/GSE185701/GSM5621761_CUTTag_Huh7_PLRG1_summits.bed.gz has been downloaded successfully\u001b[0m\n",
       "Finished processing 1 accession(s)\n",
-      "Expanding metadata list...\n",
       "Unifying and saving of metadata... \n",
-      "\u001b[92mFile /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/bright_test_annotation_sample_processed.csv has been saved successfully\u001b[0m\n",
-      "  Config file: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/bright_test/bright_test_annotation_sample_processed.yaml\n"
+      "\u001b[92mFile /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_samples.csv has been saved successfully\u001b[0m\n",
+      "  Config file: /home/bnt4me/Virginia/repos/geof2/geofetch/docs_jupyter/GSE185701_samples.yaml\n"
      ]
     }
    ],
@@ -451,7 +521,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": 13,
    "metadata": {},
    "outputs": [
     {
diff --git a/docs_jupyter/raw-data-downloading.ipynb b/docs_jupyter/raw-data-downloading.ipynb
index 734faa5..4a595fc 100644
--- a/docs_jupyter/raw-data-downloading.ipynb
+++ b/docs_jupyter/raw-data-downloading.ipynb
@@ -4,7 +4,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "# geofetch tutorial for row data\n",
+    "# geofetch tutorial for raw data\n",
     "\n",
     "The [GSE67303 data set](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE67303) has about 250 mb of data across 4 samples, so it's a quick download for a test case. Let's take a quick peek at the geofetch version:"
    ]
@@ -18,7 +18,7 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "geofetch 0.8.0\n"
+      "geofetch 0.9.0\n"
      ]
     }
    ],
@@ -46,12 +46,16 @@
      "text": [
       "usage: geofetch [-h] [-V] -i INPUT [-n NAME] [-m METADATA_ROOT]\n",
       "                [-u METADATA_FOLDER] [--just-metadata] [-r]\n",
-      "                [--config-template CONFIG_TEMPLATE] [-P PIPELINE_INTERFACES]\n",
-      "                [-k SKIP] [--acc-anno] [--use-key-subset] [-p]\n",
+      "                [--config-template CONFIG_TEMPLATE]\n",
+      "                [--pipeline_samples PIPELINE_SAMPLES]\n",
+      "                [--pipeline_project PIPELINE_PROJECT] [-k SKIP] [--acc-anno]\n",
+      "                [--discard-soft] [--const-limit-project CONST_LIMIT_PROJECT]\n",
+      "                [--const-limit-discard CONST_LIMIT_DISCARD]\n",
+      "                [--attr-limit-truncate ATTR_LIMIT_TRUNCATE] [-p]\n",
       "                [--data-source {all,samples,series}] [--filter FILTER]\n",
       "                [--filter-size FILTER_SIZE] [-g GEO_FOLDER] [-x]\n",
-      "                [-b BAM_FOLDER] [-f FQ_FOLDER] [--silent] [--verbosity V]\n",
-      "                [--logdev]\n",
+      "                [-b BAM_FOLDER] [-f FQ_FOLDER] [--use-key-subset] [--silent]\n",
+      "                [--verbosity V] [--logdev]\n",
       "\n",
       "Automatic GEO and SRA data downloader\n",
       "\n",
@@ -77,16 +81,34 @@
       "                        If set, re-download metadata even if it exists.\n",
       "  --config-template CONFIG_TEMPLATE\n",
       "                        Project config yaml file template.\n",
-      "  -P PIPELINE_INTERFACES, --pipeline_interfaces PIPELINE_INTERFACES\n",
-      "                        Optional: Specify one or more filepaths to pipeline\n",
-      "                        interface yaml files. These will be added to the\n",
-      "                        project config file to make it immediately compatible\n",
-      "                        with looper. [Default: null]\n",
+      "  --pipeline_samples PIPELINE_SAMPLES\n",
+      "                        Optional: Specify one or more filepaths to SAMPLES\n",
+      "                        pipeline interface yaml files. These will be added to\n",
+      "                        the project config file to make it immediately\n",
+      "                        compatible with looper. [Default: null]\n",
+      "  --pipeline_project PIPELINE_PROJECT\n",
+      "                        Optional: Specify one or more filepaths to PROJECT\n",
+      "                        pipeline interface yaml files. These will be added to\n",
+      "                        the project config file to make it immediately\n",
+      "                        compatible with looper. [Default: null]\n",
       "  -k SKIP, --skip SKIP  Skip some accessions. [Default: no skip].\n",
-      "  --acc-anno            Also produce annotation sheets for each accession, not\n",
-      "                        just for the whole project combined\n",
-      "  --use-key-subset      Use just the keys defined in this module when writing\n",
-      "                        out metadata.\n",
+      "  --acc-anno            Optional: Produce annotation sheets for each\n",
+      "                        accession. Project combined PEP for the whole project\n",
+      "                        won't be produced.\n",
+      "  --discard-soft        Optional: After creation of PEP files, all soft and\n",
+      "                        additional files will be deleted\n",
+      "  --const-limit-project CONST_LIMIT_PROJECT\n",
+      "                        Optional: Limit of the number of the constant sample\n",
+      "                        characters that should not be in project yaml.\n",
+      "                        [Default: 50]\n",
+      "  --const-limit-discard CONST_LIMIT_DISCARD\n",
+      "                        Optional: Limit of the number of the constant sample\n",
+      "                        characters that should not be discarded [Default: 250]\n",
+      "  --attr-limit-truncate ATTR_LIMIT_TRUNCATE\n",
+      "                        Optional: Limit of the number of sample characters.Any\n",
+      "                        attribute with more than X characters will truncate to\n",
+      "                        the first X, where X is a number of characters\n",
+      "                        [Default: 500]\n",
       "  --silent              Silence logging. Overrides verbosity.\n",
       "  --verbosity V         Set logging level (1-5 or logging module level name)\n",
       "  --logdev              Expand content of logging message format.\n",
@@ -99,15 +121,16 @@
       "                        attached to the collective series entity, or to\n",
       "                        individual samples. Allowable values are: samples,\n",
       "                        series or both (all). Ignored unless 'processed' flag\n",
-      "                        is set. [Default: all]\n",
+      "                        is set. [Default: samples]\n",
       "  --filter FILTER       Optional: Filter regex for processed filenames\n",
       "                        [Default: None].Ignored unless 'processed' flag is\n",
       "                        set.\n",
       "  --filter-size FILTER_SIZE\n",
       "                        Optional: Filter size for processed files that are\n",
-      "                        stored as sample repository [Default: None]. Supported\n",
-      "                        input formats : 12B, 12KB, 12MB, 12GB. Ignored unless\n",
-      "                        'processed' flag is set.\n",
+      "                        stored as sample repository [Default: None]. Works\n",
+      "                        only for sample data. Supported input formats : 12B,\n",
+      "                        12KB, 12MB, 12GB. Ignored unless 'processed' flag is\n",
+      "                        set.\n",
       "  -g GEO_FOLDER, --geo-folder GEO_FOLDER\n",
       "                        Optional: Specify a location to store processed GEO\n",
       "                        files. Ignored unless 'processed' flag is\n",
@@ -127,7 +150,9 @@
       "  -f FQ_FOLDER, --fq-folder FQ_FOLDER\n",
       "                        Optional: Specify folder of fastq files. Geofetch will\n",
       "                        not download sra files when corresponding fastq files\n",
-      "                        already exist. [Default: $SRAFQ:]\n"
+      "                        already exist. [Default: $SRAFQ:]\n",
+      "  --use-key-subset      Use just the keys defined in this module when writing\n",
+      "                        out metadata.\n"
      ]
     }
    ],
diff --git a/geofetch/_version.py b/geofetch/_version.py
index 777f190..3e2f46a 100644
--- a/geofetch/_version.py
+++ b/geofetch/_version.py
@@ -1,1 +1,1 @@
-__version__ = "0.8.0"
+__version__ = "0.9.0"
diff --git a/geofetch/config_processed_template.yaml b/geofetch/config_processed_template.yaml
index 74af190..ba17184 100644
--- a/geofetch/config_processed_template.yaml
+++ b/geofetch/config_processed_template.yaml
@@ -1,3 +1,5 @@
+# Autogenerated by geofetch
+
 pep_version: 2.1.0
 project_name: {project_name}
 sample_table: {sample_table}
@@ -5,7 +7,11 @@ sample_table: {sample_table}
 sample_modifiers:
   append:
     output_file_path: FILES
+    {additional_columns}
+    {pipeline_samples}
   derive:
     attributes: [output_file_path]
     sources:
       FILES: {geo_folder}/{GSE}/{file}
+
+{pipeline_project}
diff --git a/geofetch/config_template.yaml b/geofetch/config_template.yaml
index bf7b5ba..b492ae6 100644
--- a/geofetch/config_template.yaml
+++ b/geofetch/config_template.yaml
@@ -11,8 +11,9 @@ looper:
 
 sample_modifiers:
   append:
+    {additional_columns}
     SRR_files: SRA
-    pipeline_interfaces: {pipeline_interfaces}
+    {pipeline_samples}
   derive:
     attributes: [read1, read2, SRR_files]
     sources:
@@ -55,3 +56,5 @@ project_modifiers:
             FQ: "${SRAFQ}/{SRR}.fastq.gz"
             FQ1: "${SRAFQ}/{SRR}_1.fastq.gz"
             FQ2: "${SRAFQ}/{SRR}_2.fastq.gz"
+
+{pipeline_project}
diff --git a/geofetch/geofetch.py b/geofetch/geofetch.py
index 897a68a..b3a5501 100755
--- a/geofetch/geofetch.py
+++ b/geofetch/geofetch.py
@@ -1,19 +1,7 @@
 #!/usr/bin/env python3
 
-__author__ = ["Vince Reuter", "Nathan Sheffield", "Oleksandr Khoroshevskyi"]
+__author__ = ["Oleksandr Khoroshevskyi", "Vince Reuter", "Nathan Sheffield"]
 
-# Outline:
-# INPUT: A list of GSE ids, optionally including GSM ids to limit to.
-# example: GSE61150
-# 1. Grab SOFT file from
-# http://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?targ=gsm&acc=GSE61150&form=text&view=full
-# 2. parse it, produce a table with all needed fields.
-# 3. Grab SRA values from field, use this link to grab SRA metadata:
-# http://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?save=efetch&db=sra&rettype=runinfo&term=SRX079566
-# http://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?save=efetch&db=sra&rettype=runinfo&term=SRP055171
-# http://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?save=efetch&db=sra&rettype=runinfo&term=SRX883589
-
-# 4. Parse the SRA RunInfo csv file and use the download_link field to grab the .sra file
 
 import argparse
 import copy
@@ -22,11 +10,18 @@
 import re
 import subprocess
 import sys
+from string import punctuation
 
 # import tarfile
 import time
 
-from .utils import Accession, parse_accessions, parse_SOFT_line, convert_size
+from .utils import (
+    Accession,
+    parse_accessions,
+    parse_SOFT_line,
+    convert_size,
+    clean_soft_files,
+)
 from ._version import __version__
 
 from logmuse import add_logging_options, init_logger
@@ -79,12 +74,13 @@ def __init__(
         just_metadata=False,
         refresh_metadata=False,
         config_template=None,
-        pipeline_interfaces=None,
+        pipeline_samples=None,
+        pipeline_project=None,
         skip=0,
         acc_anno=False,
         use_key_subset=False,
         processed=True,
-        supp_by="samples",
+        data_source="samples",
         filter=None,
         filter_size=None,
         geo_folder=".",
@@ -98,6 +94,12 @@ def __init__(
         verbosity=None,
         logdev=False,
         input=None,
+        const_limit_project=50,
+        const_limit_discard=250,
+        attr_limit_truncate=500,
+        discard_soft=False,
+        add_dotfile=False,
+        **kwargs,
     ):
 
         # self.args = args
@@ -109,7 +111,10 @@ def __init__(
         if name:
             self.project_name = name
         else:
-            self.project_name = os.path.splitext(os.path.basename(input))[0]
+            try:
+                self.project_name = os.path.splitext(os.path.basename(input))[0]
+            except TypeError:
+                self.project_name = "project_name"
 
         if metadata_folder:
             self.metadata_expanded = expandpath(metadata_folder)
@@ -131,18 +136,28 @@ def __init__(
         self.refresh_metadata = refresh_metadata
         self.config_template = config_template
 
+        # if user specified a pipeline interface path for samples, add it into the project config
+        if pipeline_samples and pipeline_samples != "null":
+            self.file_pipeline_samples = pipeline_samples
+            self.file_pipeline_samples = (
+                f"pipeline_interfaces: {self.file_pipeline_samples}"
+            )
+        else:
+            self.file_pipeline_samples = ""
+
         # if user specified a pipeline interface path, add it into the project config
-        self.pipeline_interfaces = pipeline_interfaces
-        if self.pipeline_interfaces:
-            self.file_pipeline_interfaces = self.pipeline_interfaces
+        if pipeline_project:
+            self.file_pipeline_project = (
+                f"looper:\n    pipeline_interfaces: {pipeline_project}"
+            )
         else:
-            self.file_pipeline_interfaces = "null"
+            self.file_pipeline_project = ""
 
         self.skip = skip
         self.acc_anno = acc_anno
         self.use_key_subset = use_key_subset
         self.processed = processed
-        self.supp_by = supp_by
+        self.supp_by = data_source
 
         if filter:
             self.filter_re = re.compile(filter.lower())
@@ -156,7 +171,11 @@ def __init__(
             self.metadata_raw = os.path.join(self.metadata_raw, self.project_name)
 
         if filter_size is not None:
-            self.filter_size = convert_size(filter_size.lower())
+            try:
+                self.filter_size = convert_size(filter_size.lower())
+            except ValueError as message:
+                self._LOGGER.error(message)
+                raise SystemExit()
         else:
             self.filter_size = filter_size
 
@@ -171,13 +190,21 @@ def __init__(
         self.verbosity = verbosity
         self.logdev = logdev
 
+        self.const_limit_project = const_limit_project
+        self.const_limit_discard = const_limit_discard
+        self.attr_limit_truncate = attr_limit_truncate
+
+        self.discard_soft = discard_soft
+        self.add_dotfile = add_dotfile
+
         self._LOGGER.info(f"Metadata folder: {self.metadata_expanded}")
 
         # check to make sure prefetch is callable
         if not just_metadata and not processed:
             if not is_command_callable("prefetch"):
                 raise SystemExit(
-                    "You must first install the sratoolkit, with prefetch in your PATH."
+                    "To download raw data You must first install the sratoolkit, with prefetch in your PATH."
+                    " Installation instruction: http://geofetch.databio.org/en/latest/install/"
                 )
 
         # Some sanity checks before proceeding
@@ -218,7 +245,7 @@ def fetch_all(self, input, name=None):
             elif ncount == self.skip + 1:
                 self._LOGGER.info(f"Skipped {self.skip} accessions. Starting now.")
             self._LOGGER.info(
-                f"\033[38;5;228mProcessing accession {ncount} of {nkeys}: '{acc_GSE}'\033[0m"
+                f"\033[38;5;200mProcessing accession {ncount} of {nkeys}: '{acc_GSE}'\033[0m"
             )
 
             if len(re.findall(GSE_PATTERN, acc_GSE)) != 1:
@@ -294,17 +321,26 @@ def fetch_all(self, input, name=None):
                             meta_processed_series, key_in_list
                         )
 
-                    # adding metadata from current experiment to the project
-                    processed_metadata_samples.extend(meta_processed_samples)
-                    processed_metadata_exp.extend(meta_processed_series)
+                    # convert column names to lowercase and underscore
+                    meta_processed_samples = self.standardize_colnames(
+                        meta_processed_samples
+                    )
+                    meta_processed_series = self.standardize_colnames(
+                        meta_processed_series
+                    )
+
+                    if not self.acc_anno:
+                        # adding metadata from current experiment to the project
+                        processed_metadata_samples.extend(meta_processed_samples)
+                        processed_metadata_exp.extend(meta_processed_series)
 
-                    # save PEP for each accession
+                    # save PEP for each accession if acc-anno flag is true
                     if self.acc_anno and len(acc_GSE_list.keys()) > 1:
                         if self.supp_by == "all":
                             # samples
                             pep_acc_path_sample = os.path.join(
                                 self.metadata_raw,
-                                acc_GSE,
+                                f"{acc_GSE}_samples",
                                 acc_GSE + SAMPLE_SUPP_METADATA_FILE,
                             )
                             self.write_processed_annotation(
@@ -314,7 +350,7 @@ def fetch_all(self, input, name=None):
                             # series
                             pep_acc_path_exp = os.path.join(
                                 self.metadata_raw,
-                                acc_GSE,
+                                f"{acc_GSE}_series",
                                 acc_GSE + EXP_SUPP_METADATA_FILE,
                             )
                             self.write_processed_annotation(
@@ -323,7 +359,7 @@ def fetch_all(self, input, name=None):
                         elif self.supp_by == "samples":
                             pep_acc_path_sample = os.path.join(
                                 self.metadata_raw,
-                                acc_GSE,
+                                f"{acc_GSE}_samples",
                                 acc_GSE + SAMPLE_SUPP_METADATA_FILE,
                             )
                             self.write_processed_annotation(
@@ -332,7 +368,7 @@ def fetch_all(self, input, name=None):
                         elif self.supp_by == "series":
                             pep_acc_path_exp = os.path.join(
                                 self.metadata_raw,
-                                acc_GSE,
+                                f"{acc_GSE}_series",
                                 acc_GSE + EXP_SUPP_METADATA_FILE,
                             )
                             self.write_processed_annotation(
@@ -385,6 +421,8 @@ def fetch_all(self, input, name=None):
                 # download gsm metadata
                 result = self.get_SRA_meta(file_gse, file_sra, gsm_metadata)
                 if not result:
+                    # delete current acc if no raw data was found
+                    # del metadata_dict[acc_GSE]
                     continue
                 # Parse metadata from SRA
                 # Produce an annotated output from the GSM and SRARunInfo files.
@@ -422,7 +460,7 @@ def fetch_all(self, input, name=None):
                             gsm_metadata[experiment]["gsm_id"]
                         ]
                     except KeyError:
-                        _LOGGER.warning(f"Error in sample_name ...")
+                        self._LOGGER.info(f"KeyError in sample_name, creating new...")
                     if not sample_name or sample_name == "":
                         temp = gsm_metadata[experiment]["Sample_title"]
                         # Now do a series of transformations to cleanse the sample name
@@ -497,7 +535,7 @@ def fetch_all(self, input, name=None):
 
                     # Write to filtered SRA Runinfo file
                     wwrite.writerow(line)
-                    self._LOGGER.info(f"Get SRR: {run_name} ({experiment})")
+                    self._LOGGER.info(f"Getting SRR: {run_name} ({experiment})")
                     bam_file = (
                         ""
                         if self.bam_folder == ""
@@ -558,6 +596,11 @@ def fetch_all(self, input, name=None):
         # Logging additional information about processing
         self._LOGGER.info(f"Finished processing {len(acc_GSE_list)} accession(s)")
 
+        # Logging cleaning process:
+        if self.discard_soft:
+            self._LOGGER.info(f"Cleaning soft files ...")
+            clean_soft_files(self.metadata_raw)
+
         if len(failed_runs) > 0:
             self._LOGGER.warn(
                 f"The following samples could not be downloaded: {failed_runs}"
@@ -567,109 +610,49 @@ def fetch_all(self, input, name=None):
 
         # saving PEPs for processed data
         if self.processed:
-            if self.supp_by == "all":
-                supp_sample_path_meta = os.path.join(
-                    self.metadata_raw, self.project_name + SAMPLE_SUPP_METADATA_FILE
-                )
-                self.write_processed_annotation(
-                    processed_metadata_samples, supp_sample_path_meta
-                )
-
-                supp_series_path_meta = os.path.join(
-                    self.metadata_raw, self.project_name + EXP_SUPP_METADATA_FILE
-                )
-                self.write_processed_annotation(
-                    processed_metadata_exp, supp_series_path_meta
-                )
-
-            elif self.supp_by == "samples":
-                supp_sample_path_meta = os.path.join(
-                    self.metadata_raw, self.project_name + SAMPLE_SUPP_METADATA_FILE
-                )
-                self.write_processed_annotation(
-                    processed_metadata_samples, supp_sample_path_meta
-                )
-
-            elif self.supp_by == "series":
-                supp_series_path_meta = os.path.join(
-                    self.metadata_raw, self.project_name + EXP_SUPP_METADATA_FILE
-                )
-                self.write_processed_annotation(
-                    processed_metadata_exp, supp_series_path_meta
-                )
-        else:
-            # Combine individual accessions into project-level annotations, and write
-            # individual accession files (if requested)
-            metadata_dict_combined = {}
-            for acc_GSE, gsm_metadata in metadata_dict.items():
-                file_annotation = os.path.join(
-                    self.metadata_expanded, acc_GSE + "_annotation.csv"
-                )
-                if self.acc_anno:
-                    self.write_gsm_annotation(
-                        gsm_metadata,
-                        file_annotation,
-                        use_key_subset=self.use_key_subset,
+            if not self.acc_anno:
+                if self.supp_by == "all":
+                    supp_sample_path_meta = os.path.join(
+                        self.metadata_raw,
+                        "PEP_samples",
+                        self.project_name + SAMPLE_SUPP_METADATA_FILE,
+                    )
+                    self.write_processed_annotation(
+                        processed_metadata_samples, supp_sample_path_meta
                     )
-                metadata_dict_combined.update(gsm_metadata)
-
-            subannotation_dict_combined = {}
-            for acc_GSE, gsm_multi_table in subannotation_dict.items():
-                file_subannotation = os.path.join(
-                    self.metadata_expanded, acc_GSE + "_subannotation.csv"
-                )
-                if self.acc_anno:
-                    self.write_subannotation(gsm_multi_table, file_subannotation)
-                subannotation_dict_combined.update(gsm_multi_table)
-            self._LOGGER.info(
-                "Creating complete project annotation sheets and config file..."
-            )
-            # If the project included more than one GSE, we can now output combined
-            # annotation tables for the entire project.
-
-            # Write combined annotation sheet
-            file_annotation = os.path.join(
-                self.metadata_raw, self.project_name + "_annotation.csv"
-            )
-            self.write_gsm_annotation(
-                metadata_dict_combined,
-                file_annotation,
-                use_key_subset=self.use_key_subset,
-            )
-
-            # Write combined subannotation table
-            if len(subannotation_dict_combined) > 0:
-                file_subannotation = os.path.join(
-                    self.metadata_raw, self.project_name + "_subannotation.csv"
-                )
-                self.write_subannotation(
-                    subannotation_dict_combined, file_subannotation
-                )
-            else:
-                file_subannotation = "null"
-
-            # Write project config file
-            if not self.config_template:
-                geofetchdir = os.path.dirname(__file__)
-                self.config_template = os.path.join(geofetchdir, "config_template.yaml")
 
-            with open(self.config_template, "r") as template_file:
-                template = template_file.read()
+                    supp_series_path_meta = os.path.join(
+                        self.metadata_raw,
+                        "PEP_series",
+                        self.project_name + EXP_SUPP_METADATA_FILE,
+                    )
+                    self.write_processed_annotation(
+                        processed_metadata_exp, supp_series_path_meta
+                    )
 
-            template_values = {
-                "project_name": self.project_name,
-                "annotation": os.path.basename(file_annotation),
-                "subannotation": os.path.basename(file_subannotation),
-                "pipeline_interfaces": self.file_pipeline_interfaces,
-            }
+                elif self.supp_by == "samples":
+                    supp_sample_path_meta = os.path.join(
+                        self.metadata_raw,
+                        "PEP_samples",
+                        self.project_name + SAMPLE_SUPP_METADATA_FILE,
+                    )
+                    self.write_processed_annotation(
+                        processed_metadata_samples, supp_sample_path_meta
+                    )
 
-            for k, v in template_values.items():
-                placeholder = "{" + str(k) + "}"
-                template = template.replace(placeholder, str(v))
+                elif self.supp_by == "series":
+                    supp_series_path_meta = os.path.join(
+                        self.metadata_raw,
+                        "PEP_series",
+                        self.project_name + EXP_SUPP_METADATA_FILE,
+                    )
+                    self.write_processed_annotation(
+                        processed_metadata_exp, supp_series_path_meta
+                    )
 
-            # save .yaml file
-            config = os.path.join(self.metadata_raw, self.project_name + "_config.yaml")
-            self._write(config, template, msg_pre="  Config file: ")
+        # saving PEPs for raw data
+        else:
+            self.write_raw_annotation(metadata_dict, subannotation_dict)
 
     def expand_metadata_list(self, metadata_list, dict_key):
         """
@@ -696,12 +679,28 @@ def expand_metadata_list(self, metadata_list, dict_key):
                     for elem in metadata_list[n_elem][dict_key]:
                         separated_elements = elem.split(": ")
                         if len(separated_elements) >= 2:
-                            list_of_elem = [
-                                separated_elements[0],
-                                ": ".join(separated_elements[1:]),
-                            ]
-                            sample_char = dict([list_of_elem])
-                            metadata_list[n_elem].update(sample_char)
+
+                            # if first element is larger than 40 then treat it like simple string
+                            if len(separated_elements[0]) > 40:
+                                just_string = True
+                                if this_string != "":
+                                    this_string = ", ".join([this_string, elem])
+                                else:
+                                    this_string = elem
+                            # additional elem for all bed files
+                            elif len(separated_elements[0].split("(")) > 1:
+                                just_string = True
+                                if this_string != "":
+                                    this_string = "(".join([this_string, elem])
+                                else:
+                                    this_string = elem
+                            else:
+                                list_of_elem = [
+                                    separated_elements[0],
+                                    ": ".join(separated_elements[1:]),
+                                ]
+                                sample_char = dict([list_of_elem])
+                                metadata_list[n_elem].update(sample_char)
                         else:
                             just_string = True
                             if this_string != "":
@@ -782,6 +781,24 @@ def unify_list_keys(self, processed_meta_list):
                     processed_meta_list[list_elem][k] = ""
         return processed_meta_list
 
+    def find_genome(self, metadata_list):
+        """
+        Create new genome table by joining few columns
+        """
+        list_keys = self.get_list_of_keys(metadata_list)
+        genome_keys = [
+            "assembly",
+            "genome_build",
+        ]
+        proj_gen_keys = list(set(list_keys).intersection(genome_keys))
+
+        for sample in enumerate(metadata_list):
+            sample_genome = ""
+            for key in proj_gen_keys:
+                sample_genome = " ".join([sample_genome, sample[1][key]])
+            metadata_list[sample[0]]["sample_genome"] = sample_genome
+        return metadata_list
+
     def write_gsm_annotation(self, gsm_metadata, file_annotation, use_key_subset=False):
         """
         Write metadata sheet out as an annotation file.
@@ -823,7 +840,7 @@ def write_processed_annotation(self, processed_metadata, file_annotation_path):
             )
             return False
 
-        # create folder if does not exist
+        # create folder if it does not exist
         pep_file_folder = os.path.split(file_annotation_path)[0]
         if not os.path.exists(pep_file_folder):
             os.makedirs(pep_file_folder)
@@ -831,6 +848,21 @@ def write_processed_annotation(self, processed_metadata, file_annotation_path):
         self._LOGGER.info("Unifying and saving of metadata... ")
         processed_metadata = self.unify_list_keys(processed_metadata)
 
+        # delete rare keys
+        processed_metadata = self.find_genome(processed_metadata)
+
+        # filtering huge annotation strings that are repeating for each sample
+        processed_metadata, proj_meta = self.separate_common_meta(
+            processed_metadata,
+            self.const_limit_project,
+            self.const_limit_discard,
+            self.attr_limit_truncate,
+        )
+        meta_list_str = [
+            f"{list(i.keys())[0]}: {list(i.values())[0]}" for i in proj_meta
+        ]
+        modifiers_str = "\n    ".join(d for d in meta_list_str)
+
         with open(file_annotation_path, "w") as m_file:
             dict_writer = csv.DictWriter(m_file, processed_metadata[0].keys())
             dict_writer.writeheader()
@@ -849,7 +881,9 @@ def write_processed_annotation(self, processed_metadata, file_annotation_path):
             "project_name": self.project_name,
             "sample_table": os.path.basename(file_annotation_path),
             "geo_folder": self.geo_folder,
-            "pipeline_interfaces": self.file_pipeline_interfaces,
+            "pipeline_samples": self.file_pipeline_samples,
+            "pipeline_project": self.file_pipeline_project,
+            "additional_columns": modifiers_str,
         }
 
         for k, v in template_values.items():
@@ -860,8 +894,263 @@ def write_processed_annotation(self, processed_metadata, file_annotation_path):
         yaml_name = os.path.split(file_annotation_path)[1][:-4] + ".yaml"
         config = os.path.join(pep_file_folder, yaml_name)
         self._write(config, template, msg_pre="  Config file: ")
+
+        # save .pep.yaml file
+        if self.add_dotfile:
+            dot_yaml_path = os.path.join(pep_file_folder, ".pep.yaml")
+            self.create_dot_yaml(dot_yaml_path, yaml_name)
+
         return True
 
+    @staticmethod
+    def sanitize_name(name_str: str):
+        """
+        Function that sanitizing strings. (Replace all odd characters)
+        :param str names_str: Any string value that has to be sanitized.
+        :return: sanitized strings
+        """
+        new_str = name_str
+        for odd_char in list(punctuation):
+            new_str = new_str.replace(odd_char, "_")
+        new_str = new_str.replace(" ", "_").replace("__", "_")
+        return new_str
+
+    def write_raw_annotation(self, metadata_dict, subannotation_dict):
+        """
+        Combining individual accessions into project-level annotations, and writeing
+        individual accession files (if requested)
+        :param dict metadata_dict: dictionary of metadata
+        :param dict subannotation_dict: dictionary of sub-annotation metadata
+        """
+
+        if self.discard_soft:
+            clean_soft_files(os.path.join(self.metadata_raw))
+
+        try:
+            assert len(metadata_dict) > 0
+        except AssertionError:
+            self._LOGGER.warning(
+                "\033[33mNo PEP created, as no raw data was found!!!\033[0m"
+            )
+            return False
+
+        # checking sample_name value if it's not empty,
+        # otherwise pulling from title
+        for key, value in metadata_dict.items():
+            fixed_dict = {}
+            for key_sample, value_sample in value.items():
+                fixed_dict[key_sample] = value_sample
+                if (
+                    value_sample["sample_name"] == ""
+                    or value_sample["sample_name"] is None
+                ):
+                    fixed_dict[key_sample]["sample_name"] = value_sample["Sample_title"]
+
+                # sanitize sample names
+                fixed_dict[key_sample]["sample_name"] = self.sanitize_name(
+                    fixed_dict[key_sample]["sample_name"]
+                )
+
+            metadata_dict[key] = fixed_dict
+
+        metadata_dict_combined = {}
+        for acc_GSE, gsm_metadata in metadata_dict.items():
+            file_annotation = os.path.join(
+                self.metadata_expanded, acc_GSE + "_annotation.csv"
+            )
+            if self.acc_anno:
+                self.write_gsm_annotation(
+                    gsm_metadata,
+                    file_annotation,
+                    use_key_subset=self.use_key_subset,
+                )
+            metadata_dict_combined.update(gsm_metadata)
+
+        # filtering huge annotation strings that are repeating for each sample
+        metadata_dict_combined, proj_meta = self.separate_common_meta(
+            metadata_dict_combined,
+            self.const_limit_project,
+            self.const_limit_discard,
+            self.attr_limit_truncate,
+        )
+        meta_list_str = [
+            f"{list(i.keys())[0]}: {list(i.values())[0]}" for i in proj_meta
+        ]
+        modifiers_str = "\n    ".join(d for d in meta_list_str)
+
+        subannotation_dict_combined = {}
+        for acc_GSE, gsm_multi_table in subannotation_dict.items():
+            file_subannotation = os.path.join(
+                self.metadata_expanded, acc_GSE + "_subannotation.csv"
+            )
+            if self.acc_anno:
+                self.write_subannotation(gsm_multi_table, file_subannotation)
+            subannotation_dict_combined.update(gsm_multi_table)
+        self._LOGGER.info(
+            "Creating complete project annotation sheets and config file..."
+        )
+        # If the project included more than one GSE, we can now output combined
+        # annotation tables for the entire project.
+        # Write combined annotation sheet
+        file_annotation = os.path.join(
+            self.metadata_raw, self.project_name + "_annotation.csv"
+        )
+        self.write_gsm_annotation(
+            metadata_dict_combined,
+            file_annotation,
+            use_key_subset=self.use_key_subset,
+        )
+        # Write combined subannotation table
+        if len(subannotation_dict_combined) > 0:
+            file_subannotation = os.path.join(
+                self.metadata_raw, self.project_name + "_subannotation.csv"
+            )
+            self.write_subannotation(subannotation_dict_combined, file_subannotation)
+        else:
+            file_subannotation = "null"
+        # Write project config file
+        if not self.config_template:
+            geofetchdir = os.path.dirname(__file__)
+            self.config_template = os.path.join(geofetchdir, "config_template.yaml")
+        with open(self.config_template, "r") as template_file:
+            template = template_file.read()
+
+        template_values = {
+            "project_name": self.project_name,
+            "annotation": os.path.basename(file_annotation),
+            "subannotation": os.path.basename(file_subannotation),
+            "pipeline_samples": self.file_pipeline_samples,
+            "pipeline_project": self.file_pipeline_project,
+            "additional_columns": modifiers_str,
+        }
+        for k, v in template_values.items():
+            placeholder = "{" + str(k) + "}"
+            template = template.replace(placeholder, str(v))
+        # save .yaml file
+        yaml_name = self.project_name + "_config.yaml"
+        config = os.path.join(self.metadata_raw, yaml_name)
+        self._write(config, template, msg_pre="  Config file: ")
+
+        # save .pep.yaml file
+        if self.add_dotfile:
+            dot_yaml_path = os.path.join(self.metadata_raw, ".pep.yaml")
+            self.create_dot_yaml(dot_yaml_path, yaml_name)
+
+    @staticmethod
+    def create_dot_yaml(file_path: str, yaml_path: str):
+        """
+        Function that creates .pep.yaml file that points to actual yaml file
+        :param str file_path: Path to the .pep.yaml file that we want to create
+        :param str yaml_path: path or name of the actual yaml file
+        """
+        with open(file_path, "w+") as file:
+            file.writelines(f"config_file: {yaml_path}")
+
+    def separate_common_meta(
+        self, meta_list, max_len=50, del_limit=250, attr_limit_truncate=500
+    ):
+        """
+        This function is separating information for the experiment from a sample
+        :param list or dict meta_list: list of dictionaries of samples
+        :param int max_len: threshold of the length of the common value that can be stored in the sample table
+        :param int del_limit: threshold of the length of the common value that have to be deleted
+        :param int attr_limit_truncate: max length of the attribute in the sample csv
+        :return set: Return is a set of list, where 1 list (or dict) is
+        list of samples metadata dictionaries and 2: list of common samples metadata
+        dictionaries that are linked to the project.
+        """
+        input_is_dict = False
+        if isinstance(meta_list, dict):
+            input_is_dict = True
+            new_meta_list = []
+            for key in meta_list:
+                new_dict = meta_list[key]
+                new_dict["big_key"] = key
+                new_meta_list.append(new_dict)
+
+            meta_list = new_meta_list
+
+        list_of_keys = self.get_list_of_keys(meta_list)
+        list_keys_diff = []
+        # finding columns with common values
+        for this_key in list_of_keys:
+            value = ""
+            for nb_sample in enumerate(meta_list):
+                try:
+                    if nb_sample[0] == 0:
+                        value = meta_list[nb_sample[0]][this_key]
+                        if len(str(value)) < max_len and len(str(value)) < del_limit:
+                            list_keys_diff.append(this_key)
+                            break
+                    else:
+                        if value != meta_list[nb_sample[0]][this_key]:
+                            list_keys_diff.append(this_key)
+                            break
+                except KeyError:
+                    pass
+
+        list_keys_diff = set(list_keys_diff)
+
+        # separating sample and common metadata and creating 2 lists
+        new_meta_project = []
+        for this_key in list_of_keys:
+            first_key = True
+            for nb_sample in enumerate(meta_list):
+                try:
+                    if this_key not in list_keys_diff:
+                        if first_key:
+                            if len(str(nb_sample[1][this_key])) <= del_limit:
+                                new_meta_project.append(
+                                    {this_key: nb_sample[1][this_key]}
+                                )
+                            first_key = False
+                        del meta_list[nb_sample[0]][this_key]
+                except KeyError:
+                    pass
+
+        # Truncate huge information in metadata
+        new_list = []
+        for this_item in meta_list:
+            new_item_list = {}
+            for key, value in this_item.items():
+                if len(str(value)) < attr_limit_truncate:
+                    new_item_list[key] = value
+                else:
+                    new_item_list[key] = value[0:attr_limit_truncate] + " ..."
+            new_list.append(new_item_list)
+
+        meta_list = new_list
+
+        if input_is_dict:
+            new_sample_dict = {}
+            for sample in meta_list:
+                new_sample_dict[sample["big_key"]] = sample
+            meta_list = new_sample_dict
+
+        return meta_list, new_meta_project
+
+    def standardize_colnames(self, meta_list):
+        """
+        Standardize column names by lower-casing and underscore
+        :param list meta_list: list of dictionaries of samples
+        :return : list of dictionaries of samples with standard colnames
+        """
+        new_metalist = []
+        list_keys = self.get_list_of_keys(meta_list)
+        for item_nb, values in enumerate(meta_list):
+            new_metalist.append({})
+            for key in list_keys:
+                try:
+                    new_key_name = key.lower().strip()
+                    new_key_name = self.sanitize_name(new_key_name)
+
+                    new_metalist[item_nb][new_key_name] = values[key]
+
+                except KeyError:
+                    pass
+
+        return new_metalist
+
     def download_SRA_file(self, run_name):
         """
         Downloading SRA file by ising 'prefetch' utility from the SRA Toolkit
@@ -1259,8 +1548,7 @@ def separate_list_of_files(meta_list, col_name="files"):
 
         return separated_list
 
-    @staticmethod
-    def separate_file_url(meta_list):
+    def separate_file_url(self, meta_list):
         """
         This method is adding dict key without file_name without path
         """
@@ -1272,8 +1560,14 @@ def separate_file_url(meta_list):
             # new_dict["sample_name"] = os.path.basename(meta_elem["file"])
             try:
                 new_dict["sample_name"] = str(meta_elem["Sample_title"])
+                if new_dict["sample_name"] == "" or new_dict["sample_name"] is None:
+                    raise KeyError("sample_name Does not exist. Creating .. ")
             except KeyError:
                 new_dict["sample_name"] = os.path.basename(meta_elem["file"])
+
+            # sanitize sample names
+            new_dict["sample_name"] = self.sanitize_name(new_dict["sample_name"])
+
             separated_list.append(new_dict)
         return separated_list
 
@@ -1299,7 +1593,7 @@ def run_size_filter(self, meta_list, col_name="file_size"):
         if self.filter_size is not None:
             filtered_list = []
             for meta_elem in meta_list:
-                if meta_elem[col_name] <= self.filter_size:
+                if int(meta_elem[col_name]) <= self.filter_size:
                     filtered_list.append(meta_elem)
         else:
             self._LOGGER.info(
@@ -1590,15 +1884,25 @@ def _parse_cmdl(cmdl):
         "--config-template", default=None, help="Project config yaml file template."
     )
 
+    # Optional
     parser.add_argument(
-        "-P",
-        "--pipeline_interfaces",
+        "--pipeline-samples",
         default=None,
-        help="Optional: Specify one or more filepaths to pipeline interface yaml files. "
+        help="Optional: Specify one or more filepaths to SAMPLES pipeline interface yaml files. "
         "These will be added to the project config file to make it immediately "
         "compatible with looper. [Default: null]",
     )
 
+    # Optional
+    parser.add_argument(
+        "--pipeline-project",
+        default=None,
+        help="Optional: Specify one or more filepaths to PROJECT pipeline interface yaml files. "
+        "These will be added to the project config file to make it immediately "
+        "compatible with looper. [Default: null]",
+    )
+
+    # Optional
     parser.add_argument(
         "-k",
         "--skip",
@@ -1610,14 +1914,45 @@ def _parse_cmdl(cmdl):
     parser.add_argument(
         "--acc-anno",
         action="store_true",
-        help="Also produce annotation sheets for each accession, not just"
-        " for the whole project combined",
+        help="Optional: Produce annotation sheets for each accession."
+        " Project combined PEP for the whole project won't be produced.",
     )
 
     parser.add_argument(
-        "--use-key-subset",
+        "--discard-soft",
         action="store_true",
-        help="Use just the keys defined in this module when writing out metadata.",
+        help="Optional: After creation of PEP files, all soft and additional files will be deleted",
+    )
+
+    parser.add_argument(
+        "--const-limit-project",
+        type=int,
+        default=50,
+        help="Optional: Limit of the number of the constant sample characters "
+        "that should not be in project yaml. [Default: 50]",
+    )
+
+    parser.add_argument(
+        "--const-limit-discard",
+        type=int,
+        default=250,
+        help="Optional: Limit of the number of the constant sample characters "
+        "that should not be discarded [Default: 250]",
+    )
+
+    parser.add_argument(
+        "--attr-limit-truncate",
+        type=int,
+        default=500,
+        help="Optional: Limit of the number of sample characters."
+        "Any attribute with more than X characters will truncate to the first X,"
+        " where X is a number of characters [Default: 500]",
+    )
+
+    parser.add_argument(
+        "--add-dotfile",
+        action="store_true",
+        help="Optional: Add .pep.yaml file that points .yaml PEP file",
     )
 
     processed_group.add_argument(
@@ -1630,14 +1965,14 @@ def _parse_cmdl(cmdl):
 
     processed_group.add_argument(
         "--data-source",
-        dest="supp_by",
+        dest="data_source",
         choices=["all", "samples", "series"],
         default="samples",
         help="Optional: Specifies the source of data on the GEO record"
         " to retrieve processed data, which may be attached to the"
         " collective series entity, or to individual samples. "
         "Allowable values are: samples, series or both (all). "
-        "Ignored unless 'processed' flag is set. [Default: all]",
+        "Ignored unless 'processed' flag is set. [Default: samples]",
     )
 
     processed_group.add_argument(
@@ -1653,6 +1988,7 @@ def _parse_cmdl(cmdl):
         default=None,
         help="""Optional: Filter size for processed files
                 that are stored as sample repository [Default: None].
+                Works only for sample data.
                 Supported input formats : 12B, 12KB, 12MB, 12GB. 
                 Ignored unless 'processed' flag is set.""",
     )
@@ -1728,6 +2064,12 @@ def _parse_cmdl(cmdl):
         help=argparse.SUPPRESS,
     )
 
+    raw_group.add_argument(
+        "--use-key-subset",
+        action="store_true",
+        help="Use just the keys defined in this module when writing out metadata.",
+    )
+
     parser = add_logging_options(parser)
     return parser.parse_args(cmdl)
 
@@ -1753,9 +2095,7 @@ def main():
     """Run the script."""
     args = _parse_cmdl(sys.argv[1:])
     args_dict = vars(args)
-    print(args_dict)
     Geofetcher(**args_dict).fetch_all(args_dict["input"])
-    # print(ab.get_list_of_processed_files())
 
 
 if __name__ == "__main__":
diff --git a/geofetch/utils.py b/geofetch/utils.py
index f34ae9a..55997ff 100644
--- a/geofetch/utils.py
+++ b/geofetch/utils.py
@@ -279,6 +279,11 @@ def split_accn(accn):
 
 
 def convert_size(size_str: str) -> int:
+    """
+    Converting size, that was provided as string with suffix
+    :param str size_str: size as string with suffix: gb, mb, kb or b
+    :return int: size as int value in bytes
+    """
     abbreviation_dict = {"gb": 1073741824, "mb": 1048576, "kb": 1024, "b": 1}
     supported_formats = r"(\dgb|\dmb|\db|\dkb)$"
     reg_number = r"^\d+"
@@ -292,4 +297,27 @@ def convert_size(size_str: str) -> int:
         abb = abbreviation[0][1:]
         size_in_bytes = int(size_numb[0]) * abbreviation_dict[abb]
 
+    if not isinstance(size_in_bytes, int):
+        raise ValueError(
+            f"Incorrect type of file size was provided! You provided:'{size_str}'"
+        )
+
     return size_in_bytes
+
+
+def clean_soft_files(meta_dir: str):
+    """
+    Cleaning, deleting all soft files after downloading files
+    and creating PEPs
+    :param str meta_dir: Path to the metadata files
+    """
+    dir_files = os.listdir(meta_dir)
+
+    for item in dir_files:
+        if (
+            item.endswith(".soft")
+            or item.endswith("_file_list.txt")
+            or item.endswith("SRA.csv")
+            or item.endswith("SRA_filt.csv")
+        ):
+            os.remove(os.path.join(meta_dir, item))
diff --git a/setup.py b/setup.py
index d9f60d1..e92f5b0 100644
--- a/setup.py
+++ b/setup.py
@@ -41,6 +41,7 @@ def read_reqs(reqs_name):
     version=version,
     description="Downloads data and metadata from GEO and SRA and creates standard PEPs.",
     long_description=long_description,
+    long_description_content_type="text/markdown",
     classifiers=[
         "Development Status :: 4 - Beta",
         "License :: OSI Approved :: BSD License",


Test patch:
diff --git a/tests/test_geofetch.py b/tests/test_geofetch.py
index a64b4c0..f16b56f 100644
--- a/tests/test_geofetch.py
+++ b/tests/test_geofetch.py
@@ -1,6 +1,7 @@
-from geofetch import parse_accessions, Geofetcher
+from geofetch import parse_accessions, Geofetcher, utils
 import os
 import pytest
+import shutil
 
 INPUT_ACC_FILE = "tests/test_files/input_acc.txt"
 GSE_FILES = "tests/test_files/soft_files"
@@ -65,7 +66,11 @@ class TestListProcessedMetaFiles:
     @pytest.fixture(scope="function")
     def initiate_geofetcher(self, tmpdir):
         instance = Geofetcher(
-            just_metadata=True, processed=True, name="test", metadata_folder=tmpdir
+            just_metadata=True,
+            processed=True,
+            name="test",
+            metadata_folder=tmpdir,
+            data_source="all",
         )
         yield instance
 
@@ -91,15 +96,24 @@ def test_downloading_soft_files(self, initiate_geofetcher):
         assert "GSE138657_GSM.soft" in downloaded_meta_files
         assert "GSE138657_GSE.soft" in downloaded_meta_files
 
-    def test_creating_pep_files(self, initiate_geofetcher):
+    def test_creating_sample_pep_files(self, initiate_geofetcher):
         initiate_geofetcher.fetch_all("GSE138657")
-        downloaded_meta_files = list(os.walk(initiate_geofetcher.metadata_expanded))[0][
-            2
-        ]
+        downloaded_meta_files = list(
+            os.walk(initiate_geofetcher.metadata_expanded + "/PEP_samples")
+        )[0][2]
 
         assert "GSE138657_samples.csv" in downloaded_meta_files
         assert "GSE138657_samples.yaml" in downloaded_meta_files
 
+    def test_creating_series_pep_files(self, initiate_geofetcher):
+        initiate_geofetcher.fetch_all("GSE199313")
+        downloaded_meta_files = list(
+            os.walk(initiate_geofetcher.metadata_expanded + "/PEP_series")
+        )[0][2]
+
+        assert "GSE199313_series.csv" in downloaded_meta_files
+        assert "GSE199313_series.yaml" in downloaded_meta_files
+
 
 class TestDownloadingProcFiles:
     """
@@ -187,3 +201,65 @@ def test_filter(self, meta_list, output, initiate_geofetcher):
     def test_size_filter(self, meta_list, output, initiate_geofetcher):
         result = initiate_geofetcher.run_size_filter(meta_list)
         assert result == output
+
+    @pytest.mark.parametrize(
+        "init_meta_data,  result_sample, result_proj,",
+        [
+            (
+                [
+                    {
+                        "name": "Antonio",
+                        "number": 1,
+                        "car": "Fiat",
+                    },
+                    {
+                        "name": "Markus",
+                        "number": 1,
+                        "car": "Jeep",
+                    },
+                    {
+                        "name": "Pablo",
+                        "number": 1,
+                        "car": "Jeep",
+                    },
+                ],
+                [
+                    {
+                        "name": "Antonio",
+                        "car": "Fiat",
+                    },
+                    {
+                        "name": "Markus",
+                        "car": "Jeep",
+                    },
+                    {
+                        "name": "Pablo",
+                        "car": "Jeep",
+                    },
+                ],
+                [
+                    {
+                        "number": 1,
+                    },
+                ],
+            )
+        ],
+    )
+    def test_large_meta_separation(
+        self, init_meta_data, result_sample, result_proj, initiate_geofetcher
+    ):
+        samp, proj = initiate_geofetcher.separate_common_meta(init_meta_data, max_len=0)
+        assert samp == result_sample
+        assert proj == result_proj
+
+
+def test_clean_func(tmpdir):
+    """
+    Testing deleting soft files
+    """
+    files_dir = os.path.join(GSE_FILES, "GSE138657")
+    for file_name in os.listdir(files_dir):
+        shutil.copyfile(
+            os.path.join(files_dir, file_name), os.path.join(tmpdir, file_name)
+        )
+    utils.clean_soft_files(tmpdir)