Google Summer of Code 2019 — Final Report
By: Deepak Kumar
Tags: Bioinformatics, Gsoc 2019, Intermine

Project: InterMine Schema Validator

Name : Deepak Kumar

: Deepak Kumar Github Id: deepakkumar96

deepakkumar96 Repository: https://github.com/intermine/biovalidator

https://github.com/intermine/biovalidator Mentors: Julie Sullivan and Rachel Lyne

Julie Sullivan and Rachel Lyne Organization: InterMine

Project Links

Repository (this repository is dedicated for this project only): https://github.com/intermine/biovalidator Project’s User docs: https://github.com/intermine/biovalidator/wiki Project’s Javadoc: http://intermine.org/biovalidator/javadoc/

My Major Pull Requests and Commits:

Pull Request #1 (API and FASTA Validator): https://github.com/intermine/biovalidator/pull/4 Pull Request #2 (Biovalidator API Changes): https://github.com/intermine/biovalidator/pull/10 Pull Request #3 (GFF3 and CSV validator): https://github.com/intermine/biovalidator/pull/26 Pull Request #4 [Yet to be merged](VCF validator): https://github.com/intermine/biovalidator/pull/29 All of my Github commits: https://github.com/intermine/biovalidator/commits?author=deepakkumar96

About the Project

InterMine works with a lot different biological data files, and it wants its users to directly upload biological data files to InterMine from their browser, In this process, InterMine wants to make sure that each file uploaded by a user is syntactically and semantically correct and it is following the file format specification. So, the project’s main goal was to design and implement a library that can validate different biological data files(such as FASTA, GFF), so that if a user submitted a wrong file(ex. if a FASTA file is supposed to be containing DNA sequence but rather has PROTEIN sequences), then the library can detect error and report all errors and warnings found in the file.

Another task of this project was to create a command-line utility to validate biological files directly from the terminal, the command-line utility will use the same java library to validate the files. The Idea of command-line utility is that, if a user gets an error while uploading his/her files to InterMine, then the user can explore the file using command-line utility and fix error and warning present in the file.

Project Requirements:

A standalone Java library for validating biological file formats

A command-line utility to validate file formats from terminal

Project Tasks:

BioValidator API and FASTA validator, to validate FASTA files( DNA and PROTEIN sequences). GFF3 validator and CSV validator. VCF validator (bonus/extra task) Command-line utility for validating files from the terminal using the same library

1. BioValidator API implementation and FASTA Validator (link to API and Fasta Pull Request)

1. API design and Implementation:

The first task was to design and implement a command validator API, this API defines how errors, warning, and validation-result will be represented, API also abstract out common functionalities of the project, and each validator will be an implementation of this API.

API also defines how a user can customize validation result, user can enable/disable errors and warnings, strict/permissive mode and other can specify other rules such as whether to continue validation even if an error occurred or not.

API usage Example:

Validator API uses

1.2. Fasta Validator

Fasta is a file format that is used to store either nucleotide sequences(i.e. DNA or RNA) or amino acid (i.e. protein). The FASTA file format does not have any formal specification but there are generally accepted rules that communities follow.

Types of FASTA validation mode

fatsa : look for formatting issues and valid DNA and PROTEIN sequence

: look for formatting issues and valid DNA and PROTEIN sequence fasta-dna : validate whether a file has valid DNA sequence or not

: validate whether a file has valid DNA sequence or not fasta-protein: validates whether a file has valid PROTEIN sequence or not

Using Fasta validator with Java library:

Fasta Validator Uses Example

Using Fasta validator with Command-line tool:

$ java -jar biovalidator-fat.1.2.jar -f=filepath -t fasta

2. GFF3 and CSV validator(link to gff3 and csv Pull Request)

2.1. GFF3 validator

The goal of this task was to create a validator for GFF3 file formats(not any previous version of GFF), unlike FASTA, GFF3 has a formal specification and GFF3 validator does follow the rules given by the GFF3 specification. This task was a little difficult than FASTA as GFF3 has a lot of rules that require a lot of string manipulation, which create a lot of performance and memory consumption issues.

GFF3 validation can be used similarly the way FASTA was used either from java library or terminal using biovalidator jar file.

2.2 Csv/Tsv Validator

The third task was to create a validator for CSV/TSV file, biological data can also be represented as CSV/TSV file and CSV files are just simple text file that can contain any type of data, so the goal of this task was to create a validator that can validate the consistency of a CSV or TSV file.

CSV validator does not validate against schema but rather it validates the consistency of CSV data, it checks whether a particular column is consistent throughout all the rows of CSV data or not.

CSV validator Features:

Performs consistency check on CSV/TSV data

Detect CSV delimiter automatically

Detect whether a CSV data has header line or not(so you may add a header or not may not in your CSV file, will work in most cases)

Two types of checks performed by CSV/TSV validator:

Whether a column has one particular type of data or not(such as integer, boolean, etc..) if it is then validator checks whether all of the rows have the same type of data or not. If data of a column has mixed kind of data, then validator checks whether all of the rows of a particular column are following on more patterns or not.

CSV validator example (through command-line utility):

3. VCF validator[Not yet merged] (link to Pull Request)

This was the last task which is not yet merged but completed. The goal of this task was to create a validator for validating VCF files. VCF validator validates

4. Command-line utility:

The main goal of this project was to create a Java library for validating various biological files, but besides a java library, another requirement was to create a command-line utility for validating files from the terminal.

There are two main reasons for the command-line utility

If a user gets an error while uploading a file to InterMine, then the user can use the command-line tool to get all the detailed errors and warnings from his/her terminal. The command-line utility will be useful for a biologist or any non-tech user who just wants to know whether a file is valid or not.

Command-line utility example run:

User Documentation:

https://github.com/intermine/biovalidator/wiki

Documentation is written from a user perspective, how a user can use this library to validate supported file formats using both Java library and command-line utility.

Documentation has all the validation rules that are followed by validator while validating any biological file.

Project’s JavaDocs:

http://intermine.org/biovalidator/javadoc/

Future plans on the project: