opened
schristley
username_0
Based upon the schema William and others initially devised, formalize the schema with the OpenAPI spec and integrate with the other schema and reference libraries. The actual germline annotation sets with data for various species are stored in another AIRR [repository](https://github.com/airr-community/germline-sets)

- formalize schema
- formalize file format (current yaml).
- link repertoire/rearrangement to germline. The link is currently at the rearrangement level but might want to reconsider this, maybe at the repertoire level with software processing and annotation.
- provide API in reference library to load germline annotation sets from file and perform basic operations like gene lookup and retrieve gene annotations.
- provide tool to generate IgBlast internal tables with appropriate annotations (this is if IgBlast doesn't want to read the germline annotation file directly).
- provide tool that performs a "diff" between two germline annotation sets.
- documentation, examples, etc.

A nice little example use case would be to take an existing AIRR TSV file, specify a different germline annotations set (e.g. one with inferred alleles in it), rerun IgBlast to generate new AIRR TSV with the different germline, then perform a simple analysis comparison, like V gene allele usage, between the before/after data.
data model and format for germline annotation sets
issue
created
williamdlees
username_1
Looks good, Scott.

Should we provide a tool to generate germline sets from IMGT libraries?
comment
created
schristley
username_0
That's a good question. There are two points to that. First are the gene sequences and annotations, and the second is combining a set of those genes into a germline set. Right now my belief (which could be incorrect) is that IMGT is just a set of unstructured sequences, mouse is a good example, the genes for all the different strains are mixed together, so IMGT doesn't really define germline sets as I'm thinking about them. Thus my initial thought is no, and the germline sets will likely be hand-curated for the moment.

I see these annotations sets as living side-by-side with the IMGT sequences. That is, people will still need to download the IMGT gapped (and ungapped) FASTA sequences for tools that need access to the sequence, while AIRR annotations are provided which are not easily extracted from LIGM-DB or other sources.
comment
created
javh
username_2
```

So, you could parse this info into separate reference sets, build differently named igblast databases for each set, and then align against them specifically using the igblastn arguments `-germline_db_V`, `germline_db_D` and `-germline_db_J`.  Also, it looks like igblastn now has some arguments to restrict the database search to specific germline sequences (eg, `-germline_db_V_seqidlist`), but I don't know how these work...
comment
created
williamdlees
username_1
I like the idea of referencing the IMGT data rather than copying it.
comment
created
laserson
username_3
I'd be strongly in favor of making available a version of IMGT that corresponds to the structured schema here. Requiring people to download IMGT also just seems like unnecessary barrier, as long as we implement a mechanism so that we're basically religious about always keeping things in sync with IMGT.
comment
created
williamdlees
username_1
If we can resolve copyright with IMGT, I agree that would be the best course.
comment
created
ssnn-airr
username_4
comment
created
bcorrie
username_5
This is related to a topic that we have been discussing recently, and Scott brought up in his original post... Where is the correct place to reference the germline gene set. It is my understanding that there are at least two places where one might want to track this, one is the germline set that is used for annotation, but the other might be the germline set that is know for the individual. If I am not mistaken, one is a biological thing (this individual has this germline set - which you may or may not know) and the other is a software processing thing (this is the germline set that was used to generate this set of rearrangements).

It seems to me that the current place where this is tracked is probably not the most logical. Currently the germline_database used for an annotation is tracked for each rearrangement, which probably doesn't make sense??? 

This is being discussed here as well: https://github.com/airr-community/airr-standards/pull/156#issuecomment-447919836
comment
created
schristley
username_0
Forgot to mention this but in emails with Jiang @ NCBI, he indicated that IgBlast would be willing to use our AIRR annotation file, as replacement to their custom auxiliary_data.
comment
created
schristley
username_0
Hi @username_1 , created this issue long time ago with a wishlist of things. Amazingly, we actually completed some of them!

Maybe good to review our list, anything to add, anything to remove?

I still think some of the tools ideas are interesting, like a "diff" program. Are these things that you've development prototypes in OGRDB or elsewhere? Any other useful utility ideas you can think of? Do we care about an IMGT tool anymore?

I do want to get support in IgBlast for our format, but before we do that I think we need some actual germline set files. One ideas is to take my [earlier work](https://github.com/airr-community/germline-sets) and update to the new format. How far along are with OGRDB generating a complete germline set file, is that even your intent, or just a file containing inferred alleles?
comment
created
williamdlees
username_1
-  link repertoire/rearrangement to germline. The link is currently at the rearrangement level but might want to reconsider this, maybe at the repertoire level with software processing and annotation.

this was I think one of the things that got us into a tangle in late May/early June. One for the future possibly?

- provide API in reference library to load germline annotation sets from file and perform basic operations like gene lookup and retrieve gene annotations.

I don't see this as on the top of my personal list. Within OGRDB we intend to publish germline sets both in AIRR format and in FASTA format, which gives an easy route for people who want to do this kind of thing without understanding the json. BUt would be great if someone would like to take it on.

- provide tool to generate IgBlast internal tables with appropriate annotations (this is if IgBlast doesn't want to read the germline annotation file directly).

I have python code for this which I'm intending to use to produce OgBLAST files for germline sets as we publish them. As I get to it I will publish this in its own repo so that we can reference from airr standards. 

- provide tool that performs a "diff" between two germline annotation sets.
could be combined with the example code described above, perhaps. OGRDB now has built-in history and diff type annotations for germline sets as they are published there

- Should we provide a tool to generate germline sets from IMGT libraries?
documentation, examples, etc.

Agreed, documentation is needed. Will work on this. Have to say I do find the markup format cumbersome and unfriendly. I'm sure it gets better if you are used to it!
comment
created
bcorrie
username_5
Maybe the near future? There are a couple of low hanging fruit here in my opinion, using CURIE nomenclature for IDs.

- In our current DataProcessing object, we have germline_database. This could either be changed to refer to a CURIE that points to a GermlineSet (e.g. germline_database = "OGRDB_GERMLINESET:42") rather than a string or to treat it like an ontology if we want to have a text description of the germline database used as well as have an identifier for it (e.g. germline_database.label = "OGRDB Germline, downloaded 2021-09-17" and germline_database.id = "OGRDB_GERMLINESET:42". Presumably even if we re-engineer DataProcessing, something similar would still exist. DataProcessing objects will need to refer to the GermlineSets that were used to perform the data processing...
- subject.genotype already refers to GermlineSets, so no work needs to be done there to reference GermlineSet. We still need to consider how to track how the Genotype was produced if it was inferred.
- If we wanted to go even further, we could add rearrangement fields that store gene IDs for VDJC calls. v_call = "IGHV1-1*01", v_call_id = "OGRDB_GENE:44".
 
Presumably if you used the germline_database.id from a DataProcessing that was applied to a Rearrangement,  if you looked up (using OGDRB's API) OGRDB_GENE:44 in OGRDB_GEMRLINESET:42, you would get a gene label of IGHV1-1*01. Or alternately look up label IGHV1-1*01 in OGRDB_GEMRLINESET:42 and get "OGRDB_GENE:44". And if you look up OGRDB_GENE:44 in total you would get the full description of the AlleleDescription for OGRDB_GENE:44. And of course If you wanted to you could download the entire GermlineSet for OGRDB_GERMLINESET:42. 

For completeness, the CURIE prefixes might be something like this:

ORDB_GENE = http://ogrdb.org/airr/v2/gene/ so OGRDB_GENE:44 would resolve to: http://ogrdb.org/airr/v2/gene/44

OGRDB_GERNLINESET = http://ogrdb.org/airr/v2/germline/ so OGRDB_GERMLINESET:42 would resolve to http://ogrdb.org/airr/v2/germline/42
comment
created
bcorrie
username_5
FYI, when we annotated the Genotype for Palanichamy et al we used the above approach. Our processing was done using the IMGT 201736-4 germline set. So the subject.genotype has the germline_set_ref = "IMGT:201736-4"

If we use a CURIE prefix mapping  (using the new ontovoc notation) for URL = "http://www.imgt.org/download/V-QUEST/archives/{local_id}.zip" and we assumed the IMGT archive went back that far (which it doesn't), this would resolve to the correct IMGT germline database. If you used IMGT:202109-3 this would actually resolve to the actual Germline DB form IMGT (if I understand their archives correctly).
comment
created
bcorrie
username_5
Maybe we should move the above to a new issue to capture effort to make these links???
comment
created
williamdlees
username_1
Sure, let's move it to a new issue.
comment
closed
williamdlees
username_1
issue