Repository: biopython/biopython

Issue:
682i.cif fails on parse of resolution to structure header
### Setup

I am reporting a problem with Biopython 1.77, Python 3.8.3 (anaconda July 2020) , and CentOS 7 as follows:

```(base) [mothcw@localhost biopython]$ python
Python 3.8.3 (default, Jul  2 2020, 16:21:59) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; print(sys.version)
3.8.3 (default, Jul  2 2020, 16:21:59) 
[GCC 7.3.0]
>>> import platform; print(platform.python_implementation()); print(platform.platform())
CPython
Linux-3.10.0-1127.18.2.el7.x86_64-x86_64-with-glibc2.10
>>> import Bio; print(Bio.__version__)
1.77
```

### Expected behaviour

682i.cif would load, with no halting exception, with the code:

```
from Bio.PDB import MMCIFParser

mmCIF_parser = MMCIFParser(QUIET=True)
structure_cif = mmCIF_parser.get_structure('6g2i','6g2i.cif')
```
### Actual behaviour

Parsing fails with:

Traceback (most recent call last):
  File "./mmcif_fail.py", line 8, in <module>
    structure_cif = mmCIF_parser.get_structure('6g2i','6g2i.cif')
  File "/home/mothcw/anaconda3/lib/python3.8/site-packages/Bio/PDB/MMCIFParser.py", line 63, in get_structure
    self._structure_builder.set_header(self._get_header())
  File "/home/mothcw/anaconda3/lib/python3.8/site-packages/Bio/PDB/MMCIFParser.py", line 112, in _get_header
    self.header["resolution"] = float(self.header["resolution"])
ValueError: could not convert string to float: '.'

### Steps to reproduce

1) Download (and gunzip) 6g2i.cif from rcsb.  It is a 32MB CryoEM structure.
2) Run this code:
```
$ cat mmcif_fail.py
#!/usr/bin/env python

"""Short program to demonstrate that 6g2i.cif fails to load with biopython 1.77"""

# Parse in the small 6g2i.cif file downloaded from rcsb
from Bio.PDB import MMCIFParser
mmCIF_parser = MMCIFParser(QUIET=True)
structure_cif = mmCIF_parser.get_structure('6g2i','6g2i.cif')

print("Resolution is %s"%structure_cif.header['resolution'])
```
Note the failure message.  

### Towards a fix

There are a couple of problems afoot here.

First, line 113 or so of MMCIFParser.py is not considering _em_3d_reconstruction.resolution to find the 5.8A resolution.  I have added that as a 3rd item in the search keys - and it helps.

```
        self._update_header_entry(
            "resolution", ["_refine.ls_d_res_high", "_refine_hist.d_res_high", "_em_3d_reconstruction.resolution"]
        )
```

HOWEVER, in 6g2i.cif itself there is an empty '.' entry for _**refine.ls_d_res_high**.

_refine.ls_abs_structure_Rogers_esd              ?
_refine.ls_d_res_high                            **.**
_refine.ls_d_res_low                             ?


This period to right of d_res_high could be a bug in the file itself - but I am not totally sure of the regular expression for this field defined at RCSB.

For me right now, I have modifed MMCIFParser.py to treat both lone ? and lone period as missing for purpose of filling in the header.  The bug finally went away when I changed line 84 from:

if item != "?":

to

if item != "?" **and item != ".":**

To my eye this is redundant with the _unassigned idea of line 332.  I recommend that, if this is the same kind of thing, that you centralize the notion of unassigned so we are not patching it multiple places.

May I kindly/friendlysuggest that, at minumum, Biopython should be tested against all .cif files available at rcsb/wwwpdb prior to each release?  This is not such a painful thing to do.  I have to contend that these formally curated and deposited files should not trigger runtime exceptions like this.

Idea: Perhaps more importantly for longer term, this could be yet another case where biopython is stuck in a legacy .PDB-oriented worldview.  Perhaps we need to stop initializing 'header' when working with MMCIF.  For MMCIF files, the parsing of the .cif into the dictionary should be the central/main event, and not have a side-effect of initializing a legacy 'header' (that attempts to recreate PDB REMARK entries).  Once the dictionary is parsed, conversion to a Structure for those who want model/chain/residue/atom traversal might be interesting.  But, resolution should come from the parsed in dictionary - and not be set by code trying to replicate the old REMARK RESOLUTION of the PDB.








Discussion:
I suspect this is my code for adding the 'pdb header' info to cif data, so
should probably offer some viewpoint (I'm moving house this week so not
certain I can look at any code immediately).  Sorry I apparently didn't run
against all of the .cif files in testing, and yes that seems like a good
idea to do regularly - not being the person who has to organise the
releases :-).

Idea: Perhaps more importantly for longer term, this could be yet another
> case where biopython is over stick in a PDB-oriented worldview, and we need
> to let 'header' go altogether when working with MMCIF. For MMCIF files, the
> parsing of the .cif into the dictionary should be the central/main event,
> and not have a side-effect of initializing a legacy 'header'. Once the
> dictionary is parsed, conversion to a Structure for those who want
> model/chain/residue/atom traversal might be interesting. But, resolution
> should come from the parsed in dictionary - and not be set by code trying
> to replicate the old REMARK RESOLUTION of the PDB.


Please expound on the applications for MMCIF files outside of the MCRA
hierarchy that you are working towards?  It seems like much of
Biopython/PDB relies on MCRA, but I agree that loading the cif dictionary
should be distinguishable from interpreting it into a Structure object.  My
effort was toward establishing some core metadata that would be dependably
available for Structure objects, as I find that files of only coordinates
lose all value within a few minutes and try to avoid this with as much
automatic annotation as possible.  Clearly PDB format can’t capture
everything MMCIF does (that’s the point), but my own preference is to map
some key metadata between the two so that higher level code can rely on a
few properties and hopefully support some level of interconversion.

—
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/biopython/biopython/issues/3195>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAZW3QC6D7XG6FSH3VTT7T3R654TXANCNFSM4PT6JIQA>
> .
>

Rob,

Thanks for your quick note.  A little more on our application.

I work on a high throughput automated software pipeline that turns around hundreds to thousands of calculations on tens to hundreds (or more) PDB structures from the time a sick patient's genetic variants are presented to us on Monday, until we present academic research finding to physicians on Friday.

As inputs to the calculations (specifically here  rosetta ddg_nomomer), the code must know the experimental method , and resolution, of the structures we are processing.  There is no time to hand-curate these structures.  They are automatically selected based on cross-references from uniprot, and aligned with SIFTS resources.

I also use the mmcif dictionary to _automatically_ identify non-human variants insert into the structure to aid crystalization or purification.  I remove these.  These are in dictionary elements of form "_struct_ref_seq.*" The present MMCIF parser in Biopython is glorious in its comprehensive dictionary parser but frustrating in that the dictionary is obscured by an underscore after parsing (I 'steal' it presently - and all OK so far).

I only made the switch to MMCIF in the last year, as CryoEM structures overrun PDB storage capacity.  Also, the structural biology world has been claiming MMCIF is  "the future" for years.  Yet, when deposited files crash our pipeline....    

A much larger gripe with mmCIF is of course that the PDBe/RCSB provides no Python 3 based parsers at all.  If that were available, we should use that to be SURE we always can read our files.  In the case of 682i.cif, it could well be that the deposited .cif with the '.' is non-compliant.   (In a perfect world, Biopython's MMCIF might would take the grammar/schema of the RCSB/PDBe as the holy gospel, and parse to that.  That could stop a lot of the bugs that come up with individual items turn up to be lists, or when a '.' appears in a field that had heretofore had a '?' to represent omission.  I don't know.  I just need a parser that works and gives me access to all the details in the .cif file).

I join you in realizing that RESOLUTION is a particularly frustrating case because it is fraught with experimental nuances that the old PDB entry avoided entirely.  I even had an extended back and forth last year with PDBe  support on how to recreate the PDB REMARK RESOLUTION from .cif format  - and they gave wrong advice.  My production code to determine resolution from structures is rather like yours - but it fishes out resolution based on method. 

```
       ci.resolution = None # Not applicable is correct for NMR structures - NMR code omitted from below
        if ci.method == 'X-RAY DIFFRACTION':
            resolution_keys = ['_refine.ls_d_res_high','_reflns.d_resolution_high','_refine_hist.d_res_high']
            ci.method = 'X-RAY'
        elif ci.method == 'ELECTRON MICROSCOPY':
            ci.method = 'EM'
            resolution_keys = ['_em_3d_reconstruction.resolution']
 else:
            resolution_keys = ['_refine.ls_d_res_high','_reflns.d_resolution_high','_refine_hist.d_res_high']
            LOGGER.critical('%s Experimental method for %s is unknown'%(ci.method,ci.structure_id))
```

In summary, given what BioPython offers today, what I think we know about the data structures and biology, and that old code should not be broken, my working fantasy for the MMCIF Parser is that users of biopython could do this going forward.

```
# This only fails in case of malformed input, with clear exception as to failing line of the input
# in that case
mmcif_dict = MMCIF2Dict(filename_with_gz_ok_____or______open_file_handle) 
cif_parser = MMCIFParser()

# Build the MCRA - but don't build monuments to legacy PDB data structures :)
structure = cif_parser.build_structure(mmcif_dict,
                 maybe: handle_problems=LogCritical/Die/Ignore/CallUserFunction/etc)

# For those who want it, make public the PDB header emulator
legacy_header = cif_parser.gen_fake_legacy_pdb_header(mmcif_dict)

# Then, if you wanted, maybe do this below.  A bad thing about the current class design is that there
# is no Structure.header at runtime, and that member not documented in the pydoc, because it is 
# presently monkey-patched.  So __init__ for Structure really should set self.header={} IMHO

structure.set_header(legacy_header)
```

And of course, the existing code MMCIFParser()... get_structure... should make these same _public_ calls too.

Sorry for so much verbiage - hope it helps.  Thanks for thinking about this.  Our field so needs a "go to"  Python 3 CIF parser that ALWAYS works!   I've never seen so much wheel reinvention - and that is ultimately what is slowing adoption of CIF fundamentally IMHO.


I've submitted PR #3229 to fix the '.' values for guessing resolution and add the `_em_3d_reconstruction.resolution` key.  In the PR I've listed 5 files that still error out, but that's better than the ca. 300 that the '.' were breaking.  I agree, seems like we should be able to load all of mmCIF if we can.

I'm still holding the opinion that all Structures should have some basic metadata tags if at all possible (so perhaps optionally avoid, but not optionally include) but notice I haven't tried to make it complain mercilessly if it's not there :-).  My feeling is code should be agnostic about the source format and always be able to print _something_ descriptive about a molecule beyond the simple characteristics.  I don't need it to be legacy PDB support, just some lowest common denominator set of tags -- and would be happy to contribute/discuss more about what those should be. 
Have the deposited file bugs been reported to wwwpdb/rcsb?  Just looking at the first one informs me they are not quality-checking fully at deposition time:

HETATM 4435 C  C9   . DD9 I 5 .   ? -19.851 5.852   -22.109 1.00 45.60 ? 308 DD9 A C9   1
HETATM 4436 H  H3A  . DD9 I 5 .   ? -20.027 4.711   -15.223 0.00 47.45 ? 308 DD9 A H3A  1
HETATM 4437 H  H3A  . DD9 I 5 .   ? -20.209 6.087   -14.466 1.00 49.90 ? 308 DD9 A H3A  1
HETATM 4438 H  H4   . DD9 I 5 .   ? -18.017 5.210   -16.035 1.00 43.15 ? 308 DD9 A H4   1

If not, I'll sleuth all of these, and file a ticket with their helpdesk.  It's not right that they have sent us to their .cif format, and are not providing complete integrity checks at deposition time.

(  The error on 6gux is also in the .pdb download:

ETATM 4437  H3A DD9 A 308     -20.027   4.711 -15.223  0.00 47.45           H
ANISOU 4437  H3A DD9 A 308     6009   6009   6009      0      0      0       H  )
HETATM 4438  H3A DD9 A 308     -20.209   6.087 -14.466  1.00 49.90           H
ANISOU 4438  H3A DD9 A 308     4519   7154   7287   -522   -198   -103       H

If no other discussion, I'll sleuth the others and copy my note to rcsb/wwpdb to this ticket - in a few days.

6gux PDBConstructionException: Atom H3A defined twice in residue <Residue DD9 het=H_DD9 resseq=308 icode= >
5o61 PDBConstructionException: Blank altlocs in duplicate residue ILE (' ', 105, ' ')
1ejg PDBConstructionException: Blank altlocs in duplicate residue SER (' ', 22, ' ')
6gob PDBConstructionException: Atom CL2 defined twice in residue <Residue F6Q het=H_F6Q resseq=205 icode= >
6goh PDBConstructionException: Atom CL2 defined twice in residue <Residue F7T het=H_F7T resseq=206 icode= >
No, I did not raise with anyone upstream - please do.


On Mon, Aug 24, 2020 at 3:38 PM Chris Moth <notifications@github.com> wrote:

> Have the deposted file bugs been reported to wwwpdb/rcsb? Just looking at
> the first one informs me they are letting
>
> HETATM 4435 C C9 . DD9 I 5 . ? -19.851 5.852 -22.109 1.00 45.60 ? 308 DD9
> A C9 1
> HETATM 4436 H H3A . DD9 I 5 . ? -20.027 4.711 -15.223 0.00 47.45 ? 308 DD9
> A H3A 1
> HETATM 4437 H H3A . DD9 I 5 . ? -20.209 6.087 -14.466 1.00 49.90 ? 308 DD9
> A H3A 1
> HETATM 4438 H H4 . DD9 I 5 . ? -18.017 5.210 -16.035 1.00 43.15 ? 308 DD9
> A H4 1
>
> If not, I'll sleuth all of these, and file a ticket with their helpdesk.
> It's not right that they have sent us to their .cif format, and are not
> providing complete integrity checks at deposition time.
>
> ( The error is also in their .pdb download:
>
> ETATM 4437 H3A DD9 A 308 -20.027 4.711 -15.223 0.00 47.45 H
> ANISOU 4437 H3A DD9 A 308 6009 6009 6009 0 0 0 H )
> HETATM 4438 H3A DD9 A 308 -20.209 6.087 -14.466 1.00 49.90 H
> ANISOU 4438 H3A DD9 A 308 4519 7154 7287 -522 -198 -103 H
>
> 6gux PDBConstructionException: Atom H3A defined twice in residue <Residue
> DD9 het=H_DD9 resseq=308 icode= >
> 5o61 PDBConstructionException: Blank altlocs in duplicate residue ILE ('
> ', 105, ' ')
> 1ejg PDBConstructionException: Blank altlocs in duplicate residue SER ('
> ', 22, ' ')
> 6gob PDBConstructionException: Atom CL2 defined twice in residue <Residue
> F6Q het=H_F6Q resseq=205 icode= >
> 6goh PDBConstructionException: Atom CL2 defined twice in residue <Residue
> F7T het=H_F7T resseq=206 icode= >
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/biopython/biopython/issues/3195#issuecomment-679101429>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAZW3QG7AVX6GTGD35XJBQ3SCJNN3ANCNFSM4PT6JIQA>
> .
>


Resolving patch:
diff --git a/Bio/PDB/MMCIFParser.py b/Bio/PDB/MMCIFParser.py
--- a/Bio/PDB/MMCIFParser.py
+++ b/Bio/PDB/MMCIFParser.py
@@ -93,7 +93,7 @@ def _update_header_entry(self, target_key, keys):
                 item = val[0]
             except (TypeError, IndexError):
                 continue
-            if item != "?":
+            if item != "?" and item != ".":
                 self.header[target_key] = item
                 break
 


Test patch:
diff --git a/Tests/PDB/1SSU_mod.cif b/Tests/PDB/1SSU_mod.cif
--- a/Tests/PDB/1SSU_mod.cif
+++ b/Tests/PDB/1SSU_mod.cif
@@ -2,6 +2,7 @@ data_1SSU
 #
 _refine.ls_d_res_high                            . 
 _refine.ls_d_res_low                             ? 
+_em_3d_reconstruction.resolution 4.1
 #
 loop_
 _atom_site.group_PDB
diff --git a/Tests/test_PDB_MMCIFParser.py b/Tests/test_PDB_MMCIFParser.py
--- a/Tests/test_PDB_MMCIFParser.py
+++ b/Tests/test_PDB_MMCIFParser.py
@@ -349,9 +349,10 @@ def test_header(self):
         self.assertEqual("X-RAY DIFFRACTION", structure.header["structure_method"])
         self.assertEqual(1.7, structure.header["resolution"])
 
-        # test not confused by '.'
+        # test not confused by '.' or '?'
         structure = parser.get_structure("example", "PDB/1SSU_mod.cif")
-        self.assertIsNone(structure.header["resolution"])
+        # self.assertIsNone(structure.header["resolution"])
+        self.assertEqual(4.1, structure.header["resolution"])
 
 
 class CIFtoPDB(unittest.TestCase):