Repository: biopython/biopython

Issue:
PDBParser not parsing all atoms
### Setup

I am reporting a problem with Biopython 1.77 (conda, conda-forge channel) , Python 3.8.5, and operating system as follows: 

```python
Python 3.8.5 | packaged by conda-forge | (default, Jul 31 2020, 02:39:48)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys; print(sys.version)
3.8.5 | packaged by conda-forge | (default, Jul 31 2020, 02:39:48)
[GCC 7.5.0]
>>> import platform; print(platform.python_implementation()); print(platform.platform())
CPython
Linux-4.15.0-76-generic-x86_64-with-glibc2.10
>>> import Bio; print(Bio.__version__)
1.77
```

### Expected behaviour

Parsing all atoms in the PDB file into the Biopython `structure` object. 
  
### Actual behaviour

The PDB file parses into a `structure` object however there are atoms missing. For reference, the PDB file has been modified from the one in the PDB by  [RING](http://old.protein.bio.unipd.it/ring/about) software. 

I get the warnings listed below when I parse the structure but there's nothing to suggest the missing atoms. I've parsed other PDB files and know there is normally an "atom missing" warning that can come up so I'm confused what's happening with this structure to parse without atom warnings and yet be missing atoms. 

```
/home/ubuntu/miniconda3/envs/msci_vep/lib/python3.8/site-packages/Bio/PDB/StructureBuilder.py:89: PDBConstructionWarning: WARNING: Chain A is discontinuous at line 24627.
  warnings.warn(
/home/ubuntu/miniconda3/envs/msci_vep/lib/python3.8/site-packages/Bio/PDB/StructureBuilder.py:89: PDBConstructionWarning: WARNING: Chain B is discontinuous at line 24745.
  warnings.warn(
```

To work out how many atoms are missing I saved the Biopython `structure` as a pdb file, assuming all atoms not in the `structure` aren't being read/can't be accessed using Biopython. 

Doing a `wc -l *` of both files shows there's a difference of 137 lines. 

> 24882 1EWQ_RING.pdb
> 24745 1EWQ_RING_bioout.pdb 

I also wrote [this script](https://github.com/edarl/rand-scripts/blob/master/missingHanalysis.py) to output exactly which atoms are missing, but I can't see anything in the atom list that looks like it wouldn't parse. 

I've attached my input file (1EWQ_RING.pdb), Biopython produced output file (1EWQ_RING_bioout.pdb) and the atom difference list (1EWQ_pdb_diff.csv) I made. 

Any help as to why those atoms aren't being parsed into Biopython would be greatly appreciated!


```python
def PDB_parser(PDB_name, PDB_path):
    parser = PDBParser(QUIET=False, PERMISSIVE=True)
    structure = parser.get_structure(PDB_name, PDB_path)
    return structure

def save_struct(structure, name):
    io = PDBIO()
    io.set_structure(structure)
    try:
        io.save(name)
        print(f"Successfully saved file:   {name}")
    except IOError:
        print("Unable to save file")
```
### Steps to reproduce
Parse PDB file to structure, save structure to a new PDB file, compare atoms in both PDB files.

FILES: [1EWQ.zip](https://github.com/biopython/biopython/files/5061544/1EWQ.zip)



Discussion:
It looks to be missing only some of the hydrogen atoms (and not other kinds
of atoms). Are they by any chance unusually far from the main residue atoms
in spatial distance?

I also noticed (and this is probably not relevant), and there is something
odd about the indexing in column 2 of your input - as though all the
hydrogens were added and numbered later?

```
ATOM    930  N   GLU A   2     116.401  91.673  95.693  1.00 94.62
  N
ATOM    931  CA  GLU A   2     116.727  90.430  96.377  1.00 93.91
  C
ATOM    932  C   GLU A   2     118.234  90.285  96.533  1.00 93.24
  C
ATOM    933  O   GLU A   2     119.003  90.554  95.609  1.00 93.17
  O
ATOM  35930  HA  GLU A   2     116.296  90.422  97.279  1.00  0.00
  H
ATOM    934  CB  GLU A   2     116.156  89.227  95.619  1.00 94.14
  C
ATOM    935  CG  GLU A   2     114.641  89.272  95.454  1.00 94.32
  C
ATOM    936  CD  GLU A   2     113.930  89.779  96.701  1.00 94.43
  C
ATOM    937  OE1 GLU A   2     114.113  89.185  97.786  1.00 94.39
  O
ATOM    938  OE2 GLU A   2     113.188  90.780  96.593  1.00 94.51
  O
ATOM  35936  HB2 GLU A   2     116.588  89.164  94.719  1.00  0.00
  H
ATOM  35938  HB3 GLU A   2     116.416  88.383  96.088  1.00  0.00
  H
ATOM  35940  HG2 GLU A   2     114.407  89.855  94.676  1.00  0.00
  H
ATOM  35942  HG3 GLU A   2     114.303  88.359  95.225  1.00  0.00
  H
ATOM    939  N   GLY A   3     118.639  89.857  97.722  1.00 92.41
  N
ATOM    940  CA  GLY A   3     120.046  89.687  98.024  1.00 91.39
  C
ATOM    941  C   GLY A   3     120.223  90.010  99.489  1.00 90.60
  C
ATOM    942  O   GLY A   3     121.319  90.332  99.951  1.00 90.50
  O
ATOM  35952  H   GLY A   3     117.966  89.518  98.379  1.00  0.00
  H
ATOM  35958  HA1 GLY A   3     120.585  90.325  97.474  1.00  0.00
  H
ATOM  35960  HA2 GLY A   3     120.311  88.737  97.857  1.00  0.00
  H
```

>

The issue might be that your missing hydrogens have serial numbers larger than 99999. Then, the number extends onto the record type and the parser might not recognize those as valid atom lines. In other words, RING is producing bad PDB files that do not conform to the format specifications :) A simple way around this problem is to just renumber the atoms. You can use [pdb-tools](http://www.bonvinlab.org/pdb-tools/) as a quick fix (pdb_reatom) before loading the file into Biopython.
I think you may be right Joao - the first missing hydrogen had number 100007. So, the question might now be why doesn't the parser raise an exception, or at least a warning?
Because we don't check for malformatted records. Since we do not parse all record types specified in the PDB format, there is no way we can figure out if a particular one we are reading is wrong. As such, the parser only looks for lines starting with `ATOM  ` (note the spaces) or `HETATM`. We *could* just use `line.startswith('ATOM')` but then we'd still get the serial number wrong anyway and might have other errors that would be hard to debug.

We could add a check to detected unexpected record types in the coordinate section. We know that there are a few limited options (`ATOM  `, `HETATM`, `ANISOU`, `TER`, `ENDMDL`, `MODEL` - [docs](https://www.wwpdb.org/documentation/file-format-content/format33/sect9.html)) so we could see if a record is not any of these and throw a warning then. This would be rather simple to implement.
Thank you both for the really quick help! 

I renumbered the file and it's now being read by Biopython with no missing atoms! I was aware the numbering was off but I didn't think it was causing the parsing problem as not all of the atoms that had different numbers were being missed by Biopython. 

A warning for coordinate-type lines not being imported would be really helpful, as it took me awhile to work out which lines/atoms were missing from the `structure` to begin with. 
Thanks for raising the issue @edarl. I'll create a PR soon to add a check for these types of problems. 

Resolving patch:
diff --git a/Bio/PDB/PDBIO.py b/Bio/PDB/PDBIO.py
--- a/Bio/PDB/PDBIO.py
+++ b/Bio/PDB/PDBIO.py
@@ -378,6 +378,6 @@ def save(self, file, select=_select, write_end=True, preserve_atom_numbering=Fal
             if model_flag and model_residues_written:
                 fp.write("ENDMDL\n")
         if write_end:
-            fp.write("END\n")
+            fp.write("END   \n")
         if close_file:
             fp.close()
diff --git a/Bio/PDB/PDBParser.py b/Bio/PDB/PDBParser.py
--- a/Bio/PDB/PDBParser.py
+++ b/Bio/PDB/PDBParser.py
@@ -141,6 +141,12 @@ def _get_header(self, header_coords_trailer):
 
     def _parse_coordinates(self, coords_trailer):
         """Parse the atomic data in the PDB file (PRIVATE)."""
+        allowed_records = {
+            "ATOM  ", "HETATM", "MODEL ", "ENDMDL", "TER   ", "ANISOU",
+            "SIGATM", "SIGUIJ",  # older 2.3 format specs
+            "MASTER",  # bookkeeping records after coordinates
+        }
+
         local_line_counter = 0
         structure_builder = self.structure_builder
         current_model_id = 0
@@ -150,12 +156,15 @@ def _parse_coordinates(self, coords_trailer):
         current_segid = None
         current_residue_id = None
         current_resname = None
+
         for i in range(0, len(coords_trailer)):
             line = coords_trailer[i].rstrip("\n")
             record_type = line[0:6]
             global_line_counter = self.line_counter + local_line_counter + 1
             structure_builder.set_line_counter(global_line_counter)
-            if record_type == "ATOM  " or record_type == "HETATM":
+            if not line.strip():
+                continue  # skip empty lines
+            elif record_type == "ATOM  " or record_type == "HETATM":
                 # Initialize the Model - there was no explicit MODEL record
                 if not model_open:
                     structure_builder.init_model(current_model_id)
@@ -374,6 +383,14 @@ def _parse_coordinates(self, coords_trailer):
                 ]
                 sigatm_array = numpy.array(sigatm, "f")
                 structure_builder.set_sigatm(sigatm_array)
+            elif record_type not in allowed_records:
+                warnings.warn(
+                    "Ignoring unrecognized record '{}' at line {}".format(
+                        record_type,
+                        global_line_counter
+                    ),
+                    PDBConstructionWarning,
+                )
             local_line_counter += 1
         # EOF (does not end in END or CONECT)
         self.line_counter = self.line_counter + local_line_counter


Test patch:
diff --git a/Tests/PDB/a_structure.pdb b/Tests/PDB/a_structure.pdb
--- a/Tests/PDB/a_structure.pdb
+++ b/Tests/PDB/a_structure.pdb
@@ -5,6 +5,7 @@ REMARK 4 CHANGE IN RESNAME
 REMARK WAT 67 IS PRESENT TWICE
 REMARK RESIDUE 0 SHOULD GENERATE A POLYPEPTIDE WARNING (DUPLICATED C ATOM)
 REMARK MISSING ELEMENTS IN THE FIRST RESIDUE
+REMARK LAST ATOM ENTRY HAS OVERFLOWING SERIAL NUMBER
 MODEL
 HETATM    1  N   PCA A   1       0.525   2.690  13.317  1.00 20.26      A
 HETATM    2  CA  PCA A   1      -0.993   2.924  13.160  1.00 21.23      A
@@ -773,6 +774,7 @@ ATOM    754  O   VAL B  52       6.444  -8.913  45.508  1.00 34.10           O
 ATOM    755  CB  VAL B  52       6.275  -6.995  43.504  1.00 29.40           C
 ATOM    756  CG1 VAL B  52       5.661  -6.261  42.321  1.00 30.99           C
 ATOM    757  CG2 VAL B  52       7.588  -6.386  43.856  1.00 23.53           C
+ATOM 111757  CG3 VAL B  52       7.588  -6.386  43.856  1.00 23.53           C
 TER     758      VAL B  52
 HETATM  759  O   HOH B   0     -32.632  12.449 -23.591  1.00 63.34           O
 HETATM  759  O   HOH C   0       6.997   5.537  34.149  1.00 15.09           O
diff --git a/Tests/PDB/ions.pdb b/Tests/PDB/ions.pdb
--- a/Tests/PDB/ions.pdb
+++ b/Tests/PDB/ions.pdb
@@ -1,6 +1,4 @@
 HEADER ION STRUCTURE
 HETATM  624 MG    MG A   1      17.366   9.444  50.463  1.00 56.45          MG  
 HETATM  625 MG    MG A   2       5.160   5.994  22.521  1.00 67.47          MG  
-END
-
-
+END   
diff --git a/Tests/test_PDB.py b/Tests/test_PDB.py
--- a/Tests/test_PDB.py
+++ b/Tests/test_PDB.py
@@ -74,25 +74,26 @@ def test_1_warnings(self):
             p = PDBParser(PERMISSIVE=True)
             p.get_structure("example", "PDB/a_structure.pdb")
 
-            self.assertEqual(len(w), 14)
+            self.assertEqual(len(w), 15)
             for wrn, msg in zip(
                 w,
                 [
                     # Expected warning messages:
                     "Used element 'N' for Atom (name=N) with given element ''",
                     "Used element 'C' for Atom (name=CA) with given element ''",
-                    "Atom names ' CA ' and 'CA  ' differ only in spaces at line 17.",
+                    "Atom names ' CA ' and 'CA  ' differ only in spaces at line 18.",
                     "Used element 'CA' for Atom (name=CA  ) with given element ''",
-                    "Atom N defined twice in residue <Residue ARG het=  resseq=2 icode= > at line 21.",
-                    "disordered atom found with blank altloc before line 33.",
-                    "Residue (' ', 4, ' ') redefined at line 43.",
-                    "Blank altlocs in duplicate residue SER (' ', 4, ' ') at line 43.",
-                    "Residue (' ', 10, ' ') redefined at line 75.",
-                    "Residue (' ', 14, ' ') redefined at line 106.",
-                    "Residue (' ', 16, ' ') redefined at line 135.",
-                    "Residue (' ', 80, ' ') redefined at line 633.",
-                    "Residue (' ', 81, ' ') redefined at line 646.",
-                    "Atom O defined twice in residue <Residue HOH het=W resseq=67 icode= > at line 902.",
+                    "Atom N defined twice in residue <Residue ARG het=  resseq=2 icode= > at line 22.",
+                    "disordered atom found with blank altloc before line 34.",
+                    "Residue (' ', 4, ' ') redefined at line 44.",
+                    "Blank altlocs in duplicate residue SER (' ', 4, ' ') at line 44.",
+                    "Residue (' ', 10, ' ') redefined at line 76.",
+                    "Residue (' ', 14, ' ') redefined at line 107.",
+                    "Residue (' ', 16, ' ') redefined at line 136.",
+                    "Residue (' ', 80, ' ') redefined at line 634.",
+                    "Residue (' ', 81, ' ') redefined at line 647.",
+                    "Ignoring unrecognized record 'ATOM 1' at line 777",
+                    "Atom O defined twice in residue <Residue HOH het=W resseq=67 icode= > at line 904.",
                 ],
             ):
                 self.assertIn(msg, str(wrn))
@@ -902,7 +903,7 @@ def test_duplicated_residue_permissive(self):
             "HETATM 6289  O   HOH     5      28.182  -5.239  31.370  1.00 22.99           O\n"
             "HETATM 6513  O   HOH     6      21.829   3.361  14.003  1.00 14.25           O\n"
             "HETATM 6607  O   HOH     5      33.861  40.044  18.022  1.00 18.73           O\n"
-            "END\n"
+            "END   \n"
         )
 
         parser = PDBParser()
@@ -922,7 +923,7 @@ def test_duplicated_residue_strict(self):
             "HETATM 6289  O   HOH     5      28.182  -5.239  31.370  1.00 22.99           O\n"
             "HETATM 6513  O   HOH     6      21.829   3.361  14.003  1.00 14.25           O\n"
             "HETATM 6607  O   HOH     5      33.861  40.044  18.022  1.00 18.73           O\n"
-            "END\n"
+            "END   \n"
         )
 
         parser = PDBParser(PERMISSIVE=False)