Repository: biopython/biopython

Issue:
PDB Entity.copy() mangles disordered atoms.
Disordered atoms dont inherit `disordered_get_list()` when a parent entity is copied. Eg:

```
>>> pdbl = Bio.PDB.PDBList(server='http://www.rcsb.org/pdb/files')
>>> pdbParser = Bio.PDB.PDBParser()
>>> struct_4LGR = pdbParser.get_structure('4LGR', pdbl.retrieve_pdb_file('4LGR'))
Downloading PDB structure '4LGR'...
>>> Bio.PDB.PPBuilder().build_peptides(struct_4LGR)
[<Polypeptide start=5 end=33>, <Polypeptide start=36 end=259>, <Polypeptide start=2 end=123>]
>>> Bio.PDB.PPBuilder().build_peptides(struct_4LGR.copy())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.6/site-packages/Bio/PDB/Polypeptide.py", line 357, in build_peptides
    and is_connected(prev_res, next_res):
  File "/usr/lib/python2.6/site-packages/Bio/PDB/Polypeptide.py", line 417, in _is_connected
    nlist=n.disordered_get_list()
AttributeError: 'Atom' object has no attribute 'disordered_get_list'
```

Atom object has no attribute atom.disordered_get_list when saving PDB file
### Setup

I am reporting a problem with Biopython version, Python version, and operating
system as follows:

```python
import sys; print(sys.version)
import platform; print(platform.python_implementation()); print(platform.platform())
import Bio; print(Bio.__version__)
```
3.8.5 (default, Aug  5 2020, 08:36:46) 
[GCC 7.3.0]
CPython
Linux-5.4.0-42-generic-x86_64-with-glibc2.10
1.77

### Expected behaviour

`PDBIO.save()` function should save a `Chain` object normally.

### Actual behaviour
The function failed on calling `atom.disordered_get_list()` on some proteins. For example, `pdb/ah/pdb1ah7.ent.gz`, which was directly downloaded from RCSB PDB. 

```bash
Traceback (most recent call last):
  File "test.py", line 13, in <module>
    io.save('test.pdb')
  File "/home/minys/.conda/envs/bio/lib/python3.8/site-packages/Bio/PDB/PDBIO.py", line 339, in save
    for atom in residue.get_unpacked_list():
  File "/home/minys/.conda/envs/bio/lib/python3.8/site-packages/Bio/PDB/Residue.py", line 88, in get_unpacked_list
    undisordered_atom_list += atom.disordered_get_list()
AttributeError: 'Atom' object has no attribute 'disordered_get_list'
```


### Steps to reproduce
To run this piece of code, please download `pdb1ah7.ent.gz` in advance, for simplicity, here PDBList is not used. The task is to save a protein chain into an individual file.

This issue may be related to #455 , which seems not fixed yet.

```python
from Bio.PDB import PDBIO, PDBParser
import gzip

pdb_file = '/data/pdb/ah/pdb1ah7.ent.gz'
pdbparser = PDBParser()
target = pdbparser.get_structure('id', gzip.open(pdb_file, 'rt'))[0]['A']
io = PDBIO()
io.set_structure(target)
io.save('test.pdb')
```



Discussion:
I suspect (from just browsing the code) that the problem is neither `Bio.PDB.Atom.DisorderedAtom` nor the underlying base class `Bio.PDB.Entity.DisorderedEntityWrapper` define a `.copy()` method?

That looks right to me. I'm not sure I completely understand how disorder is handled here, but from my reading `.copy()` should be defined at the `DisorderedAtom` level, and a minimal implimentation might be:

```
def copy(self):
    shallow = copy.copy(self)
    for child in self.child_dict.values():
        shallow.disordered_add(child.copy())
    return shallow
```

This fixes my immediate problem (ie. disordered `Atom` objects in copies are now appropriately contained in `DisorderedAtom`, and `Bio.PDB.PPBuilder` now works). I may have missed something important in the above, and don't know what other behaviour needs to be tested here.

@macraild Thanks for looking at this and coming up with a proposed fix, The very fact you hit this bug tells me we don't (yet) have a suitable test for this. As for running the existing tests, there is a chapter in the main tutorial that covers this if you are interested.

@etal, @lennax, @JoaoRodrigues - are any of you familiar with disordered atoms, or know who would be best to review this proposed fix? Thanks!

Hi,

Had a look at the code. Seems to be exactly the problem. The fix seems to work rather well. 

Just at thought, wouldn't be better to add it to the DisorderedEntityWrapper object? Seems cleaner. This will add a copy method to DisorderedResidues as well, although it doesn't seem to affect anything (tested with PDB 1IGY).

Cheers

@JoaoRodrigues On a closer look, I agree. This does look like it belongs in `DisorderedEntityWrapper`. Except that then the existing tests break:

```
> python test_PDB.py
...
======================================================================
FAIL: Make a copy of a residue.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_PDB.py", line 1058, in test_entitity_copy
    ee = e.copy()
  File "/usr/lib/python2.6/site-packages/Bio/PDB/Entity.py", line 182, in copy
    shallow.add(child.copy())
  File "/usr/lib/python2.6/site-packages/Bio/PDB/Entity.py", line 182, in copy
    shallow.add(child.copy())
  File "/usr/lib/python2.6/site-packages/Bio/PDB/Entity.py", line 182, in copy
    shallow.add(child.copy())
  File "/usr/lib/python2.6/site-packages/Bio/PDB/Entity.py", line 243, in copy
    shallow.disordered_add(child.copy())
  File "/usr/lib/python2.6/site-packages/Bio/PDB/Residue.py", line 160, in disordered_add
    assert(not self.disordered_has_id(resname))
AssertionError

----------------------------------------------------------------------
Ran 35 tests in 1.862s

FAILED (failures=1)
```

Looks like `DisorderedResidues` need different treatment?

One other issue I've run into:
on Python2.6 the proposed fix (in either location) exposes something similar to this bug: http://bugs.python.org/issue5508 Everything still appears to work, but the error message is a bit disconcerting.

Check this commit here: a36605d2aafacb3f47de42844899b263c8c8de8b

The tests pass, but indeed that recursion warning shows up.. @peterjc Any ideas?

So adding this `.copy()` to `DisorderedEntityWrapper` interacts badly with the existing `.__getattr__(...)` method? Strange.

If you add the `.copy()` method to `DisorderedAtom` does everything seem OK?

Hi, 

   First of all I want to thank you for the great job you are doing with biopython.

  In connection with the problem exposed here.
When copying a structure, applying a transformation, then saving : I first ran in the problem exposed by macraild. Using the "trick" he proposed on jan 8 2015 I was able to have my script working without complaining, but I realized afterwards that the atoms with alternate locations were not affected by the transformation : they simply stayed in place.

Any idea how I can solve the problem ?

(using pytho2.7 on ubuntu, biopython version as found in **init**.py : 1.66)

Hi,

As a quick and dirty fix, in Bio.PDB.Residue.Residue.get_unpacked_list I replaced 

```
    if atom.is_disordered():
        undisordered_atom_list = (undisordered_atom_list + atom.disordered_get_list())
```

by

```
    if atom.is_disordered():
        atom.altloc=" "
        undisordered_atom_list.append(atom)
```

It seems this allows me to keep atoms for alternate location 'A'.

I have run into the exact same issue as schwarz-ben.

When copying a structure and then applying a transformation to the copied structure, the transformation is not applied to the disordered atoms.

Any chance of getting this fixed relatively quickly? Seems like a pretty major bug which should have a simply fix...
Hi @jaredsagendorf 

Could you post here a simple script that reproduces this issue? I can try to fix this asap.
I think there are actually two problems colliding : copying a structure with alt-loc atoms, and transforming a structure with alt-loc atoms.

The sample code below should do.
 * Run as is it should crash (can't copy disordered atoms)
 * Uncommenting FIX1 (as suggested above in the thread) the script should execute, but the job is imperfectly done (alt-loc atoms aren't moved, neither in the loaded structure nor in its copy.).
 * Uncommenting FIX2 is ugly (I transform disordered in ordered, keeping only data from altloc 'A') but does the job

```python
import Bio.PDB

############# FIX 1

# copy=Bio.PDB.Atom.copy
# def myCopy(self):
#     shallow = copy.copy(self)
#     for child in self.child_dict.values():
#         shallow.disordered_add(child.copy())
#     return shallow
# Bio.PDB.Atom.DisorderedAtom.copy=myCopy

############# FIX 2

# def get_unpacked_list(self):
#     """
#     Returns all atoms from the residue,
#     in case of disordered, keep only first alt loc and remove the alt-loc tag
#     """
#     atom_list = self.get_list()
#     undisordered_atom_list = []
#     for atom in atom_list:
#         if atom.is_disordered():
#             atom.altloc=" "
#             undisordered_atom_list.append(atom)
#         else:
#             undisordered_atom_list.append(atom)
#     return undisordered_atom_list
# Bio.PDB.Residue.Residue.get_unpacked_list = get_unpacked_list

###########################

inputPdb = "/home/schwarz/tmp/4lgr.pdb"
output1  = "/home/schwarz/tmp/4lgr-t1.pdb"
output2  = "/home/schwarz/tmp/4lgr-t2.pdb"

parser = Bio.PDB.PDBParser()
struct = parser.get_structure("myStruct",inputPdb)
idMat=( (1,0,0), (0,1,0), (0,0,1) )
trVect=(20.,20.,20.)
s = struct.copy()
s.transform( idMat,trVect )
struct.transform( idMat,trVect )
pdbio = Bio.PDB.PDBIO()

pdbio.set_structure(struct)
with open(output1,"w") as fd :
  pdbio.save(fd)

pdbio.set_structure(s)
with open(output2,"w") as fd :
  pdbio.save(fd)
```
Adding these lines  as suggested in this thread should allow the script t
Just reporting that I ran into the same issue while using the `SuperImposer()` class and that @schwarz-ben's solution above resolved the issue. Eagerly awaiting that PR :D
> Hi,
> 
> As a quick and dirty fix, in Bio.PDB.Residue.Residue.get_unpacked_list I replaced
> 
> ```
>     if atom.is_disordered():
>         undisordered_atom_list = (undisordered_atom_list + atom.disordered_get_list())
> ```
> 
> by
> 
> ```
>     if atom.is_disordered():
>         atom.altloc=" "
>         undisordered_atom_list.append(atom)
> ```
> 
> It seems this allows me to keep atoms for alternate location 'A'.

Hi, I am very new in using Biopython. I had the same issue and did this bad thing as @schwarz-ben  suggested. And it did work. 

@JoaoRodrigues are you able to look at this?
Will do.
Will write here as I work through the issue. Hopefully we can close this and other related issues once and for all...

1. The issue here is that when PDBIO rebuilds the full SMCRA model (we're only giving it a chain here), it loses the DisorderedAtom objects. They become simple Atom objects. This happens on `Entity.copy`. 

2. `DisorderedAtom`/`DisorderedEntityWrapper` do not implement a copy method. The call to `copy()` gets routed to the selected child object.

3. Adding a `copy` method to `DisorderedEntityWrapper` lets us move a bit forward but it fails when we are copying a partial structure (e.g. a single chain) because they have no parent. Error is:
     ```python
        return self.parent.get_full_id() + ((self.name, self.altloc),)
        AttributeError: 'NoneType' object has no attribute 'get_full_id'
    ````
    The same problem happens when doing `chain.copy()` on the isolated chain object (i.e. not strictly linked to PDBIO). Trying to figure out what would be the best way to fix this in the long term. 

4. For some reason, `Entity.get_full_id` called `_reset_full_id` when I think it should be calling `_generate_full_id`. Resetting the id is useful only when _changing_ an existing id. Making this small change allows copying the isolated chain but still does not fix the issue with the disordered atoms (two problems).

5. Adding a try/except statement to `Atom.get_full_id` to catch when `Atom.parent is None` solves the PDBIO problem. We could simply mock the full id as `(None, None, None, None,) + atom.name + atom.altloc)`?

These changes fix issues #455 and #2134.

Resolving patch:
diff --git a/Bio/PDB/Atom.py b/Bio/PDB/Atom.py
--- a/Bio/PDB/Atom.py
+++ b/Bio/PDB/Atom.py
@@ -369,7 +369,10 @@ def get_full_id(self):
         the atom and consists of the following elements:
         (structure id, model id, chain id, residue id, atom name, altloc)
         """
-        return self.parent.get_full_id() + ((self.name, self.altloc),)
+        try:
+            return self.parent.get_full_id() + ((self.name, self.altloc),)
+        except AttributeError:
+            return (None, None, None, None, self.name, self.altloc)
 
     def get_coord(self):
         """Return atomic coordinates."""
@@ -492,3 +495,11 @@ def disordered_add(self, atom):
         if occupancy > self.last_occupancy:
             self.last_occupancy = occupancy
             self.disordered_select(altloc)
+
+    def transform(self, rot, tran):
+        """Apply rotation and translation to all children.
+
+        See the documentation of Atom.transform for details.
+        """
+        for child in self:
+            child.coord = numpy.dot(child.coord, rot) + tran
diff --git a/Bio/PDB/Entity.py b/Bio/PDB/Entity.py
--- a/Bio/PDB/Entity.py
+++ b/Bio/PDB/Entity.py
@@ -266,7 +266,7 @@ def get_full_id(self):
         identifier is 10 and its insertion code "A".
         """
         if self.full_id is None:
-            self._reset_full_id()
+            self.full_id = self._generate_full_id()
         return self.full_id
 
     def transform(self, rot, tran):
@@ -384,6 +384,16 @@ def __le__(self, other):
         return self.selected_child <= other
 
     # Public methods
+    def copy(self):
+        """Copy disorderd entity recursively."""
+        shallow = copy(self)
+        shallow.child_dict = {}
+        shallow.detach_parent()
+
+        for child in self.disordered_get_list():
+            shallow.disordered_add(child.copy())
+
+        return shallow
 
     def get_id(self):
         """Return the id."""


Test patch:
diff --git a/Tests/PDB/disordered.pdb b/Tests/PDB/disordered.pdb
new file mode 100644
--- /dev/null
+++ b/Tests/PDB/disordered.pdb
@@ -0,0 +1,52 @@
+REMARK  EXTRACTED FROM 1AH7 FOR TESTS WITH DISORDERED ATOMS
+ATOM    200  N   MET A  25      51.748  19.748  27.301  1.00 10.81           N
+ATOM    201  CA  MET A  25      51.216  19.092  28.509  1.00 10.64           C
+ATOM    202  C   MET A  25      52.302  18.224  29.134  1.00 11.58           C
+ATOM    203  O   MET A  25      52.403  18.241  30.364  1.00 12.69           O
+ATOM    204  CB  MET A  25      50.007  18.248  28.112  1.00 12.80           C
+ATOM    205  CG  MET A  25      48.825  19.122  27.718  1.00 13.45           C
+ATOM    206  SD  MET A  25      47.414  18.129  27.196  1.00 15.65           S
+ATOM    207  CE  MET A  25      46.853  17.444  28.751  1.00 17.96           C
+ATOM    208  N   SER A  26      53.051  17.446  28.356  1.00 11.14           N
+ATOM    209  CA  SER A  26      54.101  16.601  28.942  1.00 12.62           C
+ATOM    210  C   SER A  26      55.196  17.395  29.604  1.00 14.35           C
+ATOM    211  O   SER A  26      55.801  16.923  30.596  1.00 16.09           O
+ATOM    212  CB  SER A  26      54.725  15.775  27.794  1.00 14.25           C
+ATOM    213  OG  SER A  26      53.755  14.801  27.474  1.00 16.42           O
+ATOM    214  N   ARG A  27      55.495  18.574  29.081  1.00 12.79           N
+ATOM    215  CA  ARG A  27      56.559  19.410  29.649  1.00 15.18           C
+ATOM    216  C   ARG A  27      56.138  20.097  30.930  1.00 14.66           C
+ATOM    217  O   ARG A  27      56.994  20.479  31.741  1.00 17.28           O
+ATOM    218  CB  ARG A  27      56.899  20.533  28.660  1.00 16.52           C
+ATOM    219  CG  ARG A  27      57.901  20.081  27.621  1.00 19.45           C
+ATOM    220  CD  ARG A  27      58.541  21.359  27.019  1.00 22.58           C
+ATOM    221  NE  ARG A  27      59.504  20.850  26.023  1.00 26.89           N
+ATOM    222  CZ AARG A  27      59.081  20.674  24.762  0.50 26.69           C
+ATOM    223  CZ BARG A  27      60.798  20.732  26.326  0.50 26.95           C
+ATOM    224  NH1AARG A  27      57.848  21.002  24.386  0.50 27.16           N
+ATOM    225  NH1BARG A  27      61.262  21.064  27.522  0.50 27.39           N
+ATOM    226  NH2AARG A  27      59.908  20.188  23.844  0.50 28.61           N
+ATOM    227  NH2BARG A  27      61.641  20.275  25.411  0.50 27.94           N
+ATOM    228  N   ASN A  28      54.843  20.267  31.133  1.00 12.66           N
+ATOM    229  CA  ASN A  28      54.332  20.911  32.332  1.00 13.60           C
+ATOM    230  C   ASN A  28      54.457  19.996  33.538  1.00 15.16           C
+ATOM    231  O   ASN A  28      53.973  18.862  33.516  1.00 17.14           O
+ATOM    232  CB  ASN A  28      52.885  21.353  32.083  1.00 13.71           C
+ATOM    233  CG  ASN A  28      52.425  22.208  33.243  1.00 16.04           C
+ATOM    234  OD1 ASN A  28      52.058  21.613  34.258  1.00 17.57           O
+ATOM    235  ND2 ASN A  28      52.438  23.528  33.110  1.00 14.80           N
+ATOM    236  N   THR A  29      55.169  20.442  34.584  1.00 17.94           N
+ATOM    237  CA  THR A  29      55.322  19.543  35.740  1.00 20.37           C
+ATOM    238  C   THR A  29      54.668  20.102  36.988  1.00 22.43           C
+ATOM    239  O   THR A  29      54.832  19.550  38.091  1.00 24.72           O
+ATOM    240  CB  THR A  29      56.815  19.293  36.034  1.00 21.10           C
+ATOM    241  OG1 THR A  29      57.412  20.577  36.259  1.00 23.66           O
+ATOM    242  CG2 THR A  29      57.493  18.580  34.878  1.00 21.88           C
+ATOM    243  N   THR A  30      53.932  21.184  36.847  1.00 22.37           N
+ATOM    244  CA  THR A  30      53.262  21.815  37.976  1.00 24.23           C
+ATOM    245  C   THR A  30      51.769  21.563  38.003  1.00 24.52           C
+ATOM    246  O   THR A  30      51.174  21.357  39.079  1.00 28.23           O
+ATOM    247  CB  THR A  30      53.532  23.335  37.953  1.00 24.92           C
+ATOM    248  OG1 THR A  30      53.011  23.864  36.731  1.00 26.49           O
+ATOM    249  CG2 THR A  30      55.027  23.636  37.979  1.00 25.70           C
+END
diff --git a/Tests/test_PDB_Disordered.py b/Tests/test_PDB_Disordered.py
new file mode 100644
--- /dev/null
+++ b/Tests/test_PDB_Disordered.py
@@ -0,0 +1,110 @@
+# Revisions copyright 2020 Joao Rodrigues. All rights reserved.
+#
+# This file is part of the Biopython distribution and governed by your
+# choice of the "Biopython License Agreement" or the "BSD 3-Clause License".
+# Please see the LICENSE file that should have been included as part of this
+# package.
+
+"""Unit tests for disordered atoms in the Bio.PDB module."""
+
+import os
+import tempfile
+import unittest
+
+from Bio.PDB import PDBParser, PDBIO
+
+
+class TestDisordered(unittest.TestCase):
+    """Tests for operations on DisorderedEntities."""
+
+    def unpack_all_atoms(self, structure):
+        """Return a list of all atoms in the structure."""
+        return [a for r in structure.get_residues() for a in r.get_unpacked_list()]
+
+    def test_copy_disordered_atom(self):
+        """Copies disordered atoms and all their children."""
+        parser = PDBParser(QUIET=1)
+        s = parser.get_structure("x", "PDB/disordered.pdb")
+
+        resi27 = s[0]["A"][27]
+        resi27_copy = resi27.copy()
+
+        self.assertNotEqual(id(resi27), id(resi27_copy))  # did we really copy
+
+        resi27_atoms = resi27.get_unpacked_list()
+        resi27_copy_atoms = resi27.get_unpacked_list()
+        self.assertEqual(len(resi27_atoms), len(resi27_copy_atoms))
+
+        for ai, aj in zip(resi27_atoms, resi27_copy_atoms):
+            self.assertEqual(ai.name, aj.name)
+
+    def test_copy_entire_chain(self):
+        """Copy propagates throughout SMCRA object."""
+        parser = PDBParser(QUIET=1)
+        s = parser.get_structure("x", "PDB/disordered.pdb")
+
+        s_copy = s.copy()
+
+        self.assertNotEqual(id(s), id(s_copy))  # did we really copy
+
+        atoms = self.unpack_all_atoms(s)
+        copy_atoms = self.unpack_all_atoms(s_copy)
+        self.assertEqual(len(atoms), len(copy_atoms))
+
+        for ai, aj in zip(atoms, copy_atoms):
+            self.assertEqual(ai.name, aj.name)
+
+    def test_transform_disordered(self):
+        """Transform propagates through disordered atoms."""
+        # This test relates to issue #455 where applying a transformation
+        # to a copied structure did not work for disordered atoms.
+        parser = PDBParser(QUIET=1)
+        s = parser.get_structure("x", "PDB/disordered.pdb")
+
+        s_copy = s.copy()
+
+        mtx = ((1, 0, 0), (0, 1, 0), (0, 0, 1))
+        tr_vec = (20.0, 0.0, 0.0)
+
+        s_copy.transform(mtx, tr_vec)  # transform copy
+
+        atoms = self.unpack_all_atoms(s)
+        copy_atoms = self.unpack_all_atoms(s_copy)
+        self.assertEqual(len(atoms), len(copy_atoms))
+        for ai, aj in zip(atoms, copy_atoms):
+            self.assertEqual(ai - aj, 20.0)  # check distance == 20.0
+
+    def test_copy_and_write_disordered(self):
+        """Extract, save, and parse again disordered atoms."""
+        parser = PDBParser(QUIET=1)
+        writer = PDBIO()
+
+        s = parser.get_structure("x", "PDB/disordered.pdb")
+
+        # Extract the chain object
+        chain = s[0]["A"]
+
+        writer.set_structure(chain)
+
+        filenumber, filename = tempfile.mkstemp()  # save to temp file
+        os.close(filenumber)
+        try:
+            writer.save(filename)
+
+            # Parse again
+            s2 = parser.get_structure("x_copy", filename)
+
+            # Do we have the same stuff?
+            atoms1 = self.unpack_all_atoms(s)
+            atoms2 = self.unpack_all_atoms(s2)
+            self.assertEqual(len(atoms1), len(atoms2))
+            for ai, aj in zip(atoms1, atoms2):
+                self.assertEqual(ai.name, aj.name)
+
+        finally:
+            os.remove(filename)
+
+
+if __name__ == "__main__":
+    runner = unittest.TextTestRunner(verbosity=2)
+    unittest.main(testRunner=runner)