Repository: biopython/biopython

Issue:
SeqIO OOM parse issue
### Setup

I am reporting a problem with Biopython version, Python version, and operating
system as follows:

```python
import sys; print(sys.version)
import platform; print(platform.python_implementation()); print(platform.platform())
import Bio; print(Bio.__version__)
```

3.8.8 (default, Apr 13 2021, 15:08:03) [MSC v.1916 64 bit (AMD64)]
CPython
Windows-10-10.0.19041-SP0
1.81

### Expected behaviour

I am using SeqIO.parse to load in Uniref50 XML (240gb) record by record to build an annotation set. Even though it is supposed to load one record at a time it is giving me OOM errors and crashing with 1.8Tb ram.

### Actual behaviour

Works fine for 8 gb file (Sprot XML)

```python
file_path = './uniref50.xml'
total=62149502
pubmed_pattern = re.compile(r'\((Ref\.\d+, )?(PubMed:\d+)(, Ref\.\d+)?(, PubMed:\d+)*\)')
seqs, ids, texts = [], [], []

with open(file_path, 'r') as handle:
    print('This prints')
    for record in SeqIO.parse(handle, 'uniprot-xml'):
        print('This does not print before OOM with 1.8Tb ram')
        annotations = record.annotations
        intersection = set(annotations.keys()) & set_keys
        seq_len = annotations.get('sequence_length', 0)
    
        if len(intersection) >= 5 and 50 < seq_len < 512:
            text = build_text(intersection, annotations, int_keys, regex_keys, text_keys, pubmed_pattern)
            seqs.append(str(record.seq))
            ids.append(record.id)
            texts.append(text)
        else:
            continue
```

### Steps to reproduce

Run the following code with uniref50 XML. The necessary dictionaries and functions can be fillers, they are not called when using the large files.



Discussion:
Do you have the URL for those test files handy?
Sure
Small
https://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot_sprot.xml.gz
Big
https://ftp.uniprot.org/pub/databases/uniprot/uniref/uniref50/uniref50.xml.gz
Thanks. I'll download a copy...

As an aside, you should be able to parse them compressed using ``gzip.open(filename)`` - see examples in the docs.
Thanks!

Opening with gzip does not seem to remedy the massive memory requirements.
I tried to reproduce this problem to see if I could help.  My system is similar but when I  run the sequence read part of the above loop using gzip on the smaller test file provided it seems to work with minimal memory overhead.

Specifically my system is:
```python
import sys;
import platform
import Bio 
sys.version, platform.python_implementation(), platform.platform(), Bio.__version__
```
('3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)]',
 'CPython',
 'Windows-10-10.0.19045-SP0',
 '1.81')

The reduced code I ran was:
```python
from Bio import SeqIO
import gzip
import re
import time
file_path = '..../uniprot_sprot.xml.gz'
total=62149502
pubmed_pattern = re.compile(r'\((Ref\.\d+, )?(PubMed:\d+)(, Ref\.\d+)?(, PubMed:\d+)*\)')
seqs, ids, texts = [], [], []

start_time = time.time()
last_time = start_time
new_time = start_time
sample_count = 10000
with gzip.open(file_path, 'r') as handle:
    print('This prints')
    for (i,record) in enumerate(SeqIO.parse(handle, 'uniprot-xml')):
        if i % sample_count == 0:
            print('---------------')            
            new_time = time.time()
            print(f'Read {i} records in time {new_time - start_time}')
            last_time = new_time
            print(record.seq)
```
which gave output below:

```
This prints
---------------
Read 0 records in time 0.0
MVRLFYNPIKYLFYRRSCKKRLRKALKKLNFYHPPKECCQIYRLLENAPGGTYFITENMTNELIMIAKDPVDKKIKSVKLYLTGNYIKINQHYYINIYMYLMRYNQIYKYPLICFSKYSKIL
---------------
Read 10000 records in time 13.858141660690308
MAEFKEQVLDILEEVCENDIVKENLDVQLFEEGILDSFAVVSLLVEFQERLDIEVSISDFDRDEWATPNMVIKKLEEIR
---------------
Read 20000 records in time 26.40064287185669
MHFESPIKRRHSRQIMVGNVPVGGGAPISVQSMTNTETCDVDATVAQIRAIADAGADIVRVSIPSMDAAEAFKKIREAVSIPLVADIHFDYKIALKVAEYGVDCLRINPGNIGNKDRIRAVVDCARDKNIPIRIGVNAGSLEKDLQKKYGEPTPDALVESAFRQIQYFDELDFHEYKLSLKASDIFMTVEAYRKIASQIDNPLHLGITEAGGLRSGTVKSSIGLGLLLMDGIGDTLRVSLAADPVHEIKVGWDMLRSLKLRNRGINFIACPSCSRQNFDVIKTMNELEERLDDITIPMDVAVIGCVVNGPGEAKEADIGLAGGSPNNLIYQDGKPDSKTKNETLVDDLERMIRKKALLKEQELANIIVKN
.....
```
Total run time about 12 minutes and process memory never went over 125MB.  

Are you able to run this simplified code?  Is it possible that the code you have after the read is leading to the memory usage? Maybe the first print in your loop is being buffered.
Oh thanks @lurkingviolet - your analysis is in line with what I would have expected.

Checking now downloading ``uniprot_sprot.xml.gz`` was fine, but my download of the larger ``uniref50.xml.gz`` didn't finish.

As a one-liner for testing, based on ``top`` memory usage was about 100MB:

```
$ time python -c "import gzip; from Bio import SeqIO; print(sum(1 for rec in SeqIO.parse(gzip.open('uniprot_sprot.xml.gz'), 'uniprot-xml')))"
570157

real	9m47.006s
user	9m30.833s
sys	0m0.479s
```
Looking back at the original example, if the record passes certain criteria, it appends data to the three lists ``seqs``, ``ids``, and ``texts``.

It is most likely that itself causes the excessive memory usage. Perhaps you are capturing more records than you expected?

I would suggest perhaps recording just the ids (for later analysis separately, perhaps using ``Bio.SeqIO.index_db(...)`` to look up the records), and/or recording this data to and output file instead?
Thanks @peterjc and @lurkingviolet for the helpful suggestions. I will try the simplified version on the large file and report back.
After an hour this
```
from Bio import SeqIO
import gzip
import re
import time
file_path = './uniprot_sprot.xml.gz'
total=62149502
pubmed_pattern = re.compile(r'\((Ref\.\d+, )?(PubMed:\d+)(, Ref\.\d+)?(, PubMed:\d+)*\)')
seqs, ids, texts = [], [], []

start_time = time.time()
last_time = start_time
new_time = start_time
sample_count = 10000
with gzip.open(file_path, 'r') as handle:
    print('This prints')
    for (i,record) in enumerate(SeqIO.parse(handle, 'uniprot-xml')):
        if i % sample_count == 0:
            print('---------------')            
            new_time = time.time()
            print(f'Read {i} records in time {new_time - start_time}')
            last_time = new_time
            print(record.seq)
```
gives

`This prints`

The smaller file immediately yields

```
This prints
---------------
Read 0 records in time 0.35556721687316895
MVRLFYNPIKYLFYRRSCKKRLRKALKKLNFYHPPKECCQIYRLLENAPGGTYFITENMTNELIMIAKDPVDKKIKSVKLYLTGNYIKINQHYYINIYMYLMRYNQIYKYPLICFSKYSKIL
```
As you guys found. Really confused why this is the case. I think I will try redownloading the file to troubleshoot if it got corrupted.


EDIT: Redownloading did not fix the issue.
Could it be that the schema is different? They list xml-schema with these Uniref downloads but not with the other Uniprot ones.

![image](https://github.com/biopython/biopython/assets/72926928/dd1a0a4b-8d08-4a80-90f8-1b8268480411)

I also am having trouble downloading the uniref file, but it does seems possible that xml schema is different and that is causing the parse to interpret the entire file as one record?  Can you run snippet like below on both files and see what you get?
```python
from xml.etree import ElementTree
import gzip
with gzip.open(file_path, 'r') as handle:
    for (i, node) in enumerate(ElementTree.iterparse(handle)):
        print(node)
        if i > 50:
            break
```
Here's a link to uniref50 xml though my google drive, perhaps the download is more stable.
https://drive.google.com/file/d/1d6G9oIjKdC-sHcBNEyhSbdazD_w9VZNq/view?usp=sharing

I ran the xml.etree code as requested
Sprot
('end', <Element '{[http://uniprot.org/uniprot}accession](http://uniprot.org/uniprot%7Daccession)' at 0x7b3d0073a7a0>)
('end', <Element '{[http://uniprot.org/uniprot}name](http://uniprot.org/uniprot%7Dname)' at 0x7b3d0073bf10>)
('end', <Element '{[http://uniprot.org/uniprot}fullName](http://uniprot.org/uniprot%7DfullName)' at 0x7b3d0073bd30>)
('end', <Element '{[http://uniprot.org/uniprot}recommendedName](http://uniprot.org/uniprot%7DrecommendedName)' at 0x7b3d0073be20>)
('end', <Element '{[http://uniprot.org/uniprot}protein](http://uniprot.org/uniprot%7Dprotein)' at 0x7b3d0073bec0>)
('end', <Element '{[http://uniprot.org/uniprot}name](http://uniprot.org/uniprot%7Dname)' at 0x7b3d0073bc90>)
('end', <Element '{[http://uniprot.org/uniprot}gene](http://uniprot.org/uniprot%7Dgene)' at 0x7b3d0073bba0>)
('end', <Element '{[http://uniprot.org/uniprot}name](http://uniprot.org/uniprot%7Dname)' at 0x7b3d0073ba10>)
('end', <Element '{[http://uniprot.org/uniprot}name](http://uniprot.org/uniprot%7Dname)' at 0x7b3d0073b7e0>)
('end', <Element '{[http://uniprot.org/uniprot}dbReference](http://uniprot.org/uniprot%7DdbReference)' at 0x7b3d0073b880>)
('end', <Element '{[http://uniprot.org/uniprot}taxon](http://uniprot.org/uniprot%7Dtaxon)' at 0x7b3d0073b740>)
('end', <Element '{[http://uniprot.org/uniprot}taxon](http://uniprot.org/uniprot%7Dtaxon)' at 0x7b3d0073b6a0>)
('end', <Element '{[http://uniprot.org/uniprot}taxon](http://uniprot.org/uniprot%7Dtaxon)' at 0x7b3d0073b6f0>)
('end', <Element '{[http://uniprot.org/uniprot}taxon](http://uniprot.org/uniprot%7Dtaxon)' at 0x7b3d0073b4c0>)
('end', <Element '{[http://uniprot.org/uniprot}taxon](http://uniprot.org/uniprot%7Dtaxon)' at 0x7b3d0073b5b0>)
('end', <Element '{[http://uniprot.org/uniprot}taxon](http://uniprot.org/uniprot%7Dtaxon)' at 0x7b3d0073b510>)
('end', <Element '{[http://uniprot.org/uniprot}taxon](http://uniprot.org/uniprot%7Dtaxon)' at 0x7b3d0073b560>)
('end', <Element '{[http://uniprot.org/uniprot}taxon](http://uniprot.org/uniprot%7Dtaxon)' at 0x7b3d0073b330>)
('end', <Element '{[http://uniprot.org/uniprot}lineage](http://uniprot.org/uniprot%7Dlineage)' at 0x7b3d0073b650>)
('end', <Element '{[http://uniprot.org/uniprot}organism](http://uniprot.org/uniprot%7Dorganism)' at 0x7b3d0073bb00>)
('end', <Element '{[http://uniprot.org/uniprot}name](http://uniprot.org/uniprot%7Dname)' at 0x7b3d0073b380>)
('end', <Element '{[http://uniprot.org/uniprot}name](http://uniprot.org/uniprot%7Dname)' at 0x7b3d0073b3d0>)
('end', <Element '{[http://uniprot.org/uniprot}dbReference](http://uniprot.org/uniprot%7DdbReference)' at 0x7b3d0073b2e0>)
('end', <Element '{[http://uniprot.org/uniprot}organismHost](http://uniprot.org/uniprot%7DorganismHost)' at 0x7b3d0073b420>)
('end', <Element '{[http://uniprot.org/uniprot}name](http://uniprot.org/uniprot%7Dname)' at 0x7b3d0073b290>)
('end', <Element '{[http://uniprot.org/uniprot}name](http://uniprot.org/uniprot%7Dname)' at 0x7b3d0073ac00>)
('end', <Element '{[http://uniprot.org/uniprot}dbReference](http://uniprot.org/uniprot%7DdbReference)' at 0x7b3d0073ade0>)
('end', <Element '{[http://uniprot.org/uniprot}organismHost](http://uniprot.org/uniprot%7DorganismHost)' at 0x7b3d0073b240>)
('end', <Element '{[http://uniprot.org/uniprot}name](http://uniprot.org/uniprot%7Dname)' at 0x7b3d0073b0b0>)
('end', <Element '{[http://uniprot.org/uniprot}name](http://uniprot.org/uniprot%7Dname)' at 0x7b3d0073b010>)
('end', <Element '{[http://uniprot.org/uniprot}dbReference](http://uniprot.org/uniprot%7DdbReference)' at 0x7b3d0073af70>)
('end', <Element '{[http://uniprot.org/uniprot}organismHost](http://uniprot.org/uniprot%7DorganismHost)' at 0x7b3d0073b100>)
('end', <Element '{[http://uniprot.org/uniprot}name](http://uniprot.org/uniprot%7Dname)' at 0x7b3d0073af20>)
('end', <Element '{[http://uniprot.org/uniprot}name](http://uniprot.org/uniprot%7Dname)' at 0x7b3d0073ae80>)
('end', <Element '{[http://uniprot.org/uniprot}dbReference](http://uniprot.org/uniprot%7DdbReference)' at 0x7b3d0073ad40>)
('end', <Element '{[http://uniprot.org/uniprot}organismHost](http://uniprot.org/uniprot%7DorganismHost)' at 0x7b3d0073afc0>)
('end', <Element '{[http://uniprot.org/uniprot}name](http://uniprot.org/uniprot%7Dname)' at 0x7b3d0073ac50>)
('end', <Element '{[http://uniprot.org/uniprot}name](http://uniprot.org/uniprot%7Dname)' at 0x7b3d0073ab10>)
('end', <Element '{[http://uniprot.org/uniprot}dbReference](http://uniprot.org/uniprot%7DdbReference)' at 0x7b3d0073ab60>)
('end', <Element '{[http://uniprot.org/uniprot}organismHost](http://uniprot.org/uniprot%7DorganismHost)' at 0x7b3d0073aca0>)
('end', <Element '{[http://uniprot.org/uniprot}title](http://uniprot.org/uniprot%7Dtitle)' at 0x7b3d0073a520>)
('end', <Element '{[http://uniprot.org/uniprot}person](http://uniprot.org/uniprot%7Dperson)' at 0x7b3d0073a430>)
('end', <Element '{[http://uniprot.org/uniprot}person](http://uniprot.org/uniprot%7Dperson)' at 0x7b3d0073a480>)
('end', <Element '{[http://uniprot.org/uniprot}authorList](http://uniprot.org/uniprot%7DauthorList)' at 0x7b3d0073a390>)
('end', <Element '{[http://uniprot.org/uniprot}citation](http://uniprot.org/uniprot%7Dcitation)' at 0x7b3d0073a890>)
('end', <Element '{[http://uniprot.org/uniprot}scope](http://uniprot.org/uniprot%7Dscope)' at 0x7b3d0073a5c0>)
('end', <Element '{[http://uniprot.org/uniprot}reference](http://uniprot.org/uniprot%7Dreference)' at 0x7b3d0073a9d0>)
('end', <Element '{[http://uniprot.org/uniprot}text](http://uniprot.org/uniprot%7Dtext)' at 0x7b3d007941d0>)
('end', <Element '{[http://uniprot.org/uniprot}comment](http://uniprot.org/uniprot%7Dcomment)' at 0x7b3d00794270>)
('end', <Element '{[http://uniprot.org/uniprot}text](http://uniprot.org/uniprot%7Dtext)' at 0x7b3d00794310>)
('end', <Element '{[http://uniprot.org/uniprot}comment](http://uniprot.org/uniprot%7Dcomment)' at 0x7b3d007942c0>)
('end', <Element '{[http://uniprot.org/uniprot}property](http://uniprot.org/uniprot%7Dproperty)' at 0x7b3d00794450>)

Uniref
('end', <Element '{[http://uniprot.org/uniref}name](http://uniprot.org/uniref%7Dname)' at 0x7b3d0076b510>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076b5b0>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076b650>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076b6a0>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076b740>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076b7e0>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076b880>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076b920>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076b9c0>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076ba60>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076bc90>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076bce0>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076bd80>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076be20>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076bec0>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076bf60>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076bfb0>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d00794040>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d00794090>)
('end', <Element '{[http://uniprot.org/uniref}dbReference](http://uniprot.org/uniref%7DdbReference)' at 0x7b3d0076bbf0>)
('end', <Element '{[http://uniprot.org/uniref}sequence](http://uniprot.org/uniref%7Dsequence)' at 0x7b3d00794180>)
('end', <Element '{[http://uniprot.org/uniref}representativeMember](http://uniprot.org/uniref%7DrepresentativeMember)' at 0x7b3d0076bb00>)
('end', <Element '{[http://uniprot.org/uniref}entry](http://uniprot.org/uniref%7Dentry)' at 0x7b3d0076b330>)
('end', <Element '{[http://uniprot.org/uniref}name](http://uniprot.org/uniref%7Dname)' at 0x7b3d0076a430>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076a3e0>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076a390>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076a340>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076a200>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076a1b0>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076a110>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d0076a020>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d00769f30>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d00769ee0>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d00769e90>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d00769e40>)
('end', <Element '{[http://uniprot.org/uniref}dbReference](http://uniprot.org/uniref%7DdbReference)' at 0x7b3d0076a2a0>)
('end', <Element '{[http://uniprot.org/uniref}sequence](http://uniprot.org/uniref%7Dsequence)' at 0x7b3d00769df0>)
('end', <Element '{[http://uniprot.org/uniref}representativeMember](http://uniprot.org/uniref%7DrepresentativeMember)' at 0x7b3d0076a2f0>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d00769da0>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d00769c10>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d00769b20>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d00769940>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d00769a30>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d007699e0>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d00769990>)
('end', <Element '{[http://uniprot.org/uniref}dbReference](http://uniprot.org/uniref%7DdbReference)' at 0x7b3d00769d50>)
('end', <Element '{[http://uniprot.org/uniref}member](http://uniprot.org/uniref%7Dmember)' at 0x7b3d00769cb0>)
('end', <Element '{[http://uniprot.org/uniref}entry](http://uniprot.org/uniref%7Dentry)' at 0x7b3d00768fe0>)
('end', <Element '{[http://uniprot.org/uniref}name](http://uniprot.org/uniref%7Dname)' at 0x7b3d00769850>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d007698a0>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d00769300>)
('end', <Element '{[http://uniprot.org/uniref}property](http://uniprot.org/uniref%7Dproperty)' at 0x7b3d00769490>)

This is the first time I've gotten an output from it! The organization seems different for sure. So I guess the issue is the schema that Biopython is using in 'uniprot-xml'?
It does sound like the bigger file may be a different XML schema, essentially a different file format to what the parser expects. My download attempt failed and I had not retried it.
My Uniprot contact provided me with this code that works
https://gist.github.com/sminot/365ee8edf4f71014b98cdb18494fb8a4
Thanks for the support! It would be great if Biopython added a uniref schema :)
Does this alternative UniProt XML format have a name and/or human readable description?
@lhallee , glad that you found a solution that worked for you.

@peterjc Below is my current best guess at what happened:

1. uniprot is xml file format for protein records with this [xml schema](https://ftp.uniprot.org/pub/databases/uniprot/current_release/knowledgebase/complete/uniprot.xsd) and looking at this [example](https://rest.uniprot.org/uniprotkb/P12345.xml) has a lot of information about the protein.
2. uniref is a different xml file format that defines clusters of gene related proteins with this [xml schema](https://ftp.uniprot.org/pub/databases/uniprot/current_release/uniref/uniref50/uniref.xsd).  These files contain sets of **references to other database** of clusters of from related genes.  In particular, looking at this [example](https://rest.uniprot.org/uniref/UniRef90_P99999.xml) there doesn't seem to be much direct protein information in these files. 
3. The original code was trying to read a uniref file with the the format set to uniprot.xml, which didn't work.  The code in https://github.com/biopython/biopython/blob/master/Bio/SeqIO/UniprotIO.py:
```python
NS = "{http://uniprot.org/uniprot}"
 for event, elem in ElementTree.iterparse(source, events=("start", "end")):
            if event == "end" and elem.tag == NS + "entry":
                yield Parser(elem, return_raw_comments=return_raw_comments).parse()
                elem.clear()
```
ends up trying to build the entire file as one giant XML document because the if condition never triggers since the XML namespaces never match.

If this is correct, I am not sure what the next step would be.  Perhaps adding a check to the UniProt parsing code to throw an error if the namespace prefix doesn't match the prefix it is expecting?  It isn't clear to me how to write a sequence parser for the uniref file format since it seems like you would have to look up the actual sequence data in a different file?  Happy to try to do more here if someone has suggestions.
Thank you for your investigation @lurkingviolet 

I agree catching this and aborting with a clear exception in the UniProt parser would be a worthwhile improvement. Perhaps there is a tag unique to the Uniref XML which we could look for?

We do have parsers in SeqIO where there is no known sequence (e.g. ``qual`` format), but it is rare. It may be that a non-SeqIO parser for Uniref XML would be more sensible.
OK @peterjc , I will attempt a PR for the exception improvement first.  After that I will try to figure out about how a parser might work.  

@lhallee do you have an example of what you were trying to do with the uniref files.  The code snippet you pointed to looks like it is flattening the sequence clusters into a table.  What did you want to do with data once it was in a table?
Hi @lurkingviolet,

The goal was to extract annotation information about the clusters. What seems more helpful is just getting the ids and parsing through swiss-prot and trembl (the normal XML) which has more information on the representative sequences, which is what I'm currently doing.

Resolving patch:
diff --git a/Bio/SeqIO/UniprotIO.py b/Bio/SeqIO/UniprotIO.py
--- a/Bio/SeqIO/UniprotIO.py
+++ b/Bio/SeqIO/UniprotIO.py
@@ -45,7 +45,15 @@ def UniprotIterator(source, alphabet=None, return_raw_comments=False):
     if alphabet is not None:
         raise ValueError("The alphabet argument is no longer supported")
     try:
-        for event, elem in ElementTree.iterparse(source, events=("start", "end")):
+        for event, elem in ElementTree.iterparse(
+            source, events=("start", "start-ns", "end")
+        ):
+            if event == "start-ns" and not (
+                elem[1].startswith("http://www.w3.org/") or NS == f"{{{elem[1]}}}"
+            ):
+                raise ValueError(
+                    f"SeqIO format 'uniprot-xml' only parses xml with namespace: {NS} but xml has namespace: {{{elem[1]}}}"
+                )
             if event == "end" and elem.tag == NS + "entry":
                 yield Parser(elem, return_raw_comments=return_raw_comments).parse()
                 elem.clear()


Test patch:
diff --git a/Tests/SwissProt/UniRef90_P99999.xml b/Tests/SwissProt/UniRef90_P99999.xml
new file mode 100644
--- /dev/null
+++ b/Tests/SwissProt/UniRef90_P99999.xml
@@ -0,0 +1,313 @@
+<?xml version="1.0" encoding="UTF-8"  standalone="no" ?>
+<UniRef xmlns="http://uniprot.org/uniref" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://uniprot.org/uniref http://www.uniprot.org/docs/uniref.xsd">
+<entry id="UniRef90_P99999" updated="2023-09-13" xmlns="http://uniprot.org/uniref">
+  <name>Cluster: Cytochrome c</name>
+  <property type="member count" value="27"/>
+  <property type="common taxon" value="Catarrhini"/>
+  <property type="common taxon ID" value="9526"/>
+  <representativeMember>
+    <dbReference type="UniProtKB ID" id="CYC_HUMAN">
+      <property type="source organism" value="Homo sapiens (Human)"/>
+      <property type="NCBI taxonomy" value="9606"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000013EAA0"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniRef50 ID" value="UniRef50_P99999"/>
+      <property type="UniProtKB accession" value="P99999"/>
+      <property type="UniProtKB accession" value="A4D166"/>
+      <property type="UniProtKB accession" value="B2R4I1"/>
+      <property type="UniProtKB accession" value="P00001"/>
+      <property type="UniProtKB accession" value="Q6NUR2"/>
+      <property type="UniProtKB accession" value="Q6NX69"/>
+      <property type="UniProtKB accession" value="Q96BV4"/>
+      <property type="length" value="105"/>
+    </dbReference>
+    <sequence length="105" checksum="8EE9689E0102506B">MGDVEKGKKIFIMKCSQCHTVEKGGKHKTGPNLHGLFGRKTGQAPGYSYTAANKNKGIIWGEDTLMEYLENPKKYIPGTKMIFVGIKKKEERADLIAYLKKATNE</sequence>
+  </representativeMember>
+  <member>
+    <dbReference type="UniProtKB ID" id="CYC_GORGO">
+      <property type="source organism" value="Gorilla gorilla gorilla (Western lowland gorilla)"/>
+      <property type="NCBI taxonomy" value="9595"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000013EAA0"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="Q6WUX8"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="CYC_PANTR">
+      <property type="source organism" value="Pan troglodytes (Chimpanzee)"/>
+      <property type="NCBI taxonomy" value="9598"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000013EAA0"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="P99998"/>
+      <property type="UniProtKB accession" value="P00001"/>
+      <property type="UniProtKB accession" value="Q96BV4"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="CYC_PONAB">
+      <property type="source organism" value="Pongo abelii (Sumatran orangutan) (Pongo pygmaeus abelii)"/>
+      <property type="NCBI taxonomy" value="9601"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000013EAA0"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="Q5RFH4"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="G4XXL9_HUMAN">
+      <property type="source organism" value="Homo sapiens (Human)"/>
+      <property type="NCBI taxonomy" value="9606"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000013EAA0"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="G4XXL9"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="A0A2J8VDX6_PONAB">
+      <property type="source organism" value="Pongo abelii (Sumatran orangutan) (Pongo pygmaeus abelii)"/>
+      <property type="NCBI taxonomy" value="9601"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000013EAA0"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="A0A2J8VDX6"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="K7CAI8_PANTR">
+      <property type="source organism" value="Pan troglodytes (Chimpanzee)"/>
+      <property type="NCBI taxonomy" value="9598"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000013EAA0"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="K7CAI8"/>
+      <property type="UniProtKB accession" value="A0A2J8KZA8"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="A0A663D840_PANTR">
+      <property type="source organism" value="Pan troglodytes (Chimpanzee)"/>
+      <property type="NCBI taxonomy" value="9598"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000013EAA0"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="A0A663D840"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="G4XXM0_PANPA">
+      <property type="source organism" value="Pan paniscus (Pygmy chimpanzee) (Bonobo)"/>
+      <property type="NCBI taxonomy" value="9597"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000013EAA0"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="G4XXM0"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="G4XXM2_9PRIM">
+      <property type="source organism" value="Gorilla gorilla (western gorilla)"/>
+      <property type="NCBI taxonomy" value="9593"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000013EAA0"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="G4XXM2"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="G4XXM4_HYLAG">
+      <property type="source organism" value="Hylobates agilis (Agile gibbon)"/>
+      <property type="NCBI taxonomy" value="9579"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000013EAA0"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="G4XXM4"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="G1RY04_NOMLE">
+      <property type="source organism" value="Nomascus leucogenys (Northern white-cheeked gibbon) (Hylobates leucogenys)"/>
+      <property type="NCBI taxonomy" value="61853"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000013EAA0"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="G1RY04"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="G4XXM3_9PRIM">
+      <property type="source organism" value="Pongo sp"/>
+      <property type="NCBI taxonomy" value="9603"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000013EAA0"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="G4XXM3"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="G4XXM6_HYLLA">
+      <property type="source organism" value="Hylobates lar (Lar gibbon) (White-handed gibbon)"/>
+      <property type="NCBI taxonomy" value="9580"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000013EAA0"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="G4XXM6"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="C9JFR7_HUMAN">
+      <property type="source organism" value="Homo sapiens (Human)"/>
+      <property type="NCBI taxonomy" value="9606"/>
+      <property type="protein name" value="Cytochrome c (Fragment)"/>
+      <property type="UniParc ID" value="UPI0001881474"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="C9JFR7"/>
+      <property type="length" value="101"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="A0A2J8VE06_PONAB">
+      <property type="source organism" value="Pongo abelii (Sumatran orangutan) (Pongo pygmaeus abelii)"/>
+      <property type="NCBI taxonomy" value="9601"/>
+      <property type="protein name" value="Cytochrome c (Fragment)"/>
+      <property type="UniParc ID" value="UPI0001881474"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="A0A2J8VE06"/>
+      <property type="length" value="101"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="A0A2J8KZC0_PANTR">
+      <property type="source organism" value="Pan troglodytes (Chimpanzee)"/>
+      <property type="NCBI taxonomy" value="9598"/>
+      <property type="protein name" value="Cytochrome c (Fragment)"/>
+      <property type="UniParc ID" value="UPI0001881474"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="UniProtKB accession" value="A0A2J8KZC0"/>
+      <property type="length" value="101"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="A0A2I3H5G3_NOMLE">
+      <property type="source organism" value="Nomascus leucogenys (Northern white-cheeked gibbon) (Hylobates leucogenys)"/>
+      <property type="NCBI taxonomy" value="61853"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI000C2E177C"/>
+      <property type="UniRef100 ID" value="UniRef100_A0A2I3H5G3"/>
+      <property type="UniRef50 ID" value="UniRef50_P99999"/>
+      <property type="UniProtKB accession" value="A0A2I3H5G3"/>
+      <property type="length" value="107"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="G4XXM5_SYMSY">
+      <property type="source organism" value="Symphalangus syndactylus (Siamang) (Hylobates syndactylus)"/>
+      <property type="NCBI taxonomy" value="9590"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI00022BDD3D"/>
+      <property type="UniRef100 ID" value="UniRef100_G4XXM5"/>
+      <property type="UniRef50 ID" value="UniRef50_P99999"/>
+      <property type="UniProtKB accession" value="G4XXM5"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniProtKB ID" id="A0A2I3GKA8_NOMLE">
+      <property type="source organism" value="Nomascus leucogenys (Northern white-cheeked gibbon) (Hylobates leucogenys)"/>
+      <property type="NCBI taxonomy" value="61853"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniParc ID" value="UPI00020ACF7D"/>
+      <property type="UniRef100 ID" value="UniRef100_A0A2I3GKA8"/>
+      <property type="UniRef50 ID" value="UniRef50_P99999"/>
+      <property type="UniProtKB accession" value="A0A2I3GKA8"/>
+      <property type="length" value="105"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniParc ID" id="UPI0024428584">
+      <property type="source organism" value="Symphalangus syndactylus"/>
+      <property type="NCBI taxonomy" value="9590"/>
+      <property type="protein name" value="cytochrome c isoform X1"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="length" value="112"/>
+      <property type="isSeed" value="true"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniParc ID" id="UPI0000128BBF">
+      <property type="source organism" value="Homo sapiens"/>
+      <property type="NCBI taxonomy" value="9606"/>
+      <property type="protein name" value="Cytochrome c"/>
+      <property type="UniRef100 ID" value="UniRef100_P99999"/>
+      <property type="length" value="104"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniParc ID" id="UPI001E2547C2">
+      <property type="source organism" value="Macaca fascicularis"/>
+      <property type="NCBI taxonomy" value="9541"/>
+      <property type="protein name" value="cytochrome c-like"/>
+      <property type="UniRef100 ID" value="UniRef100_UPI001E2547C2"/>
+      <property type="UniRef50 ID" value="UniRef50_P99999"/>
+      <property type="length" value="110"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniParc ID" id="UPI002442E06B">
+      <property type="source organism" value="Symphalangus syndactylus"/>
+      <property type="NCBI taxonomy" value="9590"/>
+      <property type="protein name" value="cytochrome c-like"/>
+      <property type="UniRef100 ID" value="UniRef100_UPI002442E06B"/>
+      <property type="UniRef50 ID" value="UniRef50_P99999"/>
+      <property type="length" value="108"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniParc ID" id="UPI0005F4224E">
+      <property type="source organism" value="Colobus angolensis palliatus"/>
+      <property type="NCBI taxonomy" value="336983"/>
+      <property type="protein name" value="cytochrome c-like"/>
+      <property type="UniRef100 ID" value="UniRef100_UPI0005F4224E"/>
+      <property type="UniRef50 ID" value="UniRef50_P99999"/>
+      <property type="length" value="96"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniParc ID" id="UPI00122DAE92">
+      <property type="source organism" value="Nomascus leucogenys"/>
+      <property type="NCBI taxonomy" value="61853"/>
+      <property type="protein name" value="cytochrome c-like"/>
+      <property type="UniRef100 ID" value="UniRef100_UPI00122DAE92"/>
+      <property type="UniRef50 ID" value="UniRef50_P99999"/>
+      <property type="length" value="93"/>
+    </dbReference>
+  </member>
+  <member>
+    <dbReference type="UniParc ID" id="UPI002441379E">
+      <property type="source organism" value="Symphalangus syndactylus"/>
+      <property type="NCBI taxonomy" value="9590"/>
+      <property type="protein name" value="cytochrome c-like"/>
+      <property type="UniRef100 ID" value="UniRef100_UPI002441379E"/>
+      <property type="UniRef50 ID" value="UniRef50_P99999"/>
+      <property type="length" value="90"/>
+    </dbReference>
+  </member>
+</entry>
+
+</UniRef>
\ No newline at end of file
diff --git a/Tests/test_SeqIO.py b/Tests/test_SeqIO.py
--- a/Tests/test_SeqIO.py
+++ b/Tests/test_SeqIO.py
@@ -2539,6 +2539,25 @@ def test_uniprot_xml5(self):
             messages,
         )
 
+    def test_uniprot_xml_namespace(self):
+        uniref_file_name = (
+            "SwissProt/UniRef90_P99999.xml"  # non-uniprot file (related uniref format)
+        )
+        with self.assertRaises(ValueError) as context:
+            records = []
+            for record in SeqIO.parse(uniref_file_name, format="uniprot-xml"):
+                records.append(record)
+        self.assertRegex(
+            str(context.exception),
+            "http://uniprot.org/uniprot",
+            "Correct namespace in error",
+        )
+        self.assertRegex(
+            str(context.exception),
+            "http://uniprot.org/uniref",
+            "Unexpected namespace in error",
+        )
+
     def test_swiss20(self):
         sequences = ["MTMAAAQGKLSPDAIDNEVISNGSAKDYLDPPPAPLVDAG...SNYDAAV"]
         ids = ["F2CXE6"]