What you really want to do is not read all the lines, but instead use the file object as an something you can iterate over such that you read the lines as you need them instead of reading all the lines at once.

https://github.com/biocore/scikit-bio/blob/master/skbio/core/distance.py#L263-266