Agree, potential for 2x representations in mem (file contents and resulting
array)
On Apr 8, 2014 7:57 PM, "Greg Caporaso" <notifications@github.com> wrote:

> In skbio/core/distance.py:
>
> >          """
> >          # We aren't using np.loadtxt because it uses *way* too much memory
> >          # (e.g, a 2GB matrix eats up 10GB, which then isn't freed after parsing
> >          # has finished). See:
> >          # http://mail.scipy.org/pipermail/numpy-tickets/2012-August/006749.html
> >
> > +        # check if it's a string and a valid path, if so read the contents
> > +        if isinstance(dm_f, str) and exists(dm_f):
> > +            fd = open(dm_f, 'U')
> > +            dm_f = fd.readlines()
>
> Is it going to be a problem to read the whole file into memory?
>
> --
> Reply to this email directly or view it on GitHub<https://github.com/biocore/scikit-bio/pull/255/files#r11420196>
> .
>