Ack sorry I owed you a run on this myself, sorry totally missed this. 

But I agree with you that we ought not to be so aghast at separating the parallelism between FITS and ASCII as the FITS handling is a pain in this model.  It does suck to not be able to pull out a column without a painful Python overhead (at least a list comprehension), but this is a problem that I believe is inherent in the PyFITS layout.

--
Barnaby Rowe
Postdoctoral Research Associate

Department of Physics & Astronomy
University College London
Gower Street
London WC1E 6BT
United Kingdom

On 18 Mar 2013, at 20:40, Mike Jarvis wrote:

> In galsim/catalog.py:
> 
> > +
> > +        # If all we care about is nobjects, we can do that quickly here.
> > +        if nobjects_only: 
> > +            self.nobjects = len(data.field(self.names[0]))
> > +            return
> > +
> > +        # data is an instance of a weird numpy FITS_rec class
> > +        # The main problem with it is that it isn't picklable, so using this 
> > +        # with multiprocessing will fail.
> > +        # So we turn this into a regular numpy array that looks just like the version
> > +        # we build in read_ascii with loadtxt.
> > +        # This assumes all fields have the same length.  If we need to support tables
> > +        # where this isn't true, we might need to add an if clause in here.
> > +        # (i.e. check the maximum, and then only include keys whose length is the same as that.)
> > +        # Seems like this would be a rare case though, so I haven't implemented it yet.
> > +        self.data = numpy.array([ [ str(x) for x in data.field(k) ] for k in self.names ]).T
> 
> It seems like the string conversion is the real slow bit:
> 
> >>> t0=time.time(); dtmp = [[i for i in data.item(j)] for j in range(len(data))]; print time.time() - t0
> 0.0318548679352
> >>> t0=time.time(); dtmp = [[str(i) for i in data.item(j)] for j in range(len(data))]; print time.time()-t0
> 1.43106102943
> >>> t0=time.time(); dtmp = numpy.array([[str(i) for i in data.item(j)] for j in range(len(data))]); print time.time()-t0
> 1.54542899132
> >>> t0=time.time(); dtmp = [[x for x in data.field(k)] for k in names ]; print time.time() - t0
> 0.0290760993958
> >>> t0=time.time(); dtmp = [[str(x) for x in data.field(k)] for k in names ]; print time.time() - t0 
> 1.47440385818
> >>> t0=time.time(); dtmp = numpy.array([[str(x) for x in data.field(k)] for k in names ]).T; print time.time() - t0
> 1.50754714012
> 
> But unfortunately, it seems to be required if we want to make a numpy array:
> 
> >>> dtmp = numpy.array([[x for x in data.field(k)] for k in names ]).T
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ValueError: setting an array element with a sequence.
> >>> dtmp = numpy.array([[i for i in data.item(j)] for j in range(len(data))])
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> ValueError: setting an array element with a sequence.
> 
> However, another option would be to just sever the parallelism between FITS and ASCII entirely. I already store an isfits attribute, so we can just have the get method use data.field(col)[index] directly, and not even try to have it stored in any kind of array. I'm thinking this is really the best option here.
> 
> —
> Reply to this email directly or view it on GitHub.
>