Add an attach method to DataFrame
State: closed

This is a method that injects the columns of a DataFrame into the calling namespace. As far as I can tell it works, even if it is hackish. However, the nosetests do not like the trickery and throw errors and failures. But, if you do

nosetests test_frame.py --pdb --pdb-failure

You can indeed see that the tests will pass from inside the debugger. I'm not really sure why. Does anyone have any ideas?

Am I missing anything about injecting variables into the calling local variables of the calling frame? I'm sure this is bad form, but it's functionality I've missed in Python.

Am I safe in assuming that column names are valid python variable names? If not, I may need to use a name validator.


Labels:

Comments:
Cool stuff, I can see this being useful, notwithstanding the black magic aspect of it. I also see the failures. Looks like the symbol table is getting screwed up somehow within nose - thinking 'myspace' for instance is global. Weird.

Python does a compile-time optimisation where local variables are loaded in a different way, that doesn't work if they're modified dynamically (LOAD_FAST in the byte code). We've run into this with IPython as well, and there doesn't seem to be any way round it.

It's not safe to assume that column names are valid variable names, or even that they're strings. See the checks we do for tab completion of column names as attributes: https://github.com/wesm/pandas/blob/master/pandas/core/frame.py#L4131

Overall, I'm mildly against including this 'feature' - I've always been advised to avoid attach in R, and it breaks the strong conventions about namespaces in Python. If you give your dataframe a short name, it's only two more characters to refer to `d.column`. Plus you run into bizarre bugs like the one you found, because Python isn't designed to support this kind of thing.

Thanks, Thomas, that makes sense. To be honest, I'm probably a -0 on including it as a method so as not to encourage it's use. Might it make sense as a utility function with some more restrictive checks and a big honking warning? I think if you're conservative with using attach in R, it's ok.

Alternatively, it could work as a context manager and let the Series be available in the context.

I'd be inclined to make a standalone function, and put it in a 'recipe' somewhere that we can point people to if they really want that behaviour. Putting in pandas in any form would imply that it worked reliably, which I don't think it can.

I don't like 'attach' much, but I do like R's much more sensible 'with' function: I write things like

```
with(mydf[, mydf$Age > 30], plot(Response ~ Dosage))
```

all the time. (Actually I would write that 'with(subset(mydef, Age > 30), plot(Response ~ Dosage)), taking advantage of subset()'s magic with()-like powers. But you can't have everything.)

@takluyver No, it is possible to reliably inject local variables. The annoying thing is that which variables are considered "local" is defined at compile-time, though, so generally most variables that one might want to create will turn out to be "global". Which is also okay -- we can reliably inject global variables. We can even restore them again afterward, if we're careful (got to watch out for that DataFrame({"**builtins**": [1, 2, 3]})!). But! This cannot be done in a way that is _threadsafe_ -- that's the unsolvable problem.

I guess lots of use cases for this don't care about thread-safety though, and I'd like to write

```
with mydf[mydf.Age > 30]: plot(Dosage, Response)
```

So here you go: https://gist.github.com/2347382

Written as standalone right now, syntax is currently 'with data(mydf): ...'. I'd suggest keeping it as a class in its own file, as per the comments at the top, but then adding **enter** and **exit** methods to DataFrame that delegate to it, so that 'with mydf: ...' will work. Doesn't know anything about Series or Panel -- should it?

...that's `__enter__` and `__exit__` methods, of course.

OK, I've got to hand it to you that it's possible. But as I think you've already spotted, it's not a solution I think we should be promoting to users:
- It still breaks the Python convention that variables shouldn't just pop out of nowhere. Being able to easily find where a variable comes from is valuable.
- Some variables go to locals, some to globals, so there's plenty of potential for unexpected side effects. If I have a column called `len`, and code inside a `with` block calls/yields to other code in the same module, it can suddenly blow up in all sorts of novel and unhelpful ways.
- As you say, it's not thread safe.
- Using ctypes to reach into Python's C API makes it implementation specific. (Of course, so does using Cython at the moment, but I understand the Cython people are working on ways to support PyPy, at least).

I can see there are use cases where none of those would be a problem, but I think it's dangerous to offer something like that unless it works really well. Once people have learnt it, it looks easy and simple enough that they'll carry on using it, and when it all goes wrong, it won't be obvious that their casual `with df:` was the culprit.

Perhaps one way forward would be to add a method which generates (textual) code to unpack the columns, like `foo, bar, baz = mydf.cols` (that's hypothetical, and doesn't work at present). With appropriate integration with IPython, you could call a magic function like `%attach mydf`, and it drops the unpacking line in the start of the next cell, ready for you to add your code beneath.

I'm not worried about variables popping out of nowhere, since the 'with mydf:' is a pretty huge signpost, and the ctypes thing is academic since there's no prospect of PyPy and numpy reconciliation anytime soon... and anyway PyPy will need _some_ way to implement locals(). But ugh, yeah, the part where the new variables leak to call-or-yielded-to functions is horrible, I can't believe I missed that. :-(

Thinking out loud...

Refusing to overwrite any existing globals would substantially reduce this problem; then the only way code outside the with: block could be affected would be (1) if they were accessing an undefined global, they'd get some random Series object instead of raising a NameError. (2) if they were injecting a new variable into the global namespace, then they could accidentally overwrite one of our globals. The former I could ignore. The latter is... well, a terrible idea, but I suppose you can't actually stop people from using global variables, though plenty of people have tried. I dunno, they're pretty obscure cases, but I'm not entirely comfortable with them.

Okay, so maybe there's no viable way to do this in CPython for "real" code. It'd still be really nice to have something like it for one-off interactive uses. Which is much simpler. Perhaps a %with magic? The primary use case for me is working with complicated subsets of data -- in the example above, you can see how with

```
with mydf[mydf.Age > 30]: plot(Dosage, Response)
```

is already much simpler to deal with than

```
plot(mydf.Dosage[mydf.Age > 30], mydf.Response[mydf.Age > 30])
```

and you can see how this advantage grows _dramatically_ as the complexity of the subsetting expression grows. And it's not just that only having to type it once is quicker -- it's a huge win interactively, because you can hit up-arrow, tweak the subsetting expression _in one place_, and re-run. But this is also why I'm not so sure about the 'attach' style usage -- I want to be able to re-run stuff quickly with slightly different data-frames, and the 'attach' syntax makes that cumbersome. Then again, maybe I should just get in the habit of always writing `tmp = mydf; ...<code involving tmp>...`.

I don't think the generate-explicit-unpacking-code idea is really addressing this use case. There might be less sophisticated users for whom the real advantage of 'attach' is just that you don't have to remember to type `mydf.` all the time, and it might be nice for them. But they might be better off just learning to type `mydf.` :-).

`with mydf:` might be a pretty huge signpost to you and me, but if you're not familiar with what it's doing, it doesn't look dangerous. Even people familiar with context managers wouldn't go looking for that as the culprit for unexpected variable changes. Refusing to overwrite any globals would help, but then you get the other problem - naive coders wondering why their `id` or `zip` or `format` column doesn't work properly.

I'd probably go down the `tmp` route myself - especially using something like the IPython notebook, where you can easily have more than one line in a cell, so it would look like:

```
g = mydf[mydf.Age > 30]
plot(g.Dosage, g.Response)
```

A magic command would be another route to go, although supporting your example could be tricky, because you need to separate the context part (`mydf[mydf.Age > 30]`) from the execution part (`plot(Dosage, Response)`), so you'd need to parse the input to some extent. I suspect that's possible, though. But I wouldn't call it `%with`, because reusing a keyword seems like a very bad idea. Maybe something like `%indf`.

It would be relatively simple to write an `%attach` style magic, which would print the names of the variables it had created. I think this would be OK, because it's hopefully clear that magic functions are intended only for interactive use, not for production code.

You could keep the colon, so it looked like `%indf mydf[mydf.Age > 30]: ...`. That'd make parsing trivial, since a python expression cannot contain a colon token. (You'd need to tokenize in case there was a colon embedded in a string or something, but that's just a matter of `import tokenize`.) That might also be nice in that it would make it natural to trigger ipython's multi-line statement mode: type `%indf <whatever>:` and hit enter, then you can type multiple lines and later you can get them all back at once by hitting up-arrow once?

mydf[15:25] has a colon in valid syntax, and it's the same token with tokenize ;-). I'm sure there's a way to handle that, but it's not entirely trivial. We could pick a completely unused token (I think $ is one), but it would be less intuitive.

I think magic commands at present are a single-line thing, though we're thinking about 'cell-level' magics.

On Wed, Apr 11, 2012 at 4:15 PM, Thomas Kluyver
reply@reply.github.com
wrote:

> mydf[15:25] has a colon in valid syntax, and it's the same token with tokenize ;-). I'm sure there's a way to handle that, but it's not entirely trivial. We could pick a completely unused token (I think $ is one), but it would be less intuitive.

Bah, I am forgetting all the obvious things today.

It's still pretty easy to handle, you just have to count parentheses
nesting. But not as trivial, yeah.

> I think magic commands at present are a single-line thing, though we're thinking about 'cell-level' magics.

Oh, too bad. I'm still a luddite running ipython in a terminal, so I
don't know about these "cell" things.

Well, cells aren't as important in the terminal, but the new notebook in particular is designed around cells. It just means the code at an `In [1]:` prompt that's executed in one block.