Repository: ipython/ipython

Issue:
whos magic command fails on Named Tuple type
In IPython version **9.4.0** (Python 3.13.1), saving a Python Named Tuple type to a variable, then running the 'whos' command, causes an exception, because apparently the Named Tuple type variable has a \_\_len\_\_ attribute (as the whos command checks for), but you cannot otherwise actually call len() on this object.

Recreate:
```
In [1]: from collections import namedtuple
In [2]: nt = namedtuple('MyNamedTuple', 'col1 col2 col3')
In [3]: whos
Variable     Type        Data/Info
----------------------------------
namedtuple   function    <function namedtuple at 0x00000257CF0625C0>
nt           type        ---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[3], line 1
----> 1 get_ipython().run_line_magic('whos', '')

File ~\AppData\Local\Programs\Python\Python313\Lib\site-packages\IPython\core\interactiveshell.py:2504, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
   2502     kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2503 with self.builtin_trap:
-> 2504     result = fn(*args, **kwargs)
   2506 # The code below prevents the output from being displayed
   2507 # when using magics with decorator @output_can_be_silenced
   2508 # when the last Python token in the expression is a ';'.
   2509 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):

File ~\AppData\Local\Programs\Python\Python313\Lib\site-packages\IPython\core\magics\namespace.py:475, in NamespaceMagics.whos(self, parameter_s)
    472     print(f"Shape: {var.shape}")
    473 elif hasattr(var, "__len__"):
    474     ## types that can be used in len function
--> 475         print(var if isinstance(var, str) else f"n={len(var)}")
    476 else:
    477     try:

TypeError: object of type 'type' has no len()
```

As a quick hack/work-around, I simply put a try/except around line 475 in **namespace.py**, but there may be a more intelligent solution.

[Edited to add Python version 3.13.1 in initial sentence.]


Discussion:
Same error I get for an `astropy.table.Table` object. 

https://docs.astropy.org/en/stable/api/astropy.table.Table.html#astropy.table.Table

minimal example:
```
from astropy.table import Table
t = Table()
%whos
```

But the issue seems to be deeper, because other than `namedtuple`,  the Table object supports `len()`:

``` 
> hasattr(t, "__len__")
True
> len(t)
0
```

wheras

```
> from collections import namedtuple
> nt = namedtuple('MyNamedTuple', 'col1 col2 col3')
> hasattr(nt, "__len__")
True
> len(nt)
TypeError: object of type 'type' has no len()
```

So in the first case I don't understand why it fails.  In the second case `__len__` is there for named tuple, but not working as expected.
Thanks for the report. I see it still happens on `main` branch after https://github.com/ipython/ipython/pull/14938. PR welcome!

CC @vlulla
Sorry for the noise, anyway...

`hasattr(Table, '__len__')` is `True` but `len(Table)` gives a type error... 

 I think the problem is that `__len__`  may be there, but not do what we want... In the python docs it says `__len__`  *should* return the length... So there is no guarantee...
Try and except OK here?  I could give it a shot...
It is interesting, in the case of astropy, the failure is actually on the Table 'type', not on the 't' variable, so it is a slightly different, and as you say, possibly deeper error. Note that **you do not even need to declare the 't' variable!** The error is occurring when it is trying to show the 'Table' type that is loaded in.

```
Python 3.13.1 (tags/v3.13.1:0671451, Dec  3 2024, 19:06:28) [MSC v.1942 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 9.4.0 -- An enhanced Interactive Python. Type '?' for help.
Tip: You can use `%hist` to view history, see the options with `%history?`

In [1]: from astropy.table import Table
In [2]: whos
Variable   Type    Data/Info
----------------------------
Table      type    ---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[2], line 1
----> 1 get_ipython().run_line_magic('whos', '')

File ~\AppData\Local\Programs\Python\Python313\Lib\site-packages\IPython\core\interactiveshell.py:2504, in InteractiveShell.run_line_magic(self, magic_name, line, _stack_depth)
   2502     kwargs['local_ns'] = self.get_local_scope(stack_depth)
   2503 with self.builtin_trap:
-> 2504     result = fn(*args, **kwargs)
   2506 # The code below prevents the output from being displayed
   2507 # when using magics with decorator @output_can_be_silenced
   2508 # when the last Python token in the expression is a ';'.
   2509 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):

File ~\AppData\Local\Programs\Python\Python313\Lib\site-packages\IPython\core\magics\namespace.py:475, in NamespaceMagics.whos(self, parameter_s)
    472     print(f"Shape: {var.shape}")
    473 elif hasattr(var, "__len__"):
    474     ## types that can be used in len function
--> 475     print(var if isinstance(var, str) else f"n={len(var)}")
    476 else:
    477     try:

TypeError: object of type 'type' has no len()
```

In any case, it seems to eventually land on the same line as the namedtuple problem. In 3.12, it correctly shows the 'type' of Table, rather than trying to show the len(), so something got mangled in this more recent release (3.12 also correctly shows my namedtuple type). The try/catch around len() will *mask* the problem and stop the error, but it may not fix the underlying issue with %whos misinterpreting types as variables.

I see the difference in **namespace.py** from 3.12 to 3.12, the initial `if vtype in seq_types` (shown in code snip below) was removed, and then an `elif` was added farther down to check instead for `__len__`. It seems (from the helpful comment) that some logic was added to try to handle the "shape" of DataFrames and Series for pandas and polars, and in the process, handling of types and seq_types did not work out correctly.
```
if vtype in seq_types:
  print("n="+str(len(var)))
elif vtype == ndarray_type:
  ```
I might suggest that whomever submitted the changes for 3.13 (or someone who might have the 'bigger picture' for the logic in that file) might have a look at the code, in light of these problems, rather than a PR for the `__len__` condition to hide the problem??


I made these changes to fix some strange printing of pandas series and dataframes (embedded `\r` in columns!). And, it appears that i have made a mistake...OOPS!  Sorry!

I suspect that changing line 475 to `print(var if isinstance(var, str) else f"n={len(var) if not isinstance(var, type) else var}")` ought to fix this issue. I have checked that this temporary fix ought to work with both the examples (nt and astropy table). I am unsure if this is general enough though.

I have looked at %whos magic and have concluded that I do not understand enough internals of python to come up with a general solution for how whos ought to behave for more complicated objects (xarray, tensors, etc) which need proper handling of shape[s].
I will try sorting this out over the weekend. Since you all are knowledgeable about this issue, I hope it's alright to list you all (@Knusper , @dale-roberts , @krassowski ) as reviewers for the upcoming PR.
Hi @vlulla, that is helpful to know! I am not very GitHub savvy and have not done a PR before (I should probably figure that out some day...), so I would be happy for you to submit the correction, and would also be happy to review the final PR. I might suggest to simply go back to the 3.12 version, and (in addition to your added comments) only add the case to specifically check for the type being either **DataFrame** or **Series**. It seems that checking just for the 'shape' attribute may have unintended consequences. For instance, when I import **numpy**, the module itself happens to have a 'shape' attribute, but we would not want to use that in the Data/Info column, we would rather see that it is a module.

Otherwise just leave everything like it was in the 3.12 version. As the 3.13 version is now, checking for the **\_\_len\_\_** attribute hijacks the correct handling of strings, which had been nicely handled in the bottom catch-all 'else' clause, and would helpfully trim the strings (and remove newlines) to show at most 50 characters, which I think is what most people would want.

Happy to discuss further, but do you think something like this looks good (it seems to work well for the cases discussed here so far)? It leaves the original check for **seq_types** (`seq_types = ['dict', 'list', 'tuple']`) at the top, and adds a more specific check for **DataFrame** and **Series** after the ndarray_type check. Then strings would also be correctly handled by the final 'else'.

So this is basically the 3.12 version, but with an elif added for the specific types of **DataFrame** and **Series**:

```python
for vname,var,vtype in zip(varnames,varlist,typelist):
    print(vformat.format(vname, vtype, varwidth=varwidth, typewidth=typewidth), end=' ')
    if vtype in seq_types:
        print("n="+str(len(var)))
    elif vtype == ndarray_type:
        vshape = str(var.shape).replace(',','').replace(' ','x')[1:-1]
        if vtype==ndarray_type:
            # numpy
            vsize  = var.size
            vbytes = vsize*var.itemsize
            vdtype = var.dtype
        if vbytes < 100000:
            print(aformat % (vshape, vsize, vdtype, vbytes))
        else:
            print(aformat % (vshape, vsize, vdtype, vbytes), end=' ')
            if vbytes < Mb:
                print('(%s kb)' % (vbytes/kb,))
            else:
                print('(%s Mb)' % (vbytes/Mb,))
    elif vtype in ['DataFrame', 'Series']:
        # Useful for DataFrames and Series
        # Ought to work for both pandas and polars
        print(f"Shape: {var.shape}")
    else:
        try:
            vstr = str(var)
        except UnicodeEncodeError:
            vstr = var.encode(DEFAULT_ENCODING,
                              'backslashreplace')
        except:
            vstr = "<object with id %d (str() failed)>" % id(var)
        vstr = vstr.replace('\n', '\\n')
        if len(vstr) < 50:
            print(vstr)
        else:
            print(vstr[:25] + "<...>" + vstr[-25:])
```
What do you think?

@dale-roberts your reasons are sound and I think reverting back is a good idea.

I encourage you to try submitting a PR. I too am very new to contributing to my favorite open source projects...and i have found doing PRs (the couple that i've done) educational and rewarding. The [CONTRIBUTING.md](https://github.com/vlulla/ipython/blob/main/CONTRIBUTING.md) does a good job of describing how you can make a PR. I believe you will be able to do a PR! And, I will be happy to help out (with either testing or reviewing) with your PR if you wish to take a crack. Please let me know if you're tentative and I'll gladly revert the change in a new PR (including you as a reviewer).

Thanks for your help with this!
@vlulla, thank you for the encouragement and instruction, I will give it a try...
@dale-roberts any progress? If you are stuck on something, is there anything i can do to help?
Sorry for the delay, no problems, it just fell off my radar. I put on my calendar to look into the PR next week. Thanks!
I wonder if we can get a quick patch in by Friday when we would be releasing 9.5.0. It looks like the problem is that we are calling `len` on classes and not only on objects. The problematic line is

https://github.com/ipython/ipython/blob/57b055533d453c336c1816066865676fb1dff990/IPython/core/magics/namespace.py#L473

It should be:

```python
from import collections.abc import Sized
# ...
elif isinstance(var, Sized) and not isinstance(var, str): 
```
Thanks for the heads-up on the Fri release. I'll try to put a PR in today, which will be basically the code above, which starts from the previous version, and adds the DataFrame and Series as a separate case. Just fixing the one len() line won't do it, because the 'default' handling of strings also got hijacked and messed up.
Okay, my first PR, all automated tests seem to have passed after I corrected a couple formatting issues. Please let me know if anything more is needed, and I'm happy to answer/discuss any further concerns.
Thanks @dale-roberts for getting to this promptly! This looks great!
Thanks @vlulla. Should I have added any more '@' people to the Pull Request, so it can get reviewed and hopefully merged before Friday? Or is the discussion in this Issue sufficient? I did not think to do that. I just want to make sure the pertinent people are aware. Thanks for any guidance.

... okay, I did add @krassowski to the pull request! Anyone else?
I do not think anyone other than @krassowski needs to be included. Once again, thanks for the help!



Patch:
diff --git a/IPython/core/magics/namespace.py b/IPython/core/magics/namespace.py
index 0ad1560b948..82cc181fec0 100644
--- a/IPython/core/magics/namespace.py
+++ b/IPython/core/magics/namespace.py
@@ -361,8 +361,7 @@ def whos(self, parameter_s=''):
           - For numpy arrays, a summary with shape, number of
             elements, typecode and size in memory.
 
-          - For objects that have shape attribute, primarily dataframe and series like
-            objects, will print the shape.
+          - For DataFrame and Series types: their shape.
 
           - Everything else: a string representation, snipping their middle if
             too long.
@@ -450,7 +449,9 @@ def type_name(v):
         Mb = 1048576  # kb**2
         for vname,var,vtype in zip(varnames,varlist,typelist):
             print(vformat.format(vname, vtype, varwidth=varwidth, typewidth=typewidth), end=' ')
-            if vtype == ndarray_type:
+            if vtype in seq_types:
+                print("n="+str(len(var)))
+            elif vtype == ndarray_type:
                 vshape = str(var.shape).replace(',','').replace(' ','x')[1:-1]
                 if vtype==ndarray_type:
                     # numpy
@@ -463,16 +464,13 @@ def type_name(v):
                 else:
                     print(aformat % (vshape, vsize, vdtype, vbytes), end=' ')
                     if vbytes < Mb:
-                        print('(%s kb)' % (vbytes/kb,))
+                        print("(%s kb)" % (vbytes / kb,))
                     else:
                         print("(%s Mb)" % (vbytes / Mb,))
-            elif hasattr(var, "shape"):
+            elif vtype in ["DataFrame", "Series"]:
                 # Useful for DataFrames and Series
                 # Ought to work for both pandas and polars
                 print(f"Shape: {var.shape}")
-            elif hasattr(var, "__len__") and not isinstance(var, str):
-                ## types that can be used in len function
-                print(f"n={len(var)}")
             else:
                 try:
                     vstr = str(var)


Test patch:
diff --git a/tests/test_magic.py b/tests/test_magic.py
index cc772e10e73..efd5ff4323c 100644
--- a/tests/test_magic.py
+++ b/tests/test_magic.py
@@ -1,6 +1,7 @@
 # -*- coding: utf-8 -*-
 """Tests for various magic functions."""
 
+import collections
 import gc
 import io
 import json
@@ -768,6 +769,49 @@ def test_whos_longstr(input, expected):
     assert expected == from_whos
 
 
+def test_whos_len_namedtuple():
+    _ip = get_ipython()
+    _ip.run_line_magic("reset", "-f")
+    _ip.user_ns["alpha"] = 123
+    _ip.user_ns["beta"] = "test"
+    _ip.user_ns["nt"] = collections.namedtuple("MyNamedTuple", "col1 col2")
+    _ip.user_ns["x"] = _ip.user_ns["nt"]("a", "b")
+    expected = (
+        "Variable   Type            Data/Info\n"
+        "------------------------------------\n"
+        "alpha      int             123\n"
+        "beta       str             test\n"
+        "nt         type            <class 'tests.test_magic.MyNamedTuple'>\n"
+        "x          MyNamedTuple    MyNamedTuple(col1='a', col2='b')\n"
+    )
+    with capture_output() as captured:
+        ip.run_line_magic("whos", "")
+    stdout = captured.stdout
+    assert stdout == expected.strip() + "\n"
+
+
+@dec.skip_without("pandas")
+def test_whos_len_pandas():
+    import pandas as pd
+
+    _ip = get_ipython()
+    _ip.run_line_magic("reset", "-f")
+    df = pd.DataFrame({"a": range(10), "b": range(10, 20)})
+    _ip.user_ns["df"] = df
+    s = df["a"]
+    _ip.user_ns["s"] = s
+    expected = (
+        "Variable   Type         Data/Info\n"
+        "---------------------------------\n"
+        "df         DataFrame    Shape: (10, 2)\n"
+        "s          Series       Shape: (10,)\n"
+    )
+    with capture_output() as captured:
+        ip.run_line_magic("whos", "")
+    stdout = captured.stdout
+    assert stdout == expected.strip() + "\n"
+
+
 def doctest_precision():
     """doctest for %precision