PDEP-5: NoRowIndex
State: closed

Initial draft of a proposal to have a "no index by default" mode

@pandas-dev/pandas-core @pandas-dev/pandas-triage would appreciate any thoughts / feedback, thanks 🙏 

Labels:
name: PDEP
description: pandas enhancement proposal

Comments:
Is there a proposed result for `df.sum()`?

```
print(pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5]}).sum())
# a     4
# b    12
# dtype: int64
```

I suppose this must no longer be a Series? Edit: Or is the result the values without any column labels?

Another one is transpose - which I think will have to lose the column labels. We use transpose internally at times to do axis=1 computations, but we can track the column labels there.
Thanks all for your comments!

> Is there a proposed result for df.sum()?

Ah, that's a really good one. I hadn't tried that out, and I don't have a good solution.

It'd be nice to do
```python
In [11]: df
Out[11]: 
 a  b  c
 1  4  7
 2  5  8
 3  6  9

In [12]: df.sum()
Out[12]: 
 a  b  c
 6 15 24
```
but I think that'd be too much of a breaking change from the regular pandas case.

Not keen on having:
```python
In [14]: df.sum()
Out[14]: 
 6
15
24
dtype: int64
```
because here the row labels would be really meaningful, but also not keen on having and extra column like
```python
In [19]: df.sum()
Out[19]: 
column  sum
     a    6
     b   15
     c   24
```

I can't see how to get this to conform with "3. Nobody should get an index unless they ask for one" without it being too big of a change to current behaviour.

I think I may need to drop "3." from this proposal, and only describe how a `NoIndex` object would look like
> I can't see how to get this to conform with "3. Nobody should get an index unless they ask for one" without it being too big of a change to current behaviour.
> 
> I think I may need to drop "3." from this proposal, and only describe how a `NoIndex` object would look like

I would personally find that the better option. 

Specifically for the reduction example: a DataFrame has named columns, so if those end up in the index in a certain operation, we use those names. So you could say that the user "asks for this", when it does an operation where the columns names are used for row labels. 
Of course it could be avoided that those column names end up in the row labels, by returning an object of a different shape (DataFrame with 1 row vs Series), but considering to change the output type / shape of reductions seems indeed a bigger / separate discussion. 

I personally also think that having an option to have to no index _to start with_ (i.e, in IO, constructors, etc) is already useful in itself (and is a smaller scoped change than the "with this option enabled, no index is _ever_ created", since we just have quite some functionality that introduce meaningful indices even when starting from a default range index / no index)

`NoRowIndex` is probably better, thanks - column labels would always be there (and would be explicitly prohibited from being `NoRowIndex`)

Thanks everyone for your comments, really appreciate them - I'll revise the proposal trying to account for everyone's thoughts

For now it'll focus exclusively on the `NoRowIndex` object - discussions around `as_index` defaults and `pd.set_option` can happen separately
Do you have a point of view as to what should happen when objects without an index that are a different length get added? Do they truncate to the shortest length, raise an error, or do something else?
that'd raise - aiming to push a revision with plenty of examples tomorrow
Thanks all for your reviews

I've pushed another commit with a first revision. I've resolved most of the comments as I think I've addressed them in the revision, either in the body or in the FAQ section

I've left open the discussion about whether ``NoRowIndex`` should be part of the public API, as I'm not sure about that yet
My one general issue is this might be throwing out the baby with the bath water, though I get why it might be easy to implement.   It also feels quite dangerous as it will be inconsistent with pretty much how pandas is documented everywhere to behave.  This will make stack overflow questions hard to answer as you'll need to know whether this option was set.

Row Indices are also broadly useful and declarative.  The point of pandas over numpy is that it adds meta-data which controls its behavior.  I can't imagine turning them off and retaining the value of Pandas.  

The problem with indices occurs with implicit and rather arbitrary data alignment during various operations, such as sum above.  In other data QLs you need to specify this behavior explicitly via a join clause.  

Better I think would be to leave indices largely alone but have an option (which can be globally set) to turn it off/on with one of the following:  strict mode / error on unaligned indices, off with ignore error (which this seems to be, and it seems very dangerous), or on (current behavior).   The dangerous mode, I only include it to be friendly, but I'd advise against it.  For stack overflow questions, an answer could often be - try setting strict mode to help debug your problem.

Data alignment may be confusing, but it's extremely important as it ensures that the user isn't incorrectly collating unaligned data, which can easily happen.   Implicit behavior like this is often the source of confusion, especially when it's not very consistent.

My biggest complaint with how it currently automagically works and doesn't warn you when probably you made a mistake.  

eg:  

https://github.com/pandas-dev/pandas/issues/49617
Thanks for taking a look - I'll respond to some of your points:
- StackOverflow requires minimal reproducible examples (which'd need to include any options which were set), so this shouldn't be an issue there. There's already [lot's of options](https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html) which can be set in pandas, I've not seen them cause confusion on SO;
- this wouldn't remove the value of pandas over numpy because column names would still be there, along with the operations such as groupby which aren't in numpy

This wouldn't be the dangerous mode you describe, because without row labels, there'd be no concept of misalignment. If you try to sum (or take `.corr`, or whatever) between two DataFrames without row labels, then:
- either they're the same length, in which case the operation succeeds (and aligns by position);
- or they're not, in which case it fails loudly and clearly.

> My biggest complaint with how it currently automagically works and doesn't warn you when probably you made a mistake.

pandas doesn't know what you're trying to do, so it can't warn when you "probably made a mistake". An option to warn whenever indexes aren't aligned would be a separate discussion - if you want to take it further, please open a new issue and I'd be happy to take a look. Let's keep this issue's discussion focused on not having row labels.
> There's already [lot's of options](https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html) which can be set in pandas

These checks aren't expensive, but they do add up (xref #49771).  I'm more concerned about the maintenance/testing burde they add up to.
Continuing the discussion started by https://github.com/pandas-dev/pandas/pull/49694#issuecomment-1314474678, it's still unclear to me which ops result in no index vs an index here. I think the closest thing to a demarcation criteria is from @jorisvandenbossche:

> So you could say that the user "asks for this", when it does an operation where the columns names are used for row labels.

I think this is a reasonable idea, but seems to me it leaves for much debate as to what the user is asking for. I think it'd be helpful to be able to list out the operations that would give the NoRowIndex user a RowIndex.

Edit: Just to make it explicit, I think the user is "asking for an index" with `DataFrame.value_counts`, yet it doesn't meet the criterion that "column names are used for row labels". What other ops are there like this? `DataFrame.describe` also comes to mind.
I don't want to bike shed too hard on the name, but I don't find `NoRowIndex` very intuitive. The format of all concrete `Index` subclasses is `DataTypeIndex` so that you can read the name as an index with datatype values, e.g., Int64Index is an index with int64 values. When I try and use the same metaphor here, an index of no rows doesn't seem to fit the pattern. 

Despite not finding the naming clear, I don't have any slam-dunk alternative. Some that came to mind are:

* `PositionalIndex` - Doesn't fit the pattern that well, but completely describes what it does.
* `LablelessIndex` or `NoLabelIndex` - seem to fit the pattern and describe the index's key feature.
* `NullIndex` or `NAIndex` - sort of fits the pattern (as if each label is `None` or `pd.NA`), but is perhaps a bit too technical.
* `IndexLess` - doesn't fit but is descriptive


A few things to respond to, thanks everyone for keeping this going

>  I'm more concerned about the maintenance/testing burde they add up to.

Yeah I think @rhshadrach shared the same concern. If the general feeling ends up being that the potential benefits don't outweigh the maintenance burden, then we'll change the status to "rejected" - it'll still have been a useful exercise, and this PDEP would serve as a future reference

---

Regarding which operations return and index vs which don't - in the end I kept that out of this proposal. This PDEP would only be about a class which allows a DataFrame/Series to act as if it didn't have row labels. Operations which insert an index would continue to do so

---

Agree about the name. Joris has suggested above that this class shouldn't be public facing at all, and that users would just see `df.index` as `None`. A good name would still be necessary for internal use though, and I like your `PositionalIndex` suggestion

---

I'll update with another revision soon-ish
> I don't want to bike shed too hard on the name, but I don't find `NoRowIndex` very intuitive. The format of all concrete `Index` subclasses is `DataTypeIndex` so that you can read the name as an index with datatype values, e.g., Int64Index is an index with int64 values. When I try and use the same metaphor here, an index of no rows doesn't seem to fit the pattern.
> 
> Despite not finding the naming clear, I don't have any slam-dunk alternative. Some that came to mind are:
> 
> * `PositionalIndex` - Doesn't fit the pattern that well, but completely describes what it does.
> * `LablelessIndex` or `NoLabelIndex` - seem to fit the pattern and describe the index's key feature.
> * `NullIndex` or `NAIndex` - sort of fits the pattern (as if each label is `None` or `pd.NA`), but is perhaps a bit too technical.
> * `IndexLess` - doesn't fit but is descriptive

Maybe `EmptyIndex` ??
> * StackOverflow requires minimal reproducible examples (which'd need to include any options which were set), so this shouldn't be an issue there. There's already [lot's of options](https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html) which can be set in pandas, I've not seen them cause confusion on SO;

The issue with the options you quoted, is that they are largely used by experts and advanced usage, which is why you don't see a lot of traffic around SO on those.  

This option would be theoretically used by someone who was a non expert (eg, "data alignment is confusing").  They run into some problem, see that it fixed it after adding it, and potentially forget they ever set it.   

My vote, fwliw, is for addressing the core issue, which is arbitrary data alignment.  I honestly think just a bit more documentation here and there will do the trick.
If you set this option, then `DataFrame` and `Series` objects would look different, i.e.:
```python
In [8]: df1  # with this option - note how there are no row labels
Out[8]:
 a  b
 1  4
 2  5
```
instead of
```python
In [3]: df1  # normal pandas - rows are labelled
Out[3]:
   a  b
0  1  4
1  2  5
```
. The `.index` property would be different. It would be clear that it's been set.

> My vote, fwliw, is for addressing the core issue, which is arbitrary data alignment. I honestly think just a bit more documentation here and there will do the trick.

Sorry but I need to insist that we keep this discussion focused on PDEP0005. If you want to open a separate issue about documenting alignment, then you're more than welcome to, and I'd be happy to take a look, but not here please
I'd like to mention documentation again, for emphasis.  A someone who found the automatic data alignment 'confusing', I think I offer a unique perspective to this problem (at least on this thread).  For example, I'm currently dealing with numpy shape.  When looking at the documentation for np.shape it would have been helpful if it simply mentioned that it was row X columns when I looked np.shape rather the just "array dimensions" (same amount of characters!).  Smirk, if you wish, but these minor details here and there repeated can go along way to helping folks new to these frameworks out.

For this problem, a quick note in different operations such as sum could say 

`
*Note automatic data alignment:  as with all pandas operations, automatic data alignment is performed.  If sum does not find values with matching indicies than NaN will used as the total.   
`

An example would even be even more awesome.


Note that my text is probably very poor, but ideally it would be written in a way to make alignment less confusing to new users.




Thanks @blazespinnaker - as that's a different topic, I've opened https://github.com/pandas-dev/pandas/issues/49939 where we can discuss better documenting alignment. Let's move that conversation there please, and keep this one focused on a labelless index
Thanks all for your reviews and comments, I really appreciate it 🙏 

After several discussions, it's become clear that there's not enough support for this to pass, at least in its current state. Hence, I've marked the proposal as 'rejected', and have explained the reasons why in a section at the bottom of the page.

As mentioned in the PDEP, this has still been a useful exercise which has resulted in 2 related proposals. It may be that the idea of not having an index can make it into pandas in the future, but it would require a more comprehensive set of changes to be made together. This particular version of the proposal, however, doesn't have a chance of making it.

No hard feelings, and thank you all for the time you dedicated to reviewing this 🤗 
> there were some comments from my previous review that weren't addressed

Apologies, and thanks for your patience + comments

I think I've addressed the comments - this is withdrawn anyway so arguably doesn't need to be perfect

@Dr-Irv is it OK to ~put this out of misery~ merge?