Feature: Memory Layout of DataFrames ndarrays can be set/changed
State: closed

- [ ] closes #26502
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @TannhauserGate42! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









* In the file [`sandbox.py`](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py):

> [Line 13:38](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L13): [E251](https://duckduckgo.com/?q=pep8%20E251) unexpected spaces around keyword / parameter equals
> [Line 13:40](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L13): [E251](https://duckduckgo.com/?q=pep8%20E251) unexpected spaces around keyword / parameter equals
> [Line 13:80](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L13): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (134 > 79 characters)
> [Line 18:80](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L18): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (86 > 79 characters)
> [Line 22:80](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L22): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (94 > 79 characters)
> [Line 26:80](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L26): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (108 > 79 characters)
> [Line 30:80](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L30): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (95 > 79 characters)
> [Line 48:38](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L48): [E251](https://duckduckgo.com/?q=pep8%20E251) unexpected spaces around keyword / parameter equals
> [Line 48:40](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L48): [E251](https://duckduckgo.com/?q=pep8%20E251) unexpected spaces around keyword / parameter equals
> [Line 48:80](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L48): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (134 > 79 characters)
> [Line 53:80](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L53): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (86 > 79 characters)
> [Line 57:80](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L57): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (94 > 79 characters)
> [Line 61:80](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L61): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (108 > 79 characters)
> [Line 65:80](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L65): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (95 > 79 characters)
> [Line 68:1](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L68): [E266](https://duckduckgo.com/?q=pep8%20E266) too many leading '#' for block comment
> [Line 73:47](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L73): [W292](https://duckduckgo.com/?q=pep8%20W292) no newline at end of file
> [Line 73:47](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L73): [W292](https://duckduckgo.com/?q=pep8%20W292) no newline at end of file
> [Line 73:47](https://github.com/pandas-dev/pandas/blob/bdc5491b58d0b07606b5e2aaa95a02d0ce9c446f/sandbox.py#L73): [W292](https://duckduckgo.com/?q=pep8%20W292) no newline at end of file



##### Comment last updated at 2019-05-26 22:14:28 UTC
At a glance, this doesn’t seem to integrate well with the rest of pandas. I don’t think a library-level global is the way to go. I’d recommend writing a proposal summarizing the issue and a design for a fix first. 

> On May 26, 2019, at 07:19, PEP 8 Speaks <notifications@github.com> wrote:
> 
> Hello @TannhauserGate42! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:
> 
> In the file pandas/core/frame.py:
> Line 411:80: E501 line too long (86 > 79 characters)
> Line 530:80: E501 line too long (89 > 79 characters)
> Line 7445:80: E501 line too long (91 > 79 characters)
> 
> In the file pandas/core/layout.py:
> Line 2:80: E501 line too long (80 > 79 characters)
> Line 69:80: E501 line too long (83 > 79 characters)
> Line 163:80: E501 line too long (90 > 79 characters)
> Line 165:80: E501 line too long (80 > 79 characters)
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub, or mute the thread.

> so you would need
> 
> * substantial tests l / asserts in the code to prove this actually works
> * full asv runs to show this doesn’t reduce performance anywhere and increases in. some cases
> * this looks very java-esque code - is there a reason you did it this way?

I apologise for my simple code.

In any case, the code is quite explicit and easy to read. Java, don't say that ...

It was not easy to find the source of the current inconsistency in pandas (first aggregation is fast, and all subsequent aggregations are slow, due to implicit and inconsistent change to F-major-ordering by a side effect)

In any case, the current pandas versions have a 'bug': pandas flips uncontrolled from C to F major order. It writes to HDFStores in C-major-order and reads as such. It flips and consumes time to F due to a transpose and subsequent copy operation which leads implicitly to F-major-ordering (I can explain this elsewhere if you are interested)

The problem with this is not only that performance suffers, as I have shown in my examples. But the developer / analyst cannot choose the memory layout appropriate for his analysis. 

This is a shortcoming, together with the inconsistent layout flip it is a no-go.





> At a glance, this doesn’t seem to integrate well with the rest of pandas. I don’t think a library-level global is the way to go. I’d recommend writing a proposal summarizing the issue and a design for a fix first.

For sure, it does not fit nicely into the current pandas, but it fixes a but and leads to much higher performance. 

The new code is very well separated and can easily be removed, once a better concept is available. 

Besides implementatory details, a global setting/switch is maybe even the right way to go. You can switch the behaviour for all your instances. Using pandas in a business application, this is safest. You don't want to keep track where you use which layout. Within a notebook you can do the same. In both cases you can chose the 'K' option, which means keep the major order of the instance to copy. The provided implementation fullfills this requirement. The current pandas does none of them. This is especially infortunate, because numpy does very well with it and taking it over would not be a great deal.

@TannhauserGate42 you are missing the point
if you claim performance improvements then you need you need to show them and show that there are no regressions (which is very doubtful)

this is also on the wrong branch; it must be a PR to master
but before this is even considered you need to show perf
> this is also on the wrong branch; it must be a PR to master
> but before this is even considered you need to show perf

Since we have 0.23.4 in our company, I branched out from that one. 
(had problems with tornado, dask, distributed, numpy and pandas on Azure and could not upgrade to a higher version) 
But, I could bring it into master too, sooner or later. 
> @TannhauserGate42 you are missing the point
> if you claim performance improvements then you need you need to show them and show that there are no regressions (which is very doubtful)

I see that point. Sure.

Are there performance tests already in place in the test suite of pandas? I can come up with some. But don't know how to integrate in the test suite ...

I would like to create a build which I can use in my application ... Is it possible to create one via GitHub from my branch? I would load it into artifactory and create a build of my two applications then ... 

There will be one regression for sure!!! But each bugfix shows a regression.

The current pandas is able to create a C-ordered frame and will implicitly convert it to a F-ordered. Depending on the performance tests already in place, these may perform better or worse due this behaviour. Now one explicitly choses the order or uses at least the consistent order set per default.

Suppose you have a test where you create a fixture and aggregate two times on the same frame. A pre-aggregation and a subsequent aggregation. Then you might aggregate on a C-ordered array first and on a F-ordered in the second place. With the new default setting one would preaggregate already on a F-ordered and then again on a F-ordered array. So I expect for sure a regression. If one sets the order with the new ArrayLayout to C, then one would also observe a change. Whether it performs better or worse depends on the use case, whether one aggregate on the rows or columns.

But let's check it. 

When I'm back in the office tomorrow, we will integrate it into our application and let's see ... 
Should it work, I will continue here.
http://pandas-docs.github.io/pandas-docs-travis/development/contributing.html#running-the-performance-test-suite

as i said this change would need to be on top of master