added random_split in generic.py, for DataFrames etc.
State: closed

Added a method random_split() for NDFrames, to split the Frame into several frames according to one axis. 

Basic use: for train/test or train/validation/test splitting of a dataframe.

Note: I'm not sure this feature fits well in NDFrame. If you think this feature can be added, I'll add more tests and docs.


Labels:
name: API Design

Comments:
A previous similar PR was closed as out of scope for pandas: #6687

@lukovnikov you can go through the discussion there, and see if you have arguments that this should be included, then we can always reconsider.

@jorisvandenbossche this is more general than train/test split, you can make as many random splits as you want (and this is something you also mentioned for #6687).

And although indexing yourself or using groupby like in #6687 can get you the same, I find having this method much easier, it's a no-brainer that seems interesting if you don't want to depend on sklearn or other libraries for easy splitting or think about it yourself. The .sample() method is also a no-brainer for selecting random samples which could've been done by the users and could also be excluded.

I can also make a general .split() method, with a randomness option but with the same notion of relative sizes of the splits (which, without randomness, are just taken from the beginning).

Edit: I'm not aware what statsmodels provides, maybe they also have something. Anyway, in my opinion an optionally random .split() in pandas would still be interesting, even if it only prevents other dependencies.

Edit: generalized it to an optionally random .split() method.

From an API design perspective, I think it's much clean to have `.random_split()` rather than a completely general function that lets flags change every bit of behavior it has. Pandas has too many of those already.

On the whole, I think this _could_ be a reasonable thing to add to pandas as long as it's called `random_split` and not the hopelessly ambiguous `split`. Pandas shouldn't have detailed stats or machine learning routines, but randomized splits really are data science 101.

If we do want to go forward with this, it's worth looking at the `DataFrame.random_split()` method that was added to dask.dataframe to verify the consistency of the API: https://github.com/blaze/dask/pull/276

#7387 is the master issue for splitting. Pls have a read over the many usecases / features that are needed for general splitting/partitioning.

Random splitting is a specialization of this `.split` and should not be a separate function, merely an argument here. You are already having a number of arguments, I don't find another to be an issue. It is more confusing IMHO to have to look for random (pun intended) functions.

further this API needs to start similarly to [np.split](http://docs.scipy.org/doc/numpy/reference/generated/numpy.split.html)

```
def split(self, n=None, weights=None, random=None, axis=0)
```

could be a decent start.

@jreback yeah, but the np.split() is a function on np, not on an array so the first argument is already the pandas object we're calling it on. The second argument of np.split() is a int or list of ints, which specifies the number of _equal_ splits or the _indices_ along which to split. In the generalized .split() method here, I aim to accept weights, which are proportionate to the desired relative sizes of the output splits. Accepting a list of indices as arguments in my opinion is a little superfluous. If you already have those indices, why don't you just use them? Anyway, they could be added as a yet another option.

I understand your concern to not to overcrowd the API but I'm more with @shoyer on the issue of a separate random_split() method. It's easier to find a dedicated function with limited number of arguments/switches than reading through long argument descriptions of over-argumented functions.

@shoyer First it was random_split() but in the second commit I made a split() out of it. In my opinion, a dedicated method is also better.

We could make a collection of splitting methods:
- isplit() that accepts indices (like numpy.split())
- rsplit() that accepts weights for relative split sizes
- random_split() that accepts weights but behaves randomly
- split(), a method with switching arguments for randomness and indices/weights

OR, similar to what @jreback mentions for #7387 (if I understood that discussion correctly) we could have
- a split() on GroupBy
- a shuffle() method on a Frame that returns a GroupBy or a Shuffler Grouper that can be passed to .groupby()
- but then we might still want at least a simple split() method on a Frame, for non-random splitting (maybe following the numpy semantics) OR a Partitioner Grouper as an argument to .groupby(), like @jreback suggested for #7387.

In my opinion, the first option is easier for beginners and is a no-brainer (and thus better for wider adoption of pandas) while the second option seems like a cleaner implementation but is less of a no-brainer.

> but the np.split() is a function on np, not on an array so the first argument is already the pandas object we're calling it on. 

this is the point of the buffer interface, so it in fact is useful, no need to repeat things, except where pandas can add value. This is where a `.split()` method comes in

we already have `.sample()` so no need for shuffle.
I think the only API expansion needed here is `.split()`

this can accomplish all of the objectives with a minimum of keywords. Adding a plethora of methods is just plain confusing and bloats the API. 

This is a very similar method in nature to `.sample` so it should follow a similar pattern.

@jreback do you mean a `.split()` on a GroupBy or a `.split()` on a Frame?

Edit: the `.split()` I implemented on the Frame (see second commit) already is optionally random. So if I add an optional `indexes` argument (that can not be used with the `weights` argument), you think it's good? The `indexes` argument will have the same semantics as the indexes argument for `np.split()` whereas the `weights` argument allows to specify proportions you want (maybe should be renamed to `proportions`).

`.split` could exist on a groupby, though not really sure of the semantics, as you can trivially

`[ grp for _, grp in df.groupby(....) ]` is basically what you want, though
`df.groupby(....).split()` might make sense for some applications here

how can random splits be realized if `.split()` is attached to a `GroupBy`? Then you also need some RandomGrouper to feed to `.groupby()`?

So my question is: can we put a general `.split()` on a Frame, in which case I can quickly extend the functionality to have `numpy.split()`-like indexes or you don't want `.split()` to become a part of a Frame's API?

it becomes trivial actually (at least for random, not partitioned)

`df.groupby(np.random.randint(0,len(df),size=df)).split()`

I think `.split()` on the frame is ok, but since we already have `.groupby` which handles partitioned, column splitting and such, it makes a lot more sense on groupby.

though it _does_ have a different return type (e.g. `.groupby......` returns a pandas object), now it would return a list of them (but I think this is ok here)

and what I would do to support 'kinds' of splitting is make things like:

`Random`
`Partition`

which are sub-classes of `pd.Grouper` to control exactly how splitting is done. Then you don't have lots of arguments, and it is pretty semantic

```
df.groupby(Random(1234)).split()
df.groupby(Partition([1,2,3])).split()
```

yes, this seems like a cleaner implementation but maybe a `.split()` on a Frame that uses `.groupby()` internally would also be interesting as a convenience method?

I could be convinced of that as well

so, I hope this implementation is good (if someone could quickly review the code, that would be helpful)
still need to write proper tests and test it though

I am personally not a fan of adding extra objects like `Random` and `Partition` to use with groupby. Using objects for this is more cognitive load than a method for users, and further I also find it a bit unintuitive to have to use groupby when I want to split randomly.

(apart from that, a `split` method on groupby can be useful, but I find it a bit a separate issue)

I agree with @jorisvandenbossche that I would rather not add extra objects to the API exclusively for use with groupby.

I think a better approach is create alternative methods (other than `.groupby`) that return `GroupBy` like objects. For example, it would make a lot of sense to make this the case for `resample` (removing the need to use `TimeGrouper`), e.g., `df.resample('24H').sum()` instead of `df.resample('24H', how='sum')` (this is a whole different issue, for course).

However, it is not clear to me why a groupby object is an intuitive result from a method that produces random partitions or splits. What is the actual use case here? Would you really want to write something like `df.groupby(RandomPartition({'train': 0.7, 'test': 0.3})).apply(model.predict)`? That doesn't look very readable to me. I would rather write:

```
test, train = df.random_split([0.7, 0.3])
test_predictions = model.predict(test)
train_prediction = model.predict(train)
```

would be 

train, test = df.groupby(RandomPartition(( 0.7, .3)).split()

in resample
it's quite easy to return the GroupBy
but I think impossible to change and maintain back compatible

eg df.resample('24h') defaults to .mean() (if it was None) then this would work

I agree that there's no easy way to change resample in a backwards compatible way. Perhaps `how='grouped'` would be a reasonable way to expose this.

We could include both approaches (general `.split()` on a Frame and the new Groupers with `.split()` on a GroupBy) in the API. The `.split()` on the GroupBy could prove useful (and is trivial anyway) as it explicitly provides semantics natural for grouped Frame. Probably most people will just use a `.split()` on the Frame though because it's easier to find. However, if someone wants some advanced (random) splitting functionality (wouldn't know what though), one could easily subclass `RandomGrouper` or `OrderedGrouper` and use it with `.groupby()`.

What do you think?

closing, but if you want to propose a nice API we can re-open