Repository: dask/dask

Issue:
The behavior of "sort" option in "from_pandas" method is not very clear
Can you please enhance the documentation around the **sort** option in the **from_pandas** method. Currently I don't know what it will do. On which column is the sort going to happen? If I specify **npartitions=10** then will it sort it by the first column and then chunk by chunk and assign it to different partitions making sure all the observations for the same category in the sorted column (say customer ID) will go into a single partition?


Discussion:
For the [from_pandas docs](https://docs.dask.org/en/latest/dataframe-api.html#dask.dataframe.from_pandas), we have the following for `sort`:

```
    sort: bool
        Sort input first to obtain cleanly divided partitions or don't sort and
        don't get cleanly divided partitions
```

This means that the input pandas DataFrame will be sorted by the **index** before conversion to a dask collection.  This ensures a monotonically increasing index, enabling a clean `divisions` calculation (a `dask.dataframe.DataFrame` object uses a `divisions` attribute to help avoid unnecessary data movement when sorting/merging/etc).

Would it be helpful to clarify that the input will be sorted by the **index**?  Also, do you have an example where this is not actually behaving correctly?
Let me explain my situation. I have a huge dataframe which has a customer id column. Initially the data frame came from multiple  csv.gz splits where customer id was scattered across the splits. Trying to sort it inside Dask dataframe was huge pain - so I used Pandas to read in the entire dataframe and sort it by customer id. Now I want to push it into dask dataframe so that I can use distributed processing capability. But to do so, I need to specify the partitions and I am worried - the from_pandas is going to mess up the sort that I had done on my dataframe. How can I ensure this is avoided?
>Trying to sort it inside Dask dataframe was huge pain

The only widely supported way to "sort" in `dask.dataframe` is by setting the index to the column you want to sort by (e.g. `ddf.set_index("column-I-want-sorted")`)

>I need to specify the partitions and I am worried - the from_pandas is going to mess up the sort that I had done on my dataframe. How can I ensure this is avoided?

You do need to specify the number of partitions you want, but your data does not need to be reordered.  As I mentioned above, by default, `from_pandas` will sort your data so that the **index** is in ascending order.  However, your data will **not** be reordered if you specify `sort=False` or if your index is already sorted (i.e. if you reset the index before calling `from_pandas`).
Okay that makes sense @rjzamora. Thanks! Going back to your first message - yes - mentioning explicitly that sorting will happen on the index makes it clear for the reader.

Resolving patch:
diff --git a/dask/dataframe/io/io.py b/dask/dataframe/io/io.py
--- a/dask/dataframe/io/io.py
+++ b/dask/dataframe/io/io.py
@@ -137,7 +137,11 @@ def from_pandas(data, npartitions=None, chunksize=None, sort=True, name=None):
 
     This splits an in-memory Pandas dataframe into several parts and constructs
     a dask.dataframe from those parts on which Dask.dataframe can operate in
-    parallel.
+    parallel.  By default, the input dataframe will be sorted by the index to
+    produce cleanly-divided partitions (with known divisions).  To preserve the
+    input ordering, make sure the input index is monotonically-increasing. The
+    ``sort=False`` option will also avoid reordering, but will not result in
+    known divisions.
 
     Note that, despite parallelism, Dask.dataframe may not always be faster
     than Pandas.  We recommend that you stay with Pandas for as long as
@@ -154,8 +158,9 @@ def from_pandas(data, npartitions=None, chunksize=None, sort=True, name=None):
     chunksize : int, optional
         The number of rows per index partition to use.
     sort: bool
-        Sort input first to obtain cleanly divided partitions or don't sort and
-        don't get cleanly divided partitions
+        Sort the input by index first to obtain cleanly divided partitions
+        (with known divisions).  If False, the input will not be sorted, and
+        all divisions will be set to None. Default is True.
     name: string, optional
         An optional keyname for the dataframe.  Defaults to hashing the input
 


Test patch: