Repository: databricks/koalas

Issue:
Convert global or limit-related variables into configurations.
https://github.com/databricks/koalas/commit/94adf06d622aaac1eb05c03407db8a1abc01632c added configuration base like Pandas. See https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html

Currently we have multiple variables that should be converted into configuration values. We should identify and convert it to configuration _all_.

- [x] Support typed values in configurations (#714)
- [x] Add 'display.max_rows' (#714)
- [x] Add 'compute.shortcut_limit' (#717) https://github.com/databricks/koalas/blob/8360c586f5ad4c8847fe394eefbd77b48d95f687/databricks/koalas/frame.py#L373-L379
- [x] Add 'compute.max_rows'  (#721) https://github.com/databricks/koalas/blob/8360c586f5ad4c8847fe394eefbd77b48d95f687/databricks/koalas/frame.py#L1463-L1470 (I think we should remove `limit` parameter).
- [x] Add 'compute.shortcut_limit' (#717) https://github.com/databricks/koalas/blob/8360c586f5ad4c8847fe394eefbd77b48d95f687/databricks/koalas/groupby.py#L806-L812
- [x] Add 'compute.shortcut_limit' (#717) https://github.com/databricks/koalas/blob/8360c586f5ad4c8847fe394eefbd77b48d95f687/databricks/koalas/groupby.py#L1512-L1518
- [x] Add 'plotting.max_rows'  (#728) https://github.com/databricks/koalas/blob/8360c586f5ad4c8847fe394eefbd77b48d95f687/databricks/koalas/plot.py#L54
- [x] Add 'plotting.sample_ratio' (#737) https://github.com/databricks/koalas/blob/8360c586f5ad4c8847fe394eefbd77b48d95f687/databricks/koalas/plot.py#L85
- [x] Add 'compute.ops_on_diff_frames' (#725) https://github.com/databricks/koalas/blob/8360c586f5ad4c8847fe394eefbd77b48d95f687/databricks/koalas/utils.py#L63
- [x] Add 'compute.default_index_type' (#723) https://github.com/databricks/koalas/blob/8360c586f5ad4c8847fe394eefbd77b48d95f687/databricks/koalas/internal.py#L477
- [x] Add 'compute.max_rows' (#736) https://github.com/databricks/koalas/pull/712/files#diff-5ba644c40e914732959b9b4867f30c53R1644
- [x] Add 'compute.shortcut_limit' (#739) https://github.com/databricks/koalas/pull/732/files#diff-5ba644c40e914732959b9b4867f30c53R1605
- [ ] Add documentation (#743). See https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html


Discussion:
@itholic are you interested in working of some of them? Don't have to do everything in one go.
cc @RainFung and @charlesdong1991 too just in case you guys are interested in this.
nice one! what a coincidence! i asked if `max_rows` should be flexible or not yesterday, and today issue is here 😆  i am working on `scatter` and `hexbin` plots now, but would like to take `Add 'plotting.max_rows'` and `Add 'plotting.sample_ratio'` later if you don't mind. @HyukjinKwon 
Please go ahead for both items! BTW, feel free to suggest different names if you have some kind of strong preferences.
@HyukjinKwon As for `SPARK_KOALAS_AUTOPATCH` and `KOALAS_USAGE_LOGGER`, we need them when initializing Koalas, so we shouldn't use the configurations.
I removed them from the list in the description.
Thanks!

Resolving patch:
diff --git a/databricks/koalas/config.py b/databricks/koalas/config.py
--- a/databricks/koalas/config.py
+++ b/databricks/koalas/config.py
@@ -105,6 +105,12 @@ def validate(self, v: Any) -> None:
 
 
 # Available options.
+#
+# NOTE: if you are fixing or adding an option here, make sure you execute `show_options()` and
+#     copy & paste the results into show_options 'docs/source/user_guide/options.rst' as well.
+#     See the examples below:
+#     >>> from databricks.koalas.config import show_options
+#     >>> show_options()
 _options = [
     Option(
         key='display.max_rows',
@@ -135,7 +141,7 @@ def validate(self, v: Any) -> None:
     Option(
         key='compute.shortcut_limit',
         doc=(
-            "'compute.shortcut_limit' sets the limit for a shortcut."
+            "'compute.shortcut_limit' sets the limit for a shortcut. "
             "It computes specified number of rows and use its schema. When the dataframe "
             "length is larger than this limit, Koalas uses PySpark to compute."),
         default=1000,
@@ -197,6 +203,29 @@ class OptionError(AttributeError, KeyError):
     pass
 
 
+def show_options():
+    """
+    Make a pretty table that can be copied and pasted into public documentation.
+    This is currently for an internal purpose.
+    """
+
+    import textwrap
+
+    header = ["Option", "Default", "Description"]
+    row_format = "{:<31} {:<14} {:<53}"
+
+    print(row_format.format("=" * 31, "=" * 14, "=" * 53))
+    print(row_format.format(*header))
+    print(row_format.format("=" * 31, "=" * 14, "=" * 53))
+
+    for option in _options:
+        doc = textwrap.fill(option.doc, 53)
+        formatted = "".join([line + "\n" + (" " * 47) for line in doc.split("\n")]).rstrip()
+        print(row_format.format(option.key, repr(option.default), formatted))
+
+    print(row_format.format("=" * 31, "=" * 14, "=" * 53))
+
+
 def get_option(key: str, default: Union[Any, _NoValueType] = _NoValue) -> Any:
     """
     Retrieves the value of the specified option.
diff --git a/databricks/koalas/internal.py b/databricks/koalas/internal.py
--- a/databricks/koalas/internal.py
+++ b/databricks/koalas/internal.py
@@ -416,62 +416,7 @@ def attach_default_index(sdf):
         """
         This method attaches a default index to Spark DataFrame. Spark does not have the index
         notion so corresponding column should be generated.
-
-        There are three types of default index can be configured by `compute.default_index_type`
-
-        - sequence: It implements a sequence that increases one by one, by Window function without
-            specifying partition. Therefore, it ends up with whole partition in single node.
-            This index type should be avoided when the data is large. This is default.
-            See example below:
-
-            >>> ks.range(3).index  # doctest: +SKIP
-            Int64Index([0, 1, 2], dtype='int64')
-
-            This is conceptually equivalent to the Spark example as below:
-
-            >>> from pyspark.sql import functions as F, Window
-            >>> spark_df = ks.range(3).to_spark()
-            >>> sequential_index = F.row_number().over(
-            ...    Window.orderBy(F.monotonically_increasing_id().asc())) - 1
-            >>> spark_df.select(sequential_index).rdd.map(lambda r: r[0]).collect()
-            [0, 1, 2]
-
-        - distributed-sequence: It implements a sequence that increases one by one, by group-by and
-            group-map approach. It still generates the sequential index globally.
-            If the default index must be the sequence in a large dataset, this
-            index has to be used.
-            Note that if more data are added to the data source after creating this index,
-            then it does not guarantee the sequential index.
-            See example below:
-
-            >>> ks.range(3).index  # doctest: +SKIP
-            Int64Index([0, 1, 2], dtype='int64')
-
-            This is conceptually equivalent to the Spark example as below:
-
-            >>> spark_df = ks.range(3).to_spark()
-            >>> spark_df.rdd.zipWithIndex().map(lambda p: p[1]).collect()
-            [0, 1, 2]
-
-        - distributed: It implements a monotonically increasing sequence simply by using
-            Spark's `monotonically_increasing_id` function. If the index does not have to be
-            a sequence that increases one by one, this index should be used.
-            Performance-wise, this index almost does not have any penalty comparing to
-            other index types. Note that we cannot use this type of index for combining
-            two dataframes because it is not guaranteed to have the same indexes in two
-            dataframes. See example below:
-
-            >>> ks.range(3).index  # doctest: +SKIP
-            Int64Index([25769803776, 60129542144, 94489280512], dtype='int64')
-
-            This is conceptually equivalent to the Spark example as below:
-
-            >>> from pyspark.sql import functions as F
-            >>> spark_df = ks.range(3).to_spark()
-            >>> spark_df.select(F.monotonically_increasing_id()) \
-            ...     .rdd.map(lambda r: r[0]).collect()  # doctest: +SKIP
-            [25769803776, 60129542144, 94489280512]
-
+        There are several types of default index can be configured by `compute.default_index_type`.
         """
         default_index_type = get_option("compute.default_index_type")
         if default_index_type == "sequence":


Test patch: