Repository: openml/openml-python

Issue:
The API docs should document the openml config options
#### Description
The API Docs: http://openml.github.io/openml-python/master/api.html
do not cover the API config options, such as 'server, api_key, verbosity,...'

Could this be added?

#### Steps/Code to Reproduce
Try to look up the OpenML-Python config options on 
http://openml.github.io/openml-python/master/api.html

#### Expected Results
Maybe a section 'Configuration' next to 'Top level Classes', 'Extensions' and 'Modules'?

#### Actual Results
Nothing there.

#### Versions
Irrelevant

Discussion:
codecov-commenter: # [Codecov](https://codecov.io/gh/openml/openml-python/pull/931?src=pr&el=h1) Report
> Merging [#931](https://codecov.io/gh/openml/openml-python/pull/931?src=pr&el=desc) into [develop](https://codecov.io/gh/openml/openml-python/commit/2bfd581e212e7ef91ec65d1d7976f0984f72724a&el=desc) will **increase** coverage by `0.03%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/openml/openml-python/pull/931/graphs/tree.svg?width=650&height=150&src=pr&token=IlImE4jvlP)](https://codecov.io/gh/openml/openml-python/pull/931?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           develop     #931      +/-   ##
===========================================
+ Coverage    87.65%   87.69%   +0.03%     
===========================================
  Files           37       37              
  Lines         4383     4397      +14     
===========================================
+ Hits          3842     3856      +14     
  Misses         541      541              
```


| [Impacted Files](https://codecov.io/gh/openml/openml-python/pull/931?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [openml/datasets/dataset.py](https://codecov.io/gh/openml/openml-python/pull/931/diff?src=pr&el=tree#diff-b3Blbm1sL2RhdGFzZXRzL2RhdGFzZXQucHk=) | `82.22% <0.00%> (+0.63%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/openml/openml-python/pull/931?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/openml/openml-python/pull/931?src=pr&el=footer). Last update [2bfd581...974c091](https://codecov.io/gh/openml/openml-python/pull/931?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/doc/usage.rst b/doc/usage.rst
index 36c8584f..d7ad0d52 100644
--- a/doc/usage.rst
+++ b/doc/usage.rst
@@ -29,6 +29,35 @@ the OpenML Python connector, followed up by a simple example.
 
 * `Introduction <examples/introduction_tutorial.html>`_
 
+~~~~~~~~~~~~~
+Configuration
+~~~~~~~~~~~~~
+
+The configuration file resides in a directory ``.openml`` in the home
+directory of the user and is called config. It consists of ``key = value`` pairs
+which are separated by newlines. The following keys are defined:
+
+* apikey:
+    * required to access the server. The `OpenML setup <https://openml.github.io/openml-python/master/examples/20_basic/introduction_tutorial.html#authentication>`_ describes how to obtain an API key.
+
+* server:
+    * default: ``http://www.openml.org``. Alternatively, use ``test.openml.org`` for the test server.
+
+* cachedir:
+    * if not given, will default to ``~/.openml/cache``
+
+* avoid_duplicate_runs:
+    * if set to ``True``, when ``run_flow_on_task`` or similar methods are called a lookup is performed to see if there already exists such a run on the server. If so, download those results instead.
+    * if not given, will default to ``True``.
+
+* connection_n_retries:
+    * number of connection retries.
+    * default: 2. Maximum number of retries: 20.
+
+* verbosity:
+    * 0: normal output
+    * 1: info output
+    * 2: debug output
 
 ~~~~~~~~~~~~
 Key concepts
diff --git a/examples/30_extended/fetch_evaluations_tutorial.py b/examples/30_extended/fetch_evaluations_tutorial.py
index de636e07..2823eabf 100644
--- a/examples/30_extended/fetch_evaluations_tutorial.py
+++ b/examples/30_extended/fetch_evaluations_tutorial.py
@@ -63,7 +63,7 @@ print(task)
 
 metric = "predictive_accuracy"
 evals = openml.evaluations.list_evaluations(
-    function=metric, task=[task_id], output_format="dataframe"
+    function=metric, tasks=[task_id], output_format="dataframe"
 )
 # Displaying the first 10 rows
 print(evals.head(n=10))
@@ -162,7 +162,7 @@ for i in range(top_n):
 # List evaluations in descending order based on predictive_accuracy with
 # hyperparameters
 evals_setups = openml.evaluations.list_evaluations_setups(
-    function="predictive_accuracy", task=[31], size=100, sort_order="desc"
+    function="predictive_accuracy", tasks=[31], size=100, sort_order="desc"
 )
 
 ""
@@ -173,7 +173,7 @@ print(evals_setups.head())
 # with hyperparameters. parameters_in_separate_columns returns parameters in
 # separate columns
 evals_setups = openml.evaluations.list_evaluations_setups(
-    function="predictive_accuracy", flow=[6767], size=100, parameters_in_separate_columns=True
+    function="predictive_accuracy", flows=[6767], size=100, parameters_in_separate_columns=True
 )
 
 ""
diff --git a/examples/30_extended/plot_svm_hyperparameters_tutorial.py b/examples/30_extended/plot_svm_hyperparameters_tutorial.py
index aac84bcd..e366c56d 100644
--- a/examples/30_extended/plot_svm_hyperparameters_tutorial.py
+++ b/examples/30_extended/plot_svm_hyperparameters_tutorial.py
@@ -20,8 +20,8 @@ import numpy as np
 # uploaded runs (called *setup*).
 df = openml.evaluations.list_evaluations_setups(
     function="predictive_accuracy",
-    flow=[8353],
-    task=[6],
+    flows=[8353],
+    tasks=[6],
     output_format="dataframe",
     # Using this flag incorporates the hyperparameters into the returned dataframe. Otherwise,
     # the dataframe would contain a field ``paramaters`` containing an unparsed dictionary.
diff --git a/examples/40_paper/2018_ida_strang_example.py b/examples/40_paper/2018_ida_strang_example.py
index 74c6fde5..687d973c 100644
--- a/examples/40_paper/2018_ida_strang_example.py
+++ b/examples/40_paper/2018_ida_strang_example.py
@@ -47,7 +47,7 @@ class_values = ["non-linear better", "linear better", "equal"]
 
 # Downloads all evaluation records related to this study
 evaluations = openml.evaluations.list_evaluations(
-    measure, flow=flow_ids, study=study_id, output_format="dataframe"
+    measure, flows=flow_ids, study=study_id, output_format="dataframe"
 )
 # gives us a table with columns data_id, flow1_value, flow2_value
 evaluations = evaluations.pivot(index="data_id", columns="flow_id", values="value").dropna()
diff --git a/examples/40_paper/2018_kdd_rijn_example.py b/examples/40_paper/2018_kdd_rijn_example.py
index e5d998e3..752419ea 100644
--- a/examples/40_paper/2018_kdd_rijn_example.py
+++ b/examples/40_paper/2018_kdd_rijn_example.py
@@ -88,8 +88,8 @@ for idx, task_id in enumerate(suite.tasks):
     # note that we explicitly only include tasks from the benchmark suite that was specified (as per the for-loop)
     evals = openml.evaluations.list_evaluations_setups(
         evaluation_measure,
-        flow=[flow_id],
-        task=[task_id],
+        flows=[flow_id],
+        tasks=[task_id],
         size=limit_per_task,
         output_format="dataframe",
     )
diff --git a/examples/40_paper/2018_neurips_perrone_example.py b/examples/40_paper/2018_neurips_perrone_example.py
index 8639e0a3..60d21211 100644
--- a/examples/40_paper/2018_neurips_perrone_example.py
+++ b/examples/40_paper/2018_neurips_perrone_example.py
@@ -91,9 +91,9 @@ def fetch_evaluations(run_full=False, flow_type="svm", metric="area_under_roc_cu
     # Fetching evaluations
     eval_df = openml.evaluations.list_evaluations_setups(
         function=metric,
-        task=task_ids,
-        flow=[flow_id],
-        uploader=[2702],
+        tasks=task_ids,
+        flows=[flow_id],
+        uploaders=[2702],
         output_format="dataframe",
         parameters_in_separate_columns=True,
     )
diff --git a/openml/evaluations/functions.py b/openml/evaluations/functions.py
index adaf419e..4c17f8ce 100644
--- a/openml/evaluations/functions.py
+++ b/openml/evaluations/functions.py
@@ -17,11 +17,11 @@ def list_evaluations(
     function: str,
     offset: Optional[int] = None,
     size: Optional[int] = None,
-    task: Optional[List] = None,
-    setup: Optional[List] = None,
-    flow: Optional[List] = None,
-    run: Optional[List] = None,
-    uploader: Optional[List] = None,
+    tasks: Optional[List[Union[str, int]]] = None,
+    setups: Optional[List[Union[str, int]]] = None,
+    flows: Optional[List[Union[str, int]]] = None,
+    runs: Optional[List[Union[str, int]]] = None,
+    uploaders: Optional[List[Union[str, int]]] = None,
     tag: Optional[str] = None,
     study: Optional[int] = None,
     per_fold: Optional[bool] = None,
@@ -41,17 +41,18 @@ def list_evaluations(
     size : int, optional
         the maximum number of runs to show
 
-    task : list, optional
-
-    setup: list, optional
-
-    flow : list, optional
-
-    run : list, optional
-
-    uploader : list, optional
-
+    tasks : list[int,str], optional
+        the list of task IDs
+    setups: list[int,str], optional
+        the list of setup IDs
+    flows : list[int,str], optional
+        the list of flow IDs
+    runs :list[int,str], optional
+        the list of run IDs
+    uploaders : list[int,str], optional
+        the list of uploader IDs
     tag : str, optional
+        filter evaluation based on given tag
 
     study : int, optional
 
@@ -85,11 +86,11 @@ def list_evaluations(
         function=function,
         offset=offset,
         size=size,
-        task=task,
-        setup=setup,
-        flow=flow,
-        run=run,
-        uploader=uploader,
+        tasks=tasks,
+        setups=setups,
+        flows=flows,
+        runs=runs,
+        uploaders=uploaders,
         tag=tag,
         study=study,
         sort_order=sort_order,
@@ -99,11 +100,11 @@ def list_evaluations(
 
 def _list_evaluations(
     function: str,
-    task: Optional[List] = None,
-    setup: Optional[List] = None,
-    flow: Optional[List] = None,
-    run: Optional[List] = None,
-    uploader: Optional[List] = None,
+    tasks: Optional[List] = None,
+    setups: Optional[List] = None,
+    flows: Optional[List] = None,
+    runs: Optional[List] = None,
+    uploaders: Optional[List] = None,
     study: Optional[int] = None,
     sort_order: Optional[str] = None,
     output_format: str = "object",
@@ -120,15 +121,16 @@ def _list_evaluations(
     function : str
         the evaluation function. e.g., predictive_accuracy
 
-    task : list, optional
-
-    setup: list, optional
-
-    flow : list, optional
-
-    run : list, optional
-
-    uploader : list, optional
+    tasks : list[int,str], optional
+        the list of task IDs
+    setups: list[int,str], optional
+        the list of setup IDs
+    flows : list[int,str], optional
+        the list of flow IDs
+    runs :list[int,str], optional
+        the list of run IDs
+    uploaders : list[int,str], optional
+        the list of uploader IDs
 
     study : int, optional
 
@@ -155,16 +157,16 @@ def _list_evaluations(
     if kwargs is not None:
         for operator, value in kwargs.items():
             api_call += "/%s/%s" % (operator, value)
-    if task is not None:
-        api_call += "/task/%s" % ",".join([str(int(i)) for i in task])
-    if setup is not None:
-        api_call += "/setup/%s" % ",".join([str(int(i)) for i in setup])
-    if flow is not None:
-        api_call += "/flow/%s" % ",".join([str(int(i)) for i in flow])
-    if run is not None:
-        api_call += "/run/%s" % ",".join([str(int(i)) for i in run])
-    if uploader is not None:
-        api_call += "/uploader/%s" % ",".join([str(int(i)) for i in uploader])
+    if tasks is not None:
+        api_call += "/task/%s" % ",".join([str(int(i)) for i in tasks])
+    if setups is not None:
+        api_call += "/setup/%s" % ",".join([str(int(i)) for i in setups])
+    if flows is not None:
+        api_call += "/flow/%s" % ",".join([str(int(i)) for i in flows])
+    if runs is not None:
+        api_call += "/run/%s" % ",".join([str(int(i)) for i in runs])
+    if uploaders is not None:
+        api_call += "/uploader/%s" % ",".join([str(int(i)) for i in uploaders])
     if study is not None:
         api_call += "/study/%d" % study
     if sort_order is not None:
@@ -276,11 +278,11 @@ def list_evaluations_setups(
     function: str,
     offset: Optional[int] = None,
     size: Optional[int] = None,
-    task: Optional[List] = None,
-    setup: Optional[List] = None,
-    flow: Optional[List] = None,
-    run: Optional[List] = None,
-    uploader: Optional[List] = None,
+    tasks: Optional[List] = None,
+    setups: Optional[List] = None,
+    flows: Optional[List] = None,
+    runs: Optional[List] = None,
+    uploaders: Optional[List] = None,
     tag: Optional[str] = None,
     per_fold: Optional[bool] = None,
     sort_order: Optional[str] = None,
@@ -299,15 +301,15 @@ def list_evaluations_setups(
         the number of runs to skip, starting from the first
     size : int, optional
         the maximum number of runs to show
-    task : list[int], optional
+    tasks : list[int], optional
         the list of task IDs
-    setup: list[int], optional
+    setups: list[int], optional
         the list of setup IDs
-    flow : list[int], optional
+    flows : list[int], optional
         the list of flow IDs
-    run : list[int], optional
+    runs : list[int], optional
         the list of run IDs
-    uploader : list[int], optional
+    uploaders : list[int], optional
         the list of uploader IDs
     tag : str, optional
         filter evaluation based on given tag
@@ -327,7 +329,7 @@ def list_evaluations_setups(
     -------
     dict or dataframe with hyperparameter settings as a list of tuples.
     """
-    if parameters_in_separate_columns and (flow is None or len(flow) != 1):
+    if parameters_in_separate_columns and (flows is None or len(flows) != 1):
         raise ValueError(
             "Can set parameters_in_separate_columns to true " "only for single flow_id"
         )
@@ -337,11 +339,11 @@ def list_evaluations_setups(
         function=function,
         offset=offset,
         size=size,
-        run=run,
-        task=task,
-        setup=setup,
-        flow=flow,
-        uploader=uploader,
+        runs=runs,
+        tasks=tasks,
+        setups=setups,
+        flows=flows,
+        uploaders=uploaders,
         tag=tag,
         per_fold=per_fold,
         sort_order=sort_order,
@@ -359,24 +361,26 @@ def list_evaluations_setups(
         setup_chunks = np.array_split(
             ary=evals["setup_id"].unique(), indices_or_sections=((length - 1) // N) + 1
         )
-        setups = pd.DataFrame()
-        for setup in setup_chunks:
-            result = pd.DataFrame(openml.setups.list_setups(setup=setup, output_format="dataframe"))
+        setup_data = pd.DataFrame()
+        for setups in setup_chunks:
+            result = pd.DataFrame(
+                openml.setups.list_setups(setup=setups, output_format="dataframe")
+            )
             result.drop("flow_id", axis=1, inplace=True)
             # concat resulting setup chunks into single datframe
-            setups = pd.concat([setups, result], ignore_index=True)
+            setup_data = pd.concat([setup_data, result], ignore_index=True)
         parameters = []
         # Convert parameters of setup into list of tuples of (hyperparameter, value)
-        for parameter_dict in setups["parameters"]:
+        for parameter_dict in setup_data["parameters"]:
             if parameter_dict is not None:
                 parameters.append(
                     {param["full_name"]: param["value"] for param in parameter_dict.values()}
                 )
             else:
                 parameters.append({})
-        setups["parameters"] = parameters
+        setup_data["parameters"] = parameters
         # Merge setups with evaluations
-        df = pd.merge(evals, setups, on="setup_id", how="left")
+        df = pd.merge(evals, setup_data, on="setup_id", how="left")
 
     if parameters_in_separate_columns:
         df = pd.concat([df.drop("parameters", axis=1), df["parameters"].apply(pd.Series)], axis=1)


Test patch:
diff --git a/tests/test_evaluations/test_evaluation_functions.py b/tests/test_evaluations/test_evaluation_functions.py
index 6fcaea2d..0127309a 100644
--- a/tests/test_evaluations/test_evaluation_functions.py
+++ b/tests/test_evaluations/test_evaluation_functions.py
@@ -41,7 +41,7 @@ class TestEvaluationFunctions(TestBase):
 
         task_id = 7312
 
-        evaluations = openml.evaluations.list_evaluations("predictive_accuracy", task=[task_id])
+        evaluations = openml.evaluations.list_evaluations("predictive_accuracy", tasks=[task_id])
 
         self.assertGreater(len(evaluations), 100)
         for run_id in evaluations.keys():
@@ -56,7 +56,7 @@ class TestEvaluationFunctions(TestBase):
 
         uploader_id = 16
         evaluations = openml.evaluations.list_evaluations(
-            "predictive_accuracy", uploader=[uploader_id], output_format="dataframe"
+            "predictive_accuracy", uploaders=[uploader_id], output_format="dataframe"
         )
         self.assertEqual(evaluations["uploader"].unique(), [uploader_id])
 
@@ -66,7 +66,7 @@ class TestEvaluationFunctions(TestBase):
         openml.config.server = self.production_server
 
         setup_id = 10
-        evaluations = openml.evaluations.list_evaluations("predictive_accuracy", setup=[setup_id])
+        evaluations = openml.evaluations.list_evaluations("predictive_accuracy", setups=[setup_id])
 
         self.assertGreater(len(evaluations), 50)
         for run_id in evaluations.keys():
@@ -81,7 +81,7 @@ class TestEvaluationFunctions(TestBase):
 
         flow_id = 100
 
-        evaluations = openml.evaluations.list_evaluations("predictive_accuracy", flow=[flow_id])
+        evaluations = openml.evaluations.list_evaluations("predictive_accuracy", flows=[flow_id])
 
         self.assertGreater(len(evaluations), 2)
         for run_id in evaluations.keys():
@@ -96,7 +96,7 @@ class TestEvaluationFunctions(TestBase):
 
         run_id = 12
 
-        evaluations = openml.evaluations.list_evaluations("predictive_accuracy", run=[run_id])
+        evaluations = openml.evaluations.list_evaluations("predictive_accuracy", runs=[run_id])
 
         self.assertEqual(len(evaluations), 1)
         for run_id in evaluations.keys():
@@ -132,9 +132,9 @@ class TestEvaluationFunctions(TestBase):
             "predictive_accuracy",
             size=size,
             offset=0,
-            task=task_ids,
-            flow=flow_ids,
-            uploader=uploader_ids,
+            tasks=task_ids,
+            flows=flow_ids,
+            uploaders=uploader_ids,
             per_fold=True,
         )
 
@@ -149,9 +149,9 @@ class TestEvaluationFunctions(TestBase):
             "predictive_accuracy",
             size=size,
             offset=0,
-            task=task_ids,
-            flow=flow_ids,
-            uploader=uploader_ids,
+            tasks=task_ids,
+            flows=flow_ids,
+            uploaders=uploader_ids,
             per_fold=False,
         )
         for run_id in evaluations.keys():
@@ -164,11 +164,11 @@ class TestEvaluationFunctions(TestBase):
         task_id = 6
         # Get all evaluations of the task
         unsorted_eval = openml.evaluations.list_evaluations(
-            "predictive_accuracy", offset=0, task=[task_id]
+            "predictive_accuracy", offset=0, tasks=[task_id]
         )
         # Get top 10 evaluations of the same task
         sorted_eval = openml.evaluations.list_evaluations(
-            "predictive_accuracy", size=size, offset=0, task=[task_id], sort_order="desc"
+            "predictive_accuracy", size=size, offset=0, tasks=[task_id], sort_order="desc"
         )
         self.assertEqual(len(sorted_eval), size)
         self.assertGreater(len(unsorted_eval), 0)
@@ -191,11 +191,11 @@ class TestEvaluationFunctions(TestBase):
         openml.config.server = self.production_server
         flow_id = [405]
         size = 100
-        evals = self._check_list_evaluation_setups(flow=flow_id, size=size)
+        evals = self._check_list_evaluation_setups(flows=flow_id, size=size)
         # check if parameters in separate columns works
         evals_cols = openml.evaluations.list_evaluations_setups(
             "predictive_accuracy",
-            flow=flow_id,
+            flows=flow_id,
             size=size,
             sort_order="desc",
             output_format="dataframe",
@@ -209,4 +209,4 @@ class TestEvaluationFunctions(TestBase):
         openml.config.server = self.production_server
         task_id = [6]
         size = 121
-        self._check_list_evaluation_setups(task=task_id, size=size)
+        self._check_list_evaluation_setups(tasks=task_id, size=size)
diff --git a/tests/test_evaluations/test_evaluations_example.py b/tests/test_evaluations/test_evaluations_example.py
index 61b6c359..5715b570 100644
--- a/tests/test_evaluations/test_evaluations_example.py
+++ b/tests/test_evaluations/test_evaluations_example.py
@@ -14,8 +14,8 @@ class TestEvaluationsExample(unittest.TestCase):
 
         df = openml.evaluations.list_evaluations_setups(
             "predictive_accuracy",
-            flow=[8353],
-            task=[6],
+            flows=[8353],
+            tasks=[6],
             output_format="dataframe",
             parameters_in_separate_columns=True,
         )  # Choose an SVM flow, for example 8353, and a task.