Repository: sktime/sktime

Issue:
[BUG] IndexError when selecting `return_X_y=False` and `return_type` is not referencing to a pandas dataframe.
**Describe the bug**
<!--
A clear and concise description of what the bug is.
-->
As per the discussion in #2697, it was [found](https://github.com/alan-turing-institute/sktime/issues/2697#issuecomment-1196290311) that `_load_provided_dataset()` must throw an error if `return_type` is not a reference to `pd.DataFrame` and `return_X_y=False`. This happens because of code appends the response variable into predictors under the column name 'class_val'.

**To Reproduce**
<!--
Add a Minimal, Complete, and Verifiable example (for more details, see e.g. https://stackoverflow.com/help/mcve

If the code is too long, feel free to put it in a public gist and link it in the issue: https://gist.github.com
-->

```python
import sktime
from sktime.datasets._data_io import _load_provided_dataset

def test_load_provided_dataset():
    name = "UnitTest"                                    # referencing to dataset available for testing in the directory 'datasets/data/'
    X, y = _load_provided_dataset(name, return_type='np2D', return_X_y=False)
    print(X.shape, y.shape)

if __name__ == "__main__":
    print("Staring program")
    test_load_provided_dataset()
    print("Done")
```

**Output of above code**
```
PS D:\Projects\sktime\sktime\sktime\datasets\tests> D:/Anaconda/envs/sktime-dev/python temp_test_data_io.py
Staring program
Traceback (most recent call last):
  File "temp_test_data_io.py", line 11, in <module>
    test_load_provided_dataset()
  File "temp_test_data_io.py", line 6, in test_load_provided_dataset
    X, y = _load_provided_dataset(name, return_type='np2D', return_X_y=False)
  File "d:\projects\sktime\sktime\sktime\datasets\_data_io.py", line 224, in _load_provided_dataset
    X["class_val"] = pd.Series(y)
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices
```

**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
This clearly is an issue because we are not checking the instance type of X, which could be `np.array` or `pd.DataFrame`.

**Solutions(?)**
* _Solution 1: Check instance and append in the appropriate axis_
  Consider the following cases 
  - Univariate, 2D, equal length data (`X=np.random.randn(20, 24), y=np.random.randn(20,)`) then `output.shape=(20,25)`
  - Univariate 3D, equal length data (`X=np.random.randn(20,1,24), y=np.random.randn(20,)`) then `output.shape=(20,1,25)`
  - Multivariate, equal length data (`X=np.random.randn(20,6,24), y=np.random.randn(20,)`) then `output.shape=(20,6,25)`; requires tiling labels which might not be desirable.
 
  I think it should also work for unequal length data but I need confirmation. The only drawback of this solution apart from the repetition of y in the multivariate case is that it might be less intuitive.

* _Solution 2: Force convert the data into pandas dataframe and give a warning about the conversion._

**Final Comments**
- There are more solutions to this but I think these two are the most straightforward. I also have the code ready for the first solution and will send a pull request if that is chosen, otherwise I'll take a day's time to send a pull request if we move forward with any other solution.
- I can also add unit tests regarding the same, I don't know if having tests for private methods are a thing or not, but I think it won't hurt having more tests.

**Versions**
<details>

<!--
Please run the following code snippet and paste the output here:
 
from sktime import show_versions; show_versions()
-->
```
D:\Anaconda\envs\sktime-dev\lib\site-packages\_distutils_hack\__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")

System:
    python: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 05:59:00) [MSC v.1929 64 bit (AMD64)]
executable: D:\Anaconda\envs\sktime-dev\python.exe
   machine: Windows-10-10.0.22000-SP0

Python dependencies:
          pip: 22.1.2
   setuptools: 63.1.0
      sklearn: 1.1.1
       sktime: 0.13.0
  statsmodels: 0.13.2
        numpy: 1.22.4
        scipy: 1.8.1
       pandas: 1.4.3
   matplotlib: 3.5.2
       joblib: 1.1.0
        numba: 0.55.2
     pmdarima: None
      tsfresh: None
```
</details>

<!-- Thanks for contributing! -->



Discussion:
Yes, that seems reasonable.
We need to stick to current docstrings to avoid breaking user code, and that means solution 1, because this is already described in the docstring of `load_UCR_UEA_dataset` as follows:

```
    y: (optional) numpy array shape (n,1). The class labels for each case in X.
        If return_X_y is False, y is appended to X.
```

I would suggest a slightly modified version of your solution 1:
* in `_load_provided_dataset`, call `load_from_tsfile` with the default `return_type="nested_univ"` first. This allows to delay any conversions to `_load_provided_dataset`
* in `_load_provided_dataset`, then deal with the `"nested_univ"` type only. At the end, before you return `X`, do the conversion using `convert`, just like currently in `load_from_tsfile`
* make sure the docstrings are up-to-date and in consistent format everywhere: in `_load_provided_dataset`, and also in `load_UCR_UEA_dataset` (which, after this operation, will be able to return more formats). You can copy variable descriptions from `load_from_tsfile` and `load_UCR_UEA_dataset`.

In addition, we should test the function, since apparently this is currently not tested.

That is, we should test `_load_provided_dataset` for one example, with:
* `return_X_y` being yes or no
* `return_type` being any of the strings in `MTYPE_LIST_PANEL`

(we can use `pytest.mark.parametrize` to do this easily, and add the tests in `datasets.tests`)

Resolving patch:
diff --git a/sktime/datasets/_data_io.py b/sktime/datasets/_data_io.py
--- a/sktime/datasets/_data_io.py
+++ b/sktime/datasets/_data_io.py
@@ -44,6 +44,17 @@
 MODULE = os.path.dirname(__file__)
 
 
+# Return appropriate return_type in case an alias was used
+def _alias_mtype_check(return_type):
+    if return_type is None:
+        return_type = "nested_univ"
+    if return_type in ["numpy2d", "numpy2D", "np2d", "np2D"]:
+        return_type = "numpyflat"
+    if return_type in ["numpy3d", "np3d", "np3D"]:
+        return_type = "numpy3D"
+    return return_type
+
+
 # time series classification data sets
 def _download_and_extract(url, extract_path=None):
     """
@@ -182,7 +193,7 @@ def _load_provided_dataset(
         name : string, file name
         split : string, default = None, or one of "TRAIN" or "TEST".
         return_X_y : default = True, if true, returns X and y separately.
-        return_type : default = None,
+        return_type : string or None, default = None.
         local_module: default = os.path.dirname(__file__),
         local_dirname: default = "data"
     """
@@ -192,36 +203,31 @@ def _load_provided_dataset(
     if split in ("TRAIN", "TEST"):
         fname = name + "_" + split + ".ts"
         abspath = os.path.join(local_module, local_dirname, name, fname)
-        X, y = load_from_tsfile(abspath, return_data_type=return_type)
+        X, y = load_from_tsfile(abspath, return_data_type="nested_univ")
     # if split is None, load both train and test set
     elif split is None:
         fname = name + "_TRAIN.ts"
         abspath = os.path.join(local_module, local_dirname, name, fname)
-        X_train, y_train = load_from_tsfile(abspath, return_data_type=return_type)
+        X_train, y_train = load_from_tsfile(abspath, return_data_type="nested_univ")
 
         fname = name + "_TEST.ts"
         abspath = os.path.join(local_module, local_dirname, name, fname)
-        X_test, y_test = load_from_tsfile(abspath, return_data_type=return_type)
+        X_test, y_test = load_from_tsfile(abspath, return_data_type="nested_univ")
 
-        if isinstance(X_train, np.ndarray):
-            X = np.concatenate([X_train, X_test])
-        elif isinstance(X_train, pd.DataFrame):
-            X = pd.concat([X_train, X_test])
-            X = X.reset_index(drop=True)
-        else:
-            raise IOError(
-                f"Invalid data structure type {type(X_train)} for loading "
-                f"classification problem "
-            )
+        X = pd.concat([X_train, X_test])
+        X = X.reset_index(drop=True)
         y = np.concatenate([y_train, y_test])
 
     else:
         raise ValueError("Invalid `split` value =", split)
-    # Return appropriately
+
+    return_type = _alias_mtype_check(return_type)
     if return_X_y:
+        X = convert(X, from_type="nested_univ", to_type=return_type)
         return X, y
     else:
         X["class_val"] = pd.Series(y)
+        X = convert(X, from_type="nested_univ", to_type=return_type)
         return X
 
 
@@ -324,7 +330,7 @@ def load_from_tsfile(
             "numpy3D"/"numpy3d"/"np3D": 3D np.ndarray (instance, variable, time index)
             "numpy2d"/"np2d"/"numpyflat": 2D np.ndarray (instance, time index)
             "pd-multiindex": pd.DataFrame with 2-level (instance, time) MultiIndex
-        Exception is raised if the data cannot be stored in therequested type.
+        Exception is raised if the data cannot be stored in the requested type.
 
     Returns
     -------
@@ -343,12 +349,7 @@ def load_from_tsfile(
     ValueError if return_data_type = numpy2d but the data are multivariate and/
     or unequal length series
     """
-    if return_data_type is None:
-        return_data_type = "nested_univ"
-    if return_data_type in ["numpy2d", "numpy2D", "np2d", "np2D"]:
-        return_data_type = "numpyflat"
-    if return_data_type in ["numpy3d", "np3d", "np3D"]:
-        return_data_type = "numpy3D"
+    return_data_type = _alias_mtype_check(return_data_type)
 
     if not isinstance(return_data_type, str):
         raise TypeError(
diff --git a/sktime/datasets/_single_problem_loaders.py b/sktime/datasets/_single_problem_loaders.py
--- a/sktime/datasets/_single_problem_loaders.py
+++ b/sktime/datasets/_single_problem_loaders.py
@@ -53,7 +53,9 @@
 MODULE = os.path.dirname(__file__)
 
 
-def load_UCR_UEA_dataset(name, split=None, return_X_y=True, extract_path=None):
+def load_UCR_UEA_dataset(
+    name, split=None, return_X_y=True, return_type=None, extract_path=None
+):
     """Load dataset from UCR UEA time series archive.
 
     Downloads and extracts dataset if not already downloaded. Data is assumed to be
@@ -76,6 +78,18 @@ def load_UCR_UEA_dataset(name, split=None, return_X_y=True, extract_path=None):
         format <name>_TRAIN.ts or <name>_TEST.ts.
     return_X_y : bool, optional (default=False)
         it returns two objects, if False, it appends the class labels to the dataframe.
+    return_type: None or str, optional (default=None)
+        Memory data format specification to return X in, if None then return the default
+        "nested_univ" type.
+        str can be any other supported Panel mtype,
+            for list of mtypes, see datatypes.SCITYPE_REGISTER
+            for specifications, see examples/AA_datatypes_and_datasets.ipynb
+        commonly used specifications:
+            "nested_univ: nested pd.DataFrame, pd.Series in cells
+            "numpy3D"/"numpy3d"/"np3D": 3D np.ndarray (instance, variable, time index)
+            "numpy2d"/"np2d"/"numpyflat": 2D np.ndarray (instance, time index)
+            "pd-multiindex": pd.DataFrame with 2-level (instance, time) MultiIndex
+        Exception is raised if the data cannot be stored in the requested type.
     extract_path : str, optional (default=None)
         the path to look for the data. If no path is provided, the function
         looks in `sktime/datasets/data/`.
@@ -96,7 +110,7 @@ def load_UCR_UEA_dataset(name, split=None, return_X_y=True, extract_path=None):
     >>> from sktime.datasets import load_UCR_UEA_dataset
     >>> X, y = load_UCR_UEA_dataset(name="ArrowHead")
     """
-    return _load_dataset(name, split, return_X_y, extract_path)
+    return _load_dataset(name, split, return_X_y, return_type, extract_path)
 
 
 def load_plaid(split=None, return_X_y=True):


Test patch:
diff --git a/sktime/datasets/tests/test_data_io.py b/sktime/datasets/tests/test_data_io.py
--- a/sktime/datasets/tests/test_data_io.py
+++ b/sktime/datasets/tests/test_data_io.py
@@ -1,12 +1,7 @@
 # -*- coding: utf-8 -*-
 """Test functions for data input and output."""
 
-__author__ = [
-    "SebasKoel",
-    "Emiliathewolf",
-    "TonyBagnall",
-    "jasonlines",
-]
+__author__ = ["SebasKoel", "Emiliathewolf", "TonyBagnall", "jasonlines", "achieveordie"]
 
 __all__ = []
 
@@ -30,8 +25,33 @@
     load_uschange,
     write_dataframe_to_tsfile,
 )
-from sktime.datasets._data_io import MODULE, _convert_tsf_to_hierarchical
-from sktime.datatypes import check_is_mtype
+from sktime.datasets._data_io import (
+    MODULE,
+    _convert_tsf_to_hierarchical,
+    _load_provided_dataset,
+)
+from sktime.datatypes import MTYPE_LIST_PANEL, check_is_mtype
+
+# Disabling test for these mtypes since they don't support certain functionality yet
+_TO_DISABLE = ["pd-long", "pd-wide", "numpyflat"]
+
+
+@pytest.mark.parametrize("return_X_y", [True, False])
+@pytest.mark.parametrize(
+    "return_type", [mtype for mtype in MTYPE_LIST_PANEL if mtype not in _TO_DISABLE]
+)
+def test_load_provided_dataset(return_X_y, return_type):
+    """Test function to check for proper loading.
+
+    Check this via permutating between all possibilities of return_X_y and return_type.
+    """
+    if return_X_y:
+        X, y = _load_provided_dataset("UnitTest", "TRAIN", return_X_y, return_type)
+    else:
+        X = _load_provided_dataset("UnitTest", "TRAIN", return_X_y, return_type)
+
+    # Check whether object is same mtype or not, via bool
+    assert check_is_mtype(X, return_type)
 
 
 def test_load_from_tsfile():