Repository: sktime/sktime
Pull request: [BUG] writing datasets to file in ts format
#### Reference Issues/PRs
<!--
Example: Fixes #1234. Fixes #302 

Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
fixes #3499

@achieveordie could you take a look please?

#### What does this implement/fix? Explain your changes.

this PR fixes a problem with writing data to .ts format and introduces a new method to make writing easier and more intuitive.
data_io contains two functions for saving ts files: write_dataframe_to_tsfile and write_ndarray_to_tsfile. write_dataframe_to_tsfile converts to ndarray and calls write_ndarray_to_tsfile. This does not work if the data frame contains unequal length series.  

In this PR write_dataframe_to_tsfile now gets the data metadata with check_is_scitype, then if equal length, converts to numpy, if not, iterates over the dataframe and writes in the format.

I have also implemented (and tested) a write_panel_to_tsfile. This gets data characteristics, if equal length converts to numpy then calls write_ndarray_to_tsfile, otherwise calls  write_dataframe_to_tsfile. The reason for this new function is there is no need for user to differentiate between dataframe and ndarray, and the default arguments to write_dataframe_to_tsfile and write_ndarray_to_tsfile are confusing (e.g. equal_length=False and  series_length=-1). There is no need to pass all these arguments in. 

There is some redundancy in write_panel_to_tsfile and write_dataframe_to_tsfile necessary to satisfy the deprecation policy: both functions call check_is_scitype and try to convert. I think write_dataframe_to_tsfile and write_ndarray_to_tsfile should be deprecated to private and the checks/conversion in write_dataframe_to_tsfile removed (so it simply writes from dataframe). I have not done this deprecation, but I can do so though.


#### Does your contribution introduce a new dependency? If yes, which one?
No
#### What should a reviewer concentrate their feedback on?
1. The testing writes and deletes files. Not sure if this is the preferred way of doing this
2. Should we deprecate write_ndarray_to_tsfile and write_dataframe_to_tsfile to being private? I think we should


#### PR checklist
<!--
Please go through the checklist below. Please feel free to remove points if they are not applicable.
-->

##### For all contributions
- [x] I've added unit tests and made sure they pass locally.
- [x] The PR title starts with either [ENH], [MNT], [DOC], or [BUG] indicating whether the PR topic is related to enhancement, maintenance, documentation, or bug.



Problem statement:
[BUG]’write_dataframe_to_tsfile‘ doesn't work for length-unequal series
**Describe the bug**
I'm trying to make my own .ts format dataset using the function write_dataframe_to_tsfile.
There is no clear statement about whether length-unequal time series is  supported in the definition of function 'write_dataframe_to_tsfile'. 
There is a parameter named 'equal_lenth', which indicates whether each series is of equal length. And this parameter is 'False' in default. So I think it means this fuction support length-uneuqal series. 
 However, if I provide **a time series with 2 variates, 250 cases in different length**,  this function will returns '**ValueError: all input arrays must have the same shape'**.If I keep those cases in same length, it works. 
I think it maybe a bug? 


**To Reproduce**
I provide the following code to reproduce the bug.
PS: one of length-equal variable df1 has been commented out.

```python
import pandas as pd
from sktime.datasets import write_dataframe_to_tsfile
from sktime.datatypes._panel._convert import from_multi_index_to_nested

##df1=pd.DataFrame([[1,1,1,1],[1,2,2,2],[2,3,3,3],[2,4,4,4],[2,5,5,5],[3,6,6,6]],columns=['a','b','c','d'])
df1=pd.DataFrame ([[1,1,1,1],[1,2,2,2],[2,3,3,3],[2,4,4,4],[3,5,5,5],[3,6,6,6]],columns=['a','b','c','d'])

df1.set_index(keys=['a','b'],inplace=True)
print(df1)

Y_nested = from_multi_index_to_nested(df1)
##print(Y_nested.head())
write_dataframe_to_tsfile(data=Y_nested,equal_length= False,fold='_TRAIN',path='./path')
```

**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->

**Additional context**
Providing length-unequal time series is important, because even the same motion may cost time more or less. 
Besides, there are some length-unequal time series archieve in UCR archieve, such like Basic_Motions so  i think sktime is supposed to provide a way for user to create a length-unequal time series in .ts format. 

**Versions**
sktime 0.13.2

<!--
Please run the following code snippet and paste the output here:
 
from sktime import show_versions; show_versions()
-->

</details>

<!-- Thanks for contributing! -->

Hints from issue discussion:
About the func write_dataframe_to_tsfile, I linked this issue to #438 #1278  
hi, thanks for that, I'll look into it
@TonyBagnall, kindly ask @achieveordie to review if/once ready, as he has been reworking the ts load/save functionality recently and documenting the ts file format spec that so far no one has properly written up.

See https://github.com/sktime/sktime/pull/3380
In fact, I've generated .ts format data correctly with length-equal data. And since sktime do not support generate  .ts format length-unequal time series for now, I'd like to give a temporary way to realize it. This way uses TimeseriesResampler module from tslearn to convert time series to equal length. Although this way may cause slower computing speed and complicate transfer between DataFrame and Ndarray format, it works well in my computer. I'm not a expert in coding but I'd like to share it if it does help anyone.  I'll submit it later.

Commit message: Merge branch 'main' into write_to_ts

# Conflicts:
#	sktime/_contrib/datasets/__init__.py

Patch to review:
diff --git a/sktime/datasets/__init__.py b/sktime/datasets/__init__.py
index af7b8a7584d..0554006bbfa 100644
--- a/sktime/datasets/__init__.py
+++ b/sktime/datasets/__init__.py
@@ -29,14 +29,15 @@
     "load_from_tsfile_to_dataframe",
     "load_from_ucr_tsv_to_dataframe",
     "make_multi_index_dataframe",
-    "write_dataframe_to_tsfile",
-    "write_ndarray_to_tsfile",
-    "write_results_to_uea_format",
-    "write_tabular_transformation_to_arff",
     "load_tsf_to_dataframe",
     "load_unit_test_tsf",
     "load_solar",
     "load_covid_3month",
+    "write_panel_to_tsfile",
+    "write_dataframe_to_tsfile",
+    "write_ndarray_to_tsfile",
+    "write_results_to_uea_format",
+    "write_tabular_transformation_to_arff",
 ]
 
 from sktime.datasets._data_io import (
@@ -50,6 +51,7 @@
     make_multi_index_dataframe,
     write_dataframe_to_tsfile,
     write_ndarray_to_tsfile,
+    write_panel_to_tsfile,
     write_results_to_uea_format,
     write_tabular_transformation_to_arff,
 )
diff --git a/sktime/datasets/_data_io.py b/sktime/datasets/_data_io.py
index 8887e2f68e0..065de682844 100644
--- a/sktime/datasets/_data_io.py
+++ b/sktime/datasets/_data_io.py
@@ -36,7 +36,13 @@
 import numpy as np
 import pandas as pd
 
-from sktime.datatypes import MTYPE_LIST_HIERARCHICAL, MTYPE_LIST_PANEL, convert
+from sktime.datatypes import (
+    MTYPE_LIST_HIERARCHICAL,
+    MTYPE_LIST_PANEL,
+    check_is_scitype,
+    convert,
+    convert_to,
+)
 from sktime.datatypes._panel._convert import _make_column_names, from_long_to_nested
 from sktime.transformations.base import BaseTransformer
 from sktime.utils.validation.panel import check_X, check_X_y
@@ -179,7 +185,8 @@ def _load_dataset(name, split, return_X_y, return_type=None, extract_path=None):
     if not os.path.exists(os.path.join(local_module, local_dirname)):
         os.makedirs(os.path.join(local_module, local_dirname))
     if name not in _list_available_datasets(extract_path):
-        local_dirname = "local_data"
+        if extract_path is None:
+            local_dirname = "local_data"
         if not os.path.exists(os.path.join(local_module, local_dirname)):
             os.makedirs(os.path.join(local_module, local_dirname))
         if name not in _list_available_datasets(
@@ -187,7 +194,7 @@ def _load_dataset(name, split, return_X_y, return_type=None, extract_path=None):
         ):
             # Dataset is not already present in the datasets directory provided.
             # If it is not there, download and install it.
-            url = "http://timeseriesclassification.com/Downloads/%s.zip" % name
+            url = "https://timeseriesclassification.com/Downloads/%s.zip" % name
             # This also tests the validitiy of the URL, can't rely on the html
             # status code as it always returns 200
             try:
@@ -197,10 +204,9 @@ def _load_dataset(name, split, return_X_y, return_type=None, extract_path=None):
                 )
             except zipfile.BadZipFile as e:
                 raise ValueError(
-                    "Invalid dataset name. ",
-                    extract_path,
-                    "Please make sure the dataset "
-                    + "is available on http://timeseriesclassification.com/.",
+                    f"Invalid dataset name ={name} is not available on extract path ="
+                    f"{extract_path}. Nor is it available on "
+                    f"https://timeseriesclassification.com/.",
                 ) from e
 
     return _load_provided_dataset(
@@ -1493,6 +1499,46 @@ def write_tabular_transformation_to_arff(
     file.close()
 
 
+def _write_header(
+    path,
+    problem_name,
+    univariate,
+    equal_length,
+    series_length,
+    class_label,
+    fold,
+    comment,
+):
+    # create path if not exist
+    dirt = f"{str(path)}/{str(problem_name)}/"
+    try:
+        os.makedirs(dirt)
+    except os.error:
+        pass  # raises os.error if path already exists
+    # create ts file in the path
+    file = open(f"{dirt}{str(problem_name)}{fold}.ts", "w")
+    # write comment if any as a block at start of file
+    if comment is not None:
+        file.write("\n# ".join(textwrap.wrap("# " + comment)))
+        file.write("\n")
+
+    """ Writes the header info for a ts file"""
+    file.write(f"@problemName {problem_name}\n")
+    file.write("@timestamps false\n")
+    file.write(f"@univariate {str(univariate).lower()}\n")
+    file.write(f"@equalLength {str(equal_length).lower()}\n")
+    if series_length > 0 and equal_length:
+        file.write(f"@seriesLength {series_length}\n")
+    # write class label line
+    if class_label is not None:
+        space_separated_class_label = " ".join(str(label) for label in class_label)
+        file.write(f"@classLabel true {space_separated_class_label}\n")
+    else:
+        file.write("@classLabel false\n")
+    file.write("@data\n")
+    return file
+
+
 def write_dataframe_to_tsfile(
     data,
     path,
@@ -1550,24 +1596,59 @@ def write_dataframe_to_tsfile(
     """
     # ensure data provided is a dataframe
     if not isinstance(data, pd.DataFrame):
-        raise ValueError("Data provided must be a DataFrame")
-    if class_value_list is not None:
-        data, class_value_list = check_X_y(data, class_value_list, coerce_to_numpy=True)
-    else:
-        data = check_X(data, coerce_to_numpy=True)
-    # ensure data provided is a dataframe
-    write_ndarray_to_tsfile(
-        data,
-        path,
-        problem_name=problem_name,
-        class_label=class_label,
-        class_value_list=class_value_list,
-        equal_length=equal_length,
-        series_length=series_length,
-        missing_values=missing_values,
-        comment=comment,
-        fold=fold,
+        raise ValueError(f"Data provided must be a DataFrame, passed a {type(data)}")
+    data_valid, _, metadata = check_is_scitype(
+        data, scitype="Panel", return_metadata=True
     )
+    if not data_valid:
+        raise ValueError("DataFrame provided is not a valid type")
+    metadata
+    if equal_length != metadata["is_equal_length"]:
+        raise ValueError(
+            f"Argument passed for equal length = {equal_length} is not "
+            f"true for the data passed"
+        )
+    if equal_length:
+        # Convert to [cases][dimensions][length] numpy.
+        data = convert_to(
+            data,
+            to_type="numpy3D",
+            as_scitype="Panel",
+            store_behaviour="freeze",
+        )
+        write_ndarray_to_tsfile(
+            data,
+            path,
+            problem_name=problem_name,
+            class_label=class_label,
+            class_value_list=class_value_list,
+            equal_length=equal_length,
+            series_length=data.shape[2],
+            missing_values=missing_values,
+            comment=comment,
+            fold=fold,
+        )
+    else:  # Write by iterating over dataframe
+        if class_value_list is not None and class_label is None:
+            class_label = np.unique(class_value_list)
+        file = _write_header(
+            path,
+            problem_name,
+            metadata["is_univariate"],
+            metadata["is_equal_length"],
+            series_length,
+            class_label,
+            fold,
+            comment,
+        )
+        n_cases, n_dimensions = data.shape
+        for i in range(0, n_cases):
+            for j in range(0, n_dimensions):
+                series = data.iloc[i, j]
+                for k in range(0, series.size - 1):
+                    file.write(f"{series[k]},")
+                file.write(f"{series[series.size-1]}:")
+            file.write(f"{class_value_list[i]}\n")
 
 
 def write_ndarray_to_tsfile(
@@ -1644,34 +1725,18 @@ def write_ndarray_to_tsfile(
         )
     if fold is None:
         fold = ""
-    # create path if not exist
-    dirt = f"{str(path)}/{str(problem_name)}/"
-    try:
-        os.makedirs(dirt)
-    except os.error:
-        pass  # raises os.error if path already exists
-    # create ts file in the path
-    file = open(f"{dirt}{str(problem_name)}{fold}.ts", "w")
-    # write comment if any as a block at start of file
-    if comment is not None:
-        file.write("\n# ".join(textwrap.wrap("# " + comment)))
-        file.write("\n")
-    # begin writing header information
-    file.write(f"@problemName {problem_name}\n")
-    file.write("@timestamps false\n")
-    file.write(f"@univariate {str(univariate).lower()}\n")
-    file.write(f"@equalLength {str(equal_length).lower()}\n")
-    if series_length > 0 and equal_length:
-        file.write(f"@seriesLength {series_length}\n")
-    # write class label line
-    if class_label is not None:
-        space_separated_class_label = " ".join(str(label) for label in class_label)
-        file.write(f"@classlabel true {space_separated_class_label}\n")
-    else:
-        file.write("@classlabel false\n")
+    file = _write_header(
+        path,
+        problem_name,
+        univariate,
+        equal_length,
+        series_length,
+        class_label,
+        fold,
+        comment,
+    )
     # begin writing the core data for each case
     # which are the series and the class value list if there is any
-    file.write("@data\n")
     for case, value in itertools.zip_longest(data, class_value_list):
         for dimension in case:
             # turn series into comma-separated row
@@ -1979,3 +2044,40 @@ def _convert_tsf_to_hierarchical(
     df = df.astype({value_column_name: "float"}, errors="ignore")
 
     return df
+
+
+def write_panel_to_tsfile(
+    data, path, target=None, problem_name="sample_data", header=None
+):
+    data_valid, _, data_metadata = check_is_scitype(
+        data, scitype="Panel", return_metadata=True
+    )
+    if not data_valid:
+        raise TypeError(" Wrong input data type ", type(data))
+    if data_metadata["is_equal_length"]:
+        # check class labels
+        data = convert_to(
+            data,
+            to_type="numpy3D",
+            as_scitype="Panel",
+            store_behaviour="freeze",
+        )
+        series_length = data.shape[2]
+        write_ndarray_to_tsfile(
+            data,
+            path,
+            problem_name=problem_name,
+            class_value_list=target,
+            equal_length=True,
+            series_length=series_length,
+            comment=header,
+        )
+    else:
+        write_dataframe_to_tsfile(
+            data,
+            path,
+            problem_name=problem_name,
+            class_value_list=target,
+            equal_length=False,
+            comment=header,
+        )
diff --git a/sktime/datasets/_single_problem_loaders.py b/sktime/datasets/_single_problem_loaders.py
index 8d3299867fe..eaba0e29203 100644
--- a/sktime/datasets/_single_problem_loaders.py
+++ b/sktime/datasets/_single_problem_loaders.py
@@ -94,7 +94,8 @@ def load_UCR_UEA_dataset(
         Exception is raised if the data cannot be stored in the requested type.
     extract_path : str, optional (default=None)
         the path to look for the data. If no path is provided, the function
-        looks in `sktime/datasets/data/`.
+        looks in `sktime/datasets/data/`. If a path is given, it can be absolute,
+        e.g. C:/Temp or relative, e.g. Temp or ./Temp.
 
     Returns
     -------
diff --git a/sktime/datasets/tests/test_data_io.py b/sktime/datasets/tests/test_data_io.py
index 7ee20b8a8ea..7e80001d556 100644
--- a/sktime/datasets/tests/test_data_io.py
+++ b/sktime/datasets/tests/test_data_io.py
@@ -6,6 +6,7 @@
 __all__ = []
 
 import os
+import shutil
 import tempfile
 
 import numpy as np
@@ -25,6 +26,7 @@
     load_UCR_UEA_dataset,
     load_uschange,
     write_dataframe_to_tsfile,
+    write_panel_to_tsfile,
 )
 from sktime.datasets._data_io import (
     MODULE,
@@ -38,6 +40,45 @@
 _TO_DISABLE = ["pd-long", "pd-wide", "numpyflat"]
 
 
+@pytest.mark.parametrize("dataset_name", ["UnitTest", "BasicMotions"])
+@pytest.mark.parametrize("return_type", ["nested_univ", "numpy3d"])
+def test_write_panel_to_tsfile_equal_length(dataset_name, return_type):
+    """Test function to write a dataset.
+
+    Loads equal and unequal length problems into both data frames and numpy arrays,
+    writes locally, reloads, then compares all class labels. It then delete the files.
+    """
+    X, y = _load_provided_dataset(dataset_name, split="TRAIN", return_type=return_type)
+    write_panel_to_tsfile(data=X, path="./Temp", target=y, problem_name=dataset_name)
+    load_path = f"./Temp/{dataset_name}/{dataset_name}.ts"
+    newX, newy = load_from_tsfile(
+        full_file_path_and_name=load_path, return_data_type=return_type
+    )
+    assert np.array_equal(y, newy)
+    shutil.rmtree("./Temp")
+
+
+@pytest.mark.parametrize("dataset_name", ["PLAID", "JapaneseVowels"])
+def test_write_panel_to_tsfile_unequal_length(dataset_name):
+    """Test function to write a dataset.
+
+    Loads equal and unequal length problems into both data frames and numpy arrays,
+    writes locally, reloads, then compares all class labels. It then delete the files.
+    """
+    X, y = _load_provided_dataset(
+        dataset_name, split="TRAIN", return_type="nested_univ"
+    )
+    write_panel_to_tsfile(
+        data=X, path=f"./Temp{dataset_name}/", target=y, problem_name=dataset_name
+    )
+    load_path = f"./Temp{dataset_name}/{dataset_name}/{dataset_name}.ts"
+    newX, newy = load_from_tsfile(
+        full_file_path_and_name=load_path, return_data_type="nested_univ"
+    )
+    assert np.array_equal(y, newy)
+    shutil.rmtree(f"./Temp{dataset_name}")
+
+
 @pytest.mark.parametrize("return_X_y", [True, False])
 @pytest.mark.parametrize(
     "return_type", [mtype for mtype in MTYPE_LIST_PANEL if mtype not in _TO_DISABLE]
@@ -1123,6 +1164,7 @@ def test_write_dataframe_to_ts_success(tmp_path, dataset):
         problem_name=dataset,
         class_label=np.unique(test_y),
         class_value_list=test_y,
+        equal_length=True,
         comment="""
           The data was derived from twelve monthly electrical power demand
           time series from Italy and first used in the paper "Intelligent


Code review comments:
diff_hunk: @@ -1550,24 +1596,59 @@ def write_dataframe_to_tsfile(
     """
     # ensure data provided is a dataframe
     if not isinstance(data, pd.DataFrame):
-        raise ValueError("Data provided must be a DataFrame")
-    if class_value_list is not None:
-        data, class_value_list = check_X_y(data, class_value_list, coerce_to_numpy=True)
-    else:
-        data = check_X(data, coerce_to_numpy=True)
-    # ensure data provided is a dataframe
-    write_ndarray_to_tsfile(
-        data,
-        path,
-        problem_name=problem_name,
-        class_label=class_label,
-        class_value_list=class_value_list,
-        equal_length=equal_length,
-        series_length=series_length,
-        missing_values=missing_values,
-        comment=comment,
-        fold=fold,
+        raise ValueError(f"Data provided must be a DataFrame, passed a {type(data)}")
+    data_valid, _, metadata = check_is_scitype(
+        data, scitype="Panel", return_metadata=True
     )
+    if not data_valid:
+        raise ValueError("DataFrame provided is not a valid type")
+    metadata
path: sktime/datasets/_data_io.py
text: @user1:
What does this line do? Probably here by mistake? I'm not sure

@author:
oops! Good spot