Repository: microsoft/Qcodes

Issue:
Dataset with nested metadata throws TypeError when exporting to NetCDF file
### Steps to reproduce
1. Prepare a Dataset
2. Add metadata to the dataset where the value is a `dict`
3. Export the dataset to netCDF

### Expected behaviour
The data is exported to netCDF where values that are `dict`s are either skipped or converted to `str`

### Actual behaviour
```python
TypeError                                 Traceback (most recent call last)
<ipython-input-155-35b4f9ff4165> in <module>
----> 1 ds.export(export_type="netcdf", path=output_dir, prefix=file_prefix)

~/repos/Qcodes/qcodes/dataset/data_set.py in export(self, export_type, path, prefix)
   1711             )
   1712 
-> 1713         self._export_path = self._export_data(
   1714             export_type=export_type,
   1715             path=path,

~/repos/Qcodes/qcodes/dataset/data_set.py in _export_data(self, export_type, path, prefix)
   1675             file_name = self._export_file_name(
   1676                 prefix=prefix, export_type=DataExportType.NETCDF)
-> 1677             return self._export_as_netcdf(path=path, file_name=file_name)
   1678 
   1679         elif DataExportType.CSV == export_type:

~/repos/Qcodes/qcodes/dataset/data_set.py in _export_as_netcdf(self, path, file_name)
   1642         file_path = os.path.join(path, file_name)
   1643         xarr_dataset = self.to_xarray_dataset()
-> 1644         xarr_dataset.to_netcdf(path=file_path)
   1645         return path
   1646 

~/anaconda3/envs/bringup/lib/python3.9/site-packages/xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf)
   1687         from ..backends.api import to_netcdf
   1688 
-> 1689         return to_netcdf(
   1690             self,
   1691             path,

~/anaconda3/envs/bringup/lib/python3.9/site-packages/xarray/backends/api.py in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf)
   1057     # validate Dataset keys, DataArray names, and attr keys/values
   1058     _validate_dataset_names(dataset)
-> 1059     _validate_attrs(dataset)
   1060 
   1061     try:

~/anaconda3/envs/bringup/lib/python3.9/site-packages/xarray/backends/api.py in _validate_attrs(dataset)
    229     # Check attrs on the dataset itself
    230     for k, v in dataset.attrs.items():
--> 231         check_attr(k, v)
    232 
    233     # Check attrs on each variable within the dataset

~/anaconda3/envs/bringup/lib/python3.9/site-packages/xarray/backends/api.py in check_attr(name, value)
    220 
    221         if not isinstance(value, (str, Number, np.ndarray, np.number, list, tuple)):
--> 222             raise TypeError(
    223                 f"Invalid value for attr {name!r}: {value!r} must be a number, "
    224                 "a string, an ndarray or a list/tuple of "

TypeError: Invalid value for attr 'extra_metadata': {
```

### System
**operating system**
Ubuntu 20.04.1 LTS on WSL2
**qcodes branch**
`master`

Tagging @jenshnielsen. What would you recommend as a best solution here - drop the nested data or convert it to `str`? We currently convert the snapshot to `str` as well so to be consistent this may be the best way to go.


Discussion:
@guenp Yes similar to the snapshot I think we should serialize the extra metadata to json. netcdf4 is very limited in the types allowed in the attributes dict
@jenshnielsen That would be great, thanks! I've implemented a quick-and-dirty fix for this such that I was able to upload my data but a more sustainable solution would be great :) The main request I have is that we add some functionality to load the data from netCDF file as well and deserialize the data back to a nested dictionary.

Another issue I ran into is that the serialized nested dicts are very verbose and take up multiple pages when using the __repr__ functionality in a jupyter notebook. I posted a question on this in the xarray gitter: https://gitter.im/pydata/xarray
Hi @guenp looking more closely at this I am a bit confused as to how you produced this dataset.
If I do the following 

```
dataset1D.add_metadata("sometag", {1:1})
```
on some dataset I will get an unsupported type error from sqlite

```
2021-04-15 15:30:23,942 ¦ qcodes.dataset.sqlite.connection ¦ ERROR ¦ connection ¦ atomic ¦ 106 ¦ Rolling back due to unhandled exception
Traceback (most recent call last):
  File "c:\users\jens-work\source\repos\qcodes\qcodes\dataset\sqlite\connection.py", line 103, in atomic
    yield conn
  File "c:\users\jens-work\source\repos\qcodes\qcodes\dataset\sqlite\connection.py", line 159, in atomic_transaction
    c = transaction(atomic_conn, sql, *args)
  File "c:\users\jens-work\source\repos\qcodes\qcodes\dataset\sqlite\connection.py", line 134, in transaction
    c.execute(sql, args)
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.
2021-04-15 15:30:23,951 ¦ qcodes.dataset.sqlite.connection ¦ ERROR ¦ connection ¦ atomic ¦ 106 ¦ Rolling back due to unhandled exception
Traceback (most recent call last):
  File "c:\users\jens-work\source\repos\qcodes\qcodes\dataset\sqlite\connection.py", line 103, in atomic
    yield conn
  File "c:\users\jens-work\source\repos\qcodes\qcodes\dataset\sqlite\connection.py", line 159, in atomic_transaction
    c = transaction(atomic_conn, sql, *args)
  File "c:\users\jens-work\source\repos\qcodes\qcodes\dataset\sqlite\connection.py", line 134, in transaction
    c.execute(sql, args)
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.
```

So I would not expect it to be possible to produce a dataset with nested metadata. I guess I must be overlooking something?
@jenshnielsen Ah, I think you're right. This dataset was produced with Nanotune using `json.dumps`: , see [nanotune/tests/data_savers.py#L33](https://github.com/microsoft/nanotune/blob/3d25fa29330fea9a0a9ee09ce9eada94178517af/nanotune/tests/data_savers.py#L33).

So the problem is that when qcodes exports the dataset to xarray, the whole `DataSet._metadata` dict is nested under the `"extra_metadata"` key in `.attrs` rather than storing them by key/value pair.

Here's a minimal example to reproduce:
```python
from qcodes import Measurement
import json
import numpy as np

meas = Measurement()
meas.register_custom_parameter(
    "voltage", paramtype="numeric", label="voltage x", unit="V"
)
meas.register_custom_parameter(
    "current",
    paramtype="numeric",
    label="dc current",
    unit="A",
    setpoints=("voltage",),
)

voltage, current = np.random.rand(), np.random.rand()
with meas.run() as datasaver:
    datasaver.add_result(("voltage", voltage), ("current", current))
    datasaver.dataset.add_metadata("my_param1", 42)
    datasaver.dataset.add_metadata("my_param2", "other")

    nt_metadata = {
        "foo": {
            "bar": {
                "baz": "test"
            },
            "spam": [1, 2, 3],
        }
    }
    datasaver.dataset.add_metadata("foo_metadata", json.dumps(nt_metadata))

datasaver.dataset.export(export_type="netcdf", path=".", prefix="export_test_")
```

this raises the above error:
```
TypeError: Invalid value for attr 'extra_metadata': {'my_param1': 42, 'my_param2': 'other', 'foo_metadata': '{"foo": {"bar": {"baz": "test"}, "spam": [1, 2, 3]}}'} must be a number, a string, an ndarray or a list/tuple of numbers/strings for serialization to netCDF files
```
@guenp Thanks, that we can write a test for and fix :) 

Resolving patch:
diff --git a/qcodes/dataset/exporters/export_to_xarray.py b/qcodes/dataset/exporters/export_to_xarray.py
--- a/qcodes/dataset/exporters/export_to_xarray.py
+++ b/qcodes/dataset/exporters/export_to_xarray.py
@@ -73,11 +73,9 @@ def _add_metadata_to_xarray(
     if dataset.completed_timestamp_raw is not None:
         xrdataset.attrs[
             "completed_timestamp_raw"] = dataset.completed_timestamp_raw
-    if len(dataset._metadata) > 0:
-        xrdataset.attrs['extra_metadata'] = {}
-
-        for metadata_tag, metadata in dataset._metadata.items():
-            xrdataset.attrs['extra_metadata'][metadata_tag] = metadata
+    if len(dataset.metadata) > 0:
+        for metadata_tag, metadata in dataset.metadata.items():
+            xrdataset.attrs[metadata_tag] = metadata
 
 
 def load_to_xarray_dataset(dataset: DataSet, data: ParameterData) -> xr.Dataset:


Test patch:
diff --git a/qcodes/tests/dataset/test_dataset_export.py b/qcodes/tests/dataset/test_dataset_export.py
--- a/qcodes/tests/dataset/test_dataset_export.py
+++ b/qcodes/tests/dataset/test_dataset_export.py
@@ -4,6 +4,7 @@
 import pytest
 import xarray as xr
 
+import qcodes
 from qcodes import new_data_set
 from qcodes.dataset.descriptions.dependencies import InterDependencies_
 from qcodes.dataset.descriptions.param_spec import ParamSpecBase
@@ -230,6 +231,34 @@ def test_export_to_xarray_ds_dict_extra_metadata(mock_dataset):
         _assert_xarray_metadata_is_as_expected(datarray, mock_dataset)
 
 
+def test_export_to_xarray_extra_metadate_can_be_stored(mock_dataset, tmp_path):
+
+    nt_metadata = {
+        "foo": {
+            "bar": {
+                "baz": "test"
+            },
+            "spam": [1, 2, 3],
+        }
+    }
+    mock_dataset.add_metadata("foo_metadata", json.dumps(nt_metadata))
+    mock_dataset.export(export_type="netcdf", path=str(tmp_path))
+    data_as_xarray = mock_dataset.to_xarray_dataset()
+
+    loaded_data = xr.load_dataset(
+        tmp_path/f"{qcodes.config.dataset.export_prefix}{mock_dataset.run_id}.nc"
+    )
+
+    # check that the metadata in the qcodes dataset is roundtripped to the loaded
+    # dataset
+    for key in mock_dataset.metadata.keys():
+        assert mock_dataset.metadata[key] == loaded_data.attrs[key]
+    # check that the added metadata roundtrip correctly
+    assert loaded_data.attrs["foo_metadata"] == json.dumps(nt_metadata)
+    # check that all attrs roundtrip correctly within the xarray ds
+    assert loaded_data.attrs == data_as_xarray.attrs
+
+
 def _assert_xarray_metadata_is_as_expected(xarray_ds, qc_dataset):
 
     assert xarray_ds.ds_name == qc_dataset.name