Repository: sunpy/sunpy

Issue:
PSP/ISOIS CDF data via CDAWeb: Download works, but to_dataframe() fails
### Describe the bug

I try to obtain Parker Solar Probe (PSP) in-situ particle data of the ISOIS instrument from CDAWeb via Sunpy's new CDAWeb client (#5558). The actual data files are CDF's, so I guess the new CDF option (#5435) is used (both by @dstansby).

The download and generation of TimeSeries object is working (although the latter causes a lot of `PerformanceWarning: DataFrame is highly fragmented`, see code example output below), but when I try to actually obtain the data with `.to_dataframe()`, `.data`, or `.to_table()`, I get either a `ValueError: Shape of passed values is (x, x, x), indices imply (y, y)` or a `ValueError: Length of values (x) does not match length of index (y)`, respectively.

### To Reproduce

``` python
>>> from sunpy.net import Fido
>>> from sunpy.net import attrs as a
>>> from sunpy.timeseries import TimeSeries

>>> # defining function to quickly try different datasets:
>>> def load_psp(dataset_name):
>>>   dataset = a.cdaweb.Dataset(dataset_name)
>>>   trange = a.Time('2021/07/01', '2021/07/02')
>>>   result = Fido.search(trange, dataset)
>>>   downloaded_files = Fido.fetch(result)
>>>   data = TimeSeries(downloaded_files, concatenate=True)
>>>   df = data.to_dataframe()
>>>   return df, data

  
>>> df, data = load_psp('PSP_ISOIS-EPIHI_L2-HET-RATES3600')
WARNING: SunpyUserWarning: astropy did not recognize units of "counts". Assigning dimensionless units. If you think this unit should not be dimensionless, please raise an issue at https://github.com/sunpy/sunpy/issues [sunpy.io.cdf]
/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/sunpy/io/cdf.py:94: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  df[var_key] = data
  
>>> print(df)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/frame.py", line 1012, in __repr__
    return self.to_string(**repr_params)
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/frame.py", line 1175, in to_string
    formatter = fmt.DataFrameFormatter(
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/io/formats/format.py", line 604, in __init__
    self.truncate()
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/io/formats/format.py", line 790, in truncate
    self._truncate_horizontally()
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/io/formats/format.py", line 808, in _truncate_horizontally
    self.tr_frame = concat((left, right), axis=1)
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/reshape/concat.py", line 359, in concat
    return op.get_result()
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/reshape/concat.py", line 592, in get_result
    new_data = concatenate_managers(
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/internals/concat.py", line 202, in concatenate_managers
    return _concat_managers_axis0(mgrs_indexers, axes, copy)
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/internals/concat.py", line 287, in _concat_managers_axis0
    return BlockManager(tuple(blocks), axes)
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 958, in __init__
    self._verify_integrity()
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 965, in _verify_integrity
    raise construction_error(tot_items, block.shape[1:], self.axes)
ValueError: Shape of passed values is (126, 25, 3), indices imply (21, 126)

>>> data.data
WARNING: SunpyUserWarning: Using .data to access the dataframe is discouraged; use .to_dataframe() instead. [sunpy.timeseries.timeseriesbase]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/frame.py", line 1012, in __repr__
    return self.to_string(**repr_params)
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/frame.py", line 1175, in to_string
    formatter = fmt.DataFrameFormatter(
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/io/formats/format.py", line 604, in __init__
    self.truncate()
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/io/formats/format.py", line 790, in truncate
    self._truncate_horizontally()
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/io/formats/format.py", line 808, in _truncate_horizontally
    self.tr_frame = concat((left, right), axis=1)
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/reshape/concat.py", line 359, in concat
    return op.get_result()
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/reshape/concat.py", line 592, in get_result
    new_data = concatenate_managers(
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/internals/concat.py", line 202, in concatenate_managers
    return _concat_managers_axis0(mgrs_indexers, axes, copy)
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/internals/concat.py", line 287, in _concat_managers_axis0
    return BlockManager(tuple(blocks), axes)
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 958, in __init__
    self._verify_integrity()
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 965, in _verify_integrity
    raise construction_error(tot_items, block.shape[1:], self.axes)
ValueError: Shape of passed values is (126, 25, 3), indices imply (21, 126)

>>> data.to_table()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/sunpy/timeseries/timeseriesbase.py", line 628, in to_table
    table = Table.from_pandas(self._data)
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/astropy/table/table.py", line 3841, in from_pandas
    masks = [np.array(column.isnull()) for column in columns]
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/astropy/table/table.py", line 3841, in <listcomp>
    masks = [np.array(column.isnull()) for column in columns]
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/series.py", line 5283, in isnull
    return super().isnull()
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/generic.py", line 7299, in isnull
    return isna(self).__finalize__(self, method="isnull")
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/dtypes/missing.py", line 143, in isna
    return _isna(obj)
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/dtypes/missing.py", line 183, in _isna
    result = obj._constructor(result, index=obj.index, name=obj.name, copy=False)
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/series.py", line 443, in __init__
    com.require_length_match(data, index)
  File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/pandas/core/common.py", line 557, in require_length_match
    raise ValueError(
ValueError: Length of values (25) does not match length of index (21)


>>> # in the following, trying some other ISOIS data products (different time resolution and/or sub-instruments)
>>> df, data = load_psp('PSP_ISOIS-EPIHI_L2-HET-RATES60')
>>> print(df)
[..]
ValueError: Shape of passed values is (126, 25, 3), indices imply (1222, 126)

>>> df, data = load_psp('PSP_ISOIS-EPIHI_L2-HET-RATES300')
>>> print(df)
[..]
ValueError: Shape of passed values is (126, 50, 3), indices imply (10, 126)

>>> df, data = load_psp('PSP_ISOIS-EPIHI_L2-LET1-RATES60')
>>> print(df)
[..]
ValueError: Shape of passed values is (126, 25, 3), indices imply (1223, 126)

>>> df, data = load_psp('PSP_ISOIS-EPIHI_L2-LET2-RATES60')
>>> print(df)
[..]
ValueError: Shape of passed values is (126, 25, 3), indices imply (1223, 126)

>>> df, data = load_psp('PSP_ISOIS-EPIHI_L2-LET2-RATES3600')
>>> print(df)
[..]
ValueError: Shape of passed values is (126, 25, 3), indices imply (21, 126)
```




### What happened?

See output above.

### Expected behavior

Yielding a Pandas dataframe or Astropy table.

### Screenshots

_No response_

### System Details

``` python
==============================
sunpy Installation Information
==============================

General
#######
OS: Ubuntu (20.04, Linux 5.11.0-46-generic)
Arch: 64bit, (x86_64)
sunpy: 3.1.3
Installation path: /home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages

Required Dependencies
#####################
astropy: 5.0
numpy: 1.22.1
packaging: 21.3
parfive: 1.5.1

Optional Dependencies
#####################
asdf: Missing, need asdf>=2.6.0; extra == "all"
beautifulsoup4: 4.10.0
cdflib: 0.3.20
dask: 2022.1.0
drms: 0.6.2
glymur: 0.9.7.post1
h5netcdf: 0.13.1
h5py: 3.6.0
matplotlib: 3.5.1
mpl-animators: 1.0.0
pandas: 1.4.0
python-dateutil: 2.8.2
reproject: Missing, need reproject; extra == "all"
scikit-image: 0.18.3
scipy: 1.7.3
sqlalchemy: 1.4.31
tqdm: 4.62.3
zeep: 4.1.0
```

### Installation method

conda


Discussion:
The issue here is that at least some of the data is 3 dimensional - depending on time, and two other dimensions. We currently allow 1 dimensional (e.g. density) data, and 2 dimensional data (e.g. magnetic field with 3 components), but pretend the 2D data is 1D by splitting it into individual columns. I don't think `TimeSeries` is designed for handling anything other than a set of 1D data columns, so I'm going to suggest we raise a warning and ignore any variables that have data with dimensions >= 3. Any thoughts on taking that approach?
At least that approach would allow the <3-dimensional data to be read. 

For example, for the file `psp_isois-epihi_l2-het-rates60_20210701_v3.2.0.cdf` there are 198 rVariables & zVariables, but only 14 of them - all connected to sectored measurements - have an additional `DEPEND_2`. (And actually this could be ignored because this dependency is just the following array with the sector number: `array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24], dtype=int8)`. But I don't know if one wants to start messing with the given dependencies...) 

### EDIT: The following problem doesn't seem to be connected with this issue (see next comment)

> In this special case, it's also strange that if I try to get the 3-D data directly with `cdflib`, for all of them I get a `No records found for variable` error, so I'm wondering if the data files are as they should be. E.g.:
> ```python
> >>> import cdflib
> >>> file = "/home/gieseler/sunpy/data/psp_isois-epihi_l2-het-rates60_20210701_v11.cdf"
> >>> cdf = cdflib.CDF(file)
> >>> cdf_info = cdf.cdf_info()
> >>> all_var_keys = cdf_info['rVariables'] + cdf_info['zVariables']
> >>> var_attrs = {key: cdf.varattsget(key) for key in all_var_keys}
> >>> var_attrs["B_He_SECT_Rate"]
> {'DICT_KEY': 'particle_flux>differential_directional_number_rate', 'DISPLAY_TYPE': 'spectrogram', 'VAR_TYPE': 'data', 'UNITS': 'counts s!E-1!N', 'SCALEMAX': array([1000.], dtype=float32), 'SCALEMIN': array([1.e-04], dtype=float32), 'VALIDMAX': array([20000000.], dtype=float32), 'VALIDMIN': array([0.], dtype=float32), 'MNEMONIC': 'B_He_SECT>rate', 'CATDESC': 'He sectored count rate side B', 'LABLAXIS': 'He rate B', 'FIELDNAM': 'B_He_SECT_Rate', 'DEPEND_0': 'Epoch', 'FILLVAL': array([-1.e+31], dtype=float32), 'FORMAT': 'F10.1', 'DEPEND_1': 'He_SECT_ENERGY', 'LABL_PTR_1': 'He_SECT_ENERGY_LABL', 'SCALETYP': 'log', 'LABL_PTR_2': 'HET_R17_SECTORS_LABL', 'DEPEND_2': 'HET_R17_SECTORS'}
> >>> cdf["B_He_SECT_Uncertainty"]
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/cdflib/cdfread.py", line 147, in __getitem__
>     return self.varget(variable)
>   File "/home/gieseler/anaconda3/envs/test310/lib/python3.10/site-packages/cdflib/cdfread.py", line 549, in varget
>     raise ValueError(f'No records found for variable {variable}')
> ValueError: No records found for variable B_He_SECT_Uncertainty
> ```
> (Didn't yet manage to try the other data products, though.)
Addendum: The `No records found for variable` error I mentioned above only occurs for PSP_ISOIS-EPIHI_L2-**HET**-RATES, while PSP_ISOIS-EPIHI_L2-**LET1**-RATES and PSP_ISOIS-EPIHI_L2-**LET2**-RATES do contain sectored ("3-dimensional") data. So it seems to be a separate problem with the HET data, but doesn't really have to do with our issue here.

Resolving patch:
diff --git a/sunpy/io/cdf.py b/sunpy/io/cdf.py
--- a/sunpy/io/cdf.py
+++ b/sunpy/io/cdf.py
@@ -89,7 +89,10 @@ def read_cdf(fname):
                           'Assigning dimensionless units.')
                 unit = u.dimensionless_unscaled
 
-            if data.ndim == 2:
+            if data.ndim > 2:
+                # Skip data with dimensions >= 3 and give user warning
+                warn_user(f'The variable "{var_key}" has been skipped because it has more than 2 dimensions, which is unsupported.')
+            elif data.ndim == 2:
                 # Multiple columns, give each column a unique label
                 for i, col in enumerate(data.T):
                     df[var_key + f'_{i}'] = col


Test patch: