Repository: sunpy/sunpy

Issue:
Edit sunpy.io._cdf.read_cdf to avoid Pandas PerformanceWarning: DataFrame is highly fragmented.
### Describe the performance issue

When reading cdf files with a lot of variables, for each variable `PerformanceWarning: DataFrame is highly fragmented.` is raised by Pandas, as each variable is added to a DataFrame using `df[var_key] = data`.

### To Reproduce

``` python
>>> # Importing sunpy_soar registers the client with sunpy
>>> import sunpy_soar
>>> from sunpy.net import Fido
>>> from sunpy.timeseries import TimeSeries
>>> import sunpy.net.attrs as a
>>> 
>>> # Create search attributes
>>> time = a.Time('2022-02-01', '2022-02-01')
>>> level = a.Level(2)
>>> product = a.soar.Product('EPD-STEP-MAIN')
>>> 
>>> # Do search
>>> result = Fido.search(time & level & product)
>>> print(result)
>>> 
>>> # Download files
>>> files = Fido.fetch(result)
>>> print(files)
>>> 
>>> # Read files, using _read_cdf in the background
>>> solo_step = TimeSeries(files, concatenate=True)
  df[var_key + f'_{i}'] = col
/home/jagies/mambaforge/envs/dev_solo/lib/python3.9/site-packages/sunpy/io/_cdf.py:123: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  df[var_key] = data
/home/jagies/mambaforge/envs/dev_solo/lib/python3.9/site-packages/sunpy/io/_cdf.py:123: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  df[var_key] = data
/home/jagies/mambaforge/envs/dev_solo/lib/python3.9/site-packages/sunpy/io/_cdf.py:123: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  df[var_key] = data
WARNING: SunpyUserWarning: The variable "RTN_Pixels" has been skipped because it has more than 2 dimensions, which is unsupported. [sunpy.io._cdf]
```
(skipped a lot of the PerformanceWarnings as they go over mulitple screens)

### Proposed fix

According to some comments online (e.g., https://stackoverflow.com/questions/75766112/performancewarning-dataframe-is-highly-fragmented-when-creating-new-dataframe-c), it's memory wise more efficient to gather the data for all variables in a dictionary throughout the for-loop (over each variable), and in the end concat it to the desired DataFrame. 

I implemented those changes and will make a corresponding PR for it in a second. That gets rid of the PerformanceWarnings, and it seems to run a little bit faster on my machine, for the above example using `%timeit -n1 -r3` it's `7.69 s ± 10.8 ms per loop (mean ± std. dev. of 3 runs, 1 loop each)` vs. `6.79 s ± 67.5 ms per loop (mean ± std. dev. of 3 runs, 1 loop each)`. 

I don't have any official statement that this fix is really better or the way to go, but as it removes annoying warnings and seems to be not slower, maybe it's worthwhile to implement it.
Edit sunpy.io._cdf.read_cdf to avoid Pandas PerformanceWarning: DataFrame is highly fragmented.
### Describe the performance issue

When reading cdf files with a lot of variables, for each variable `PerformanceWarning: DataFrame is highly fragmented.` is raised by Pandas, as each variable is added to a DataFrame using `df[var_key] = data`.

### To Reproduce

``` python
>>> # Importing sunpy_soar registers the client with sunpy
>>> import sunpy_soar
>>> from sunpy.net import Fido
>>> from sunpy.timeseries import TimeSeries
>>> import sunpy.net.attrs as a
>>> 
>>> # Create search attributes
>>> time = a.Time('2022-02-01', '2022-02-01')
>>> level = a.Level(2)
>>> product = a.soar.Product('EPD-STEP-MAIN')
>>> 
>>> # Do search
>>> result = Fido.search(time & level & product)
>>> print(result)
>>> 
>>> # Download files
>>> files = Fido.fetch(result)
>>> print(files)
>>> 
>>> # Read files, using _read_cdf in the background
>>> solo_step = TimeSeries(files, concatenate=True)
  df[var_key + f'_{i}'] = col
/home/jagies/mambaforge/envs/dev_solo/lib/python3.9/site-packages/sunpy/io/_cdf.py:123: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  df[var_key] = data
/home/jagies/mambaforge/envs/dev_solo/lib/python3.9/site-packages/sunpy/io/_cdf.py:123: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  df[var_key] = data
/home/jagies/mambaforge/envs/dev_solo/lib/python3.9/site-packages/sunpy/io/_cdf.py:123: PerformanceWarning: DataFrame is highly fragmented.  This is usually the result of calling `frame.insert` many times, which has poor performance.  Consider joining all columns at once using pd.concat(axis=1) instead. To get a de-fragmented frame, use `newframe = frame.copy()`
  df[var_key] = data
WARNING: SunpyUserWarning: The variable "RTN_Pixels" has been skipped because it has more than 2 dimensions, which is unsupported. [sunpy.io._cdf]
```
(skipped a lot of the PerformanceWarnings as they go over mulitple screens)

### Proposed fix

According to some comments online (e.g., https://stackoverflow.com/questions/75766112/performancewarning-dataframe-is-highly-fragmented-when-creating-new-dataframe-c), it's memory wise more efficient to gather the data for all variables in a dictionary throughout the for-loop (over each variable), and in the end concat it to the desired DataFrame. 

I implemented those changes and will make a corresponding PR for it in a second. That gets rid of the PerformanceWarnings, and it seems to run a little bit faster on my machine, for the above example using `%timeit -n1 -r3` it's `7.69 s ± 10.8 ms per loop (mean ± std. dev. of 3 runs, 1 loop each)` vs. `6.79 s ± 67.5 ms per loop (mean ± std. dev. of 3 runs, 1 loop each)`. 

I don't have any official statement that this fix is really better or the way to go, but as it removes annoying warnings and seems to be not slower, maybe it's worthwhile to implement it.


Discussion:

Resolving patch:
diff --git a/sunpy/io/_cdf.py b/sunpy/io/_cdf.py
--- a/sunpy/io/_cdf.py
+++ b/sunpy/io/_cdf.py
@@ -60,7 +60,7 @@ def read_cdf(fname):
             continue
         # TODO: use to_astropy_time() instead here when we drop pandas in timeseries
         index = CDFepoch.to_datetime(index)
-        df = pd.DataFrame(index=pd.DatetimeIndex(name=index_key, data=index))
+        df_dict = {}
         units = {}
 
         for var_key in sorted(var_keys):
@@ -111,13 +111,13 @@ def read_cdf(fname):
             elif data.ndim == 2:
                 # Multiple columns, give each column a unique label
                 for i, col in enumerate(data.T):
-                    df[var_key + f'_{i}'] = col
+                    df_dict[var_key + f'_{i}'] = col
                     units[var_key + f'_{i}'] = unit
             else:
                 # Single column
-                df[var_key] = data
+                df_dict[var_key] = data
                 units[var_key] = unit
-
+        df = pd.DataFrame(df_dict, index=pd.DatetimeIndex(name=index_key, data=index))
         all_ts.append(GenericTimeSeries(data=df, units=units, meta=meta))
 
     if not len(all_ts):


Test patch: