Repository: wright-group/WrightTools

Issue:
dataset compression
shall we compress our channels and variables? (I think yes)

if so, how?

[h5py offers 3 lossless compression filters](http://docs.h5py.org/en/latest/high/dataset.html#lossless-compression-filters)

I think we should use GZIP, the most universal compression option we have available


Discussion:
Using a dataset and workup from @darienmorrow's MoS2 project:

Uncompressed: 3.9 MB, 3.893 s 
Gzip - 4: 5.2 MB, 4.078 s
Gzip - 9: 5.2 MB, 4.135 s
lzf - 5.3 MB, 4.051 s


With this test data, using a real set of data from multiple instruments, We see that applying compression filters are a net increase in file size and a degradation in speed.
I think we should at some point ensure that compression is available to the user (in the `create_variable`/`create_channel` functions, which then call `h5py.require_dataset`). However my recommendation is to default to uncompressed.
I am therefore moving this to the "future", and adding the 'Help Wanted' label.

http://docs.h5py.org/en/latest/high/group.html#Group.create_dataset

In particular, I think simulations are going to benefit more from compression, as they may have a) more points than can practically be collected in lab and b) more repetition of values.

I think a significant contributor to the file size being large is the number of 1 by 1 datasets used to store variables that are not moving. These are quite useful in data analysis, but do not compress well.
thanks @ksunden this seems definitive, and I agree with your suspicions

Resolving patch:
diff --git a/WrightTools/data/_data.py b/WrightTools/data/_data.py
--- a/WrightTools/data/_data.py
+++ b/WrightTools/data/_data.py
@@ -885,6 +885,12 @@ def create_channel(
             require_kwargs["dtype"] = values.dtype
         if np.prod(require_kwargs["shape"]) == 1:
             require_kwargs["chunks"] = None
+        if "compression" in kwargs:
+            require_kwargs["compression"] = kwargs["compression"]
+        if "compression_opts" in kwargs:
+            require_kwargs["compression_opts"] = kwargs["compression_opts"]
+        if "shuffle" in kwargs:
+            require_kwargs["shuffle"] = kwargs["shuffle"]
         # create dataset
         dataset_id = self.require_dataset(name=name, **require_kwargs).id
         channel = Channel(self, dataset_id, units=units, **kwargs)
@@ -943,9 +949,18 @@ def create_variable(
             shape = values.shape
             dtype = values.dtype
             fillvalue = None
+        require_kwargs = {"chunks": True}
+        if "compression" in kwargs:
+            require_kwargs["compression"] = kwargs["compression"]
+        if "compression_opts" in kwargs:
+            require_kwargs["compression_opts"] = kwargs["compression_opts"]
+        if "shuffle" in kwargs:
+            require_kwargs["shuffle"] = kwargs["shuffle"]
+        if np.prod(shape) == 1:
+            require_kwargs["chunks"] = None
         # create dataset
         id = self.require_dataset(
-            name=name, data=values, shape=shape, dtype=dtype, fillvalue=fillvalue
+            name=name, data=values, shape=shape, dtype=dtype, fillvalue=fillvalue, **require_kwargs
         ).id
         variable = Variable(self, id, units=units, **kwargs)
         # finish


Test patch:
diff --git a/tests/data/dataset_creation.py b/tests/data/dataset_creation.py
--- a/tests/data/dataset_creation.py
+++ b/tests/data/dataset_creation.py
@@ -4,6 +4,8 @@
 import numpy as np
 import pytest
 
+import os
+
 
 def test_create_variable():
     data = wt.Data()
@@ -31,6 +33,13 @@ def test_exception():
         d.create_channel(name="w1")
 
 
+def test_create_compressed_channel():
+    data = wt.Data()
+    child1 = data.create_channel("hi", shape=(1024, 1024), compression="gzip")
+    data["hi"][:] = 0
+    assert os.path.getsize(data.filepath) < 1e6
+
+
 if __name__ == "__main__":
     test_create_variable()
     test_create_channel()