Repository: BAMWelDX/weldx

Issue:
set default inline threshold
I think it would make sense to set a (sensible) default threshold to inline array values for most of our use cases

I think 10 should be a good value because it will inline a single (static) 3x rotation matrix and short list of vectors e.g. movements between two points
```yaml
coordinates: !weldx!core/variable-0.1.0
    name: coordinates
    dimensions: [time, c]
    dtype: <f8
    data: !core/ndarray-1.0.0
      data:
      - [20.0, 0.0, 3.0]
      - [330.0, 0.0, 3.0]
      datatype: float64
      shape: [2, 3]
```

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/BAMWelDX/weldx/pull/643?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BAMWelDX) Report
> Merging [#643](https://codecov.io/gh/BAMWelDX/weldx/pull/643?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BAMWelDX) (4a5c011) into [master](https://codecov.io/gh/BAMWelDX/weldx/commit/44e3475e78500cdd47664fbdff78be7fef59228f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BAMWelDX) (44e3475) will **increase** coverage by `0.01%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/BAMWelDX/weldx/pull/643/graphs/tree.svg?width=650&height=150&src=pr&token=wdof1qQTsn&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BAMWelDX)](https://codecov.io/gh/BAMWelDX/weldx/pull/643?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BAMWelDX)

```diff
@@            Coverage Diff             @@
##           master     #643      +/-   ##
==========================================
+ Coverage   94.49%   94.51%   +0.01%     
==========================================
  Files          93       93              
  Lines        6018     6032      +14     
==========================================
+ Hits         5687     5701      +14     
  Misses        331      331              
```


| [Impacted Files](https://codecov.io/gh/BAMWelDX/weldx/pull/643?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BAMWelDX) | Coverage Δ | |
|---|---|---|
| [weldx/asdf/file.py](https://codecov.io/gh/BAMWelDX/weldx/pull/643/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BAMWelDX#diff-d2VsZHgvYXNkZi9maWxlLnB5) | `96.25% <100.00%> (+0.20%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/BAMWelDX/weldx/pull/643?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BAMWelDX).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BAMWelDX)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/BAMWelDX/weldx/pull/643?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BAMWelDX). Last update [44e3475...4a5c011](https://codecov.io/gh/BAMWelDX/weldx/pull/643?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BAMWelDX). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=BAMWelDX).

github-actions[bot]: ## Unit Test Results
       1 files  ±0         1 suites  ±0   1m 42s [:stopwatch:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "duration of all tests") -7s
1 934 tests +4  1 934 [:heavy_check_mark:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "passed tests") +4  0 [:zzz:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "skipped / disabled tests") ±0  0 [:x:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "failed tests") ±0 

Results for commit 4a5c0116. ± Comparison against base commit 04327b79.


Patch:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 19501b8..25677fa 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -9,6 +9,12 @@
 added
 =====
 
+- `WeldxFile` handles an ``array_inline_threshold`` parameter to indicate if short arrays
+  will be serialized as strings, or as binary block. Note that this does not affect arrays,
+  which are being shared across several objects in the same file.
+  `[#643] <https://github.com/BAMWelDX/weldx/pull/643>`__
+
+
 removed
 =======
 
diff --git a/weldx/asdf/file.py b/weldx/asdf/file.py
index 1e7c557..d779263 100644
--- a/weldx/asdf/file.py
+++ b/weldx/asdf/file.py
@@ -22,7 +22,7 @@ from typing import (
 
 import asdf
 import numpy as np
-from asdf import AsdfFile, generic_io
+from asdf import AsdfFile, config_context, generic_io
 from asdf import open as open_asdf
 from asdf.exceptions import AsdfWarning
 from asdf.tags.core import Software
@@ -47,6 +47,7 @@ __all__ = [
     "WeldxFile",
     "DEFAULT_ARRAY_COMPRESSION",
     "DEFAULT_ARRAY_COPYING",
+    "DEFAULT_ARRAY_INLINE_THRESHOLD",
     "_PROTECTED_KEYS",
 ]
 
@@ -72,6 +73,9 @@ DEFAULT_ARRAY_COMPRESSION = "input"
 DEFAULT_ARRAY_COPYING = True
 """Stored Arrays will be copied to memory, or not. If False, use memory mapping."""
 
+DEFAULT_ARRAY_INLINE_THRESHOLD = 10
+"""Arrays with less or equal elements will be inlined (stored as string, not binary)."""
+
 _PROTECTED_KEYS = (
     "history",
     "asdf_library",
@@ -139,6 +143,10 @@ class WeldxFile(_ProtectedViewDict):
         When `False`, when reading files, attempt to memory map (memmap) underlying data
         arrays when possible. This avoids blowing the memory when working with very
         large datasets.
+    array_inline_threshold :
+        arrays below this threshold will be serialized as string, if larger as binary
+        block. Note that this does not affect arrays, which are being shared across
+        several objects in the same file.
 
     Examples
     --------
@@ -204,6 +212,7 @@ class WeldxFile(_ProtectedViewDict):
         software_history_entry: Mapping = None,
         compression: str = DEFAULT_ARRAY_COMPRESSION,
         copy_arrays: bool = DEFAULT_ARRAY_COPYING,
+        array_inline_threshold: int = DEFAULT_ARRAY_INLINE_THRESHOLD,
     ):
         if write_kwargs is None:
             write_kwargs = dict(all_array_compression=compression)
@@ -211,6 +220,9 @@ class WeldxFile(_ProtectedViewDict):
         if asdffile_kwargs is None:
             asdffile_kwargs = dict(copy_arrays=copy_arrays)
 
+        # this parameter is now (asdf-2.8) a asdf.config parameter, so we store it here.
+        self._array_inline_threshold = array_inline_threshold
+
         # TODO: ensure no mismatching args for compression and copy_arrays.
         self._write_kwargs = write_kwargs
         self._asdffile_kwargs = asdffile_kwargs
@@ -300,6 +312,21 @@ class WeldxFile(_ProtectedViewDict):
         # initialize protected key interface.
         super().__init__(protected_keys=_PROTECTED_KEYS, data=self._asdf_handle.tree)
 
+    @contextmanager
+    def _config_context(self, **kwargs):
+        # Temporarily set (default) options in asdf.config_context. This is useful
+        # during writing/updating data.
+        if (
+            "array_inline_threshold" not in kwargs
+            or kwargs["array_inline_threshold"] is None
+        ):
+            kwargs["array_inline_threshold"] = self._array_inline_threshold
+
+        with config_context() as config:
+            for k, v in kwargs.items():
+                setattr(config, k, v)
+            yield
+
     def _write_tree(
         self, filename_or_path_like, tree, asdffile_kwargs, write_kwargs, created
     ) -> AsdfFile:
@@ -310,7 +337,8 @@ class WeldxFile(_ProtectedViewDict):
         # 2. file exists, but should be updated with new tree
         if created:
             asdf_file = asdf.AsdfFile(tree=tree, **asdffile_kwargs)
-            asdf_file.write_to(filename_or_path_like, **write_kwargs)
+            with self._config_context():
+                asdf_file.write_to(filename_or_path_like, **write_kwargs)
             generic_file = generic_io.get_file(filename_or_path_like, mode="rw")
             asdf_file._fd = generic_file
         else:
@@ -318,7 +346,8 @@ class WeldxFile(_ProtectedViewDict):
                 raise RuntimeError("inconsistent mode, need to write data.")
             asdf_file = open_asdf(filename_or_path_like, **asdffile_kwargs, mode="rw")
             asdf_file.tree = tree
-            asdf_file.update(**write_kwargs)
+            with self._config_context():
+                asdf_file.update(**write_kwargs)
         return asdf_file
 
     @property
@@ -454,14 +483,14 @@ class WeldxFile(_ProtectedViewDict):
         **kwargs,
     ):
         """Get this docstring overwritten by AsdfFile.update."""
-        self._asdf_handle.update(
-            all_array_storage=all_array_storage,
-            all_array_compression=all_array_compression,
-            pad_blocks=pad_blocks,
-            include_block_index=include_block_index,
-            version=version,
-            **kwargs,
-        )
+        with self._config_context(**kwargs):
+            self._asdf_handle.update(
+                all_array_storage=all_array_storage,
+                all_array_compression=all_array_compression,
+                pad_blocks=pad_blocks,
+                include_block_index=include_block_index,
+                version=version,
+            )
 
     sync.__doc__ = AsdfFile.update.__doc__
 
@@ -719,6 +748,7 @@ class WeldxFile(_ProtectedViewDict):
             write_kwargs=self._write_kwargs,
             sync=self.sync_upon_close,
             software_history_entry=self.software_history_entry,
+            array_inline_threshold=self._array_inline_threshold,
         )
         return wx
 
@@ -758,7 +788,10 @@ class WeldxFile(_ProtectedViewDict):
         return AttrDict(self)
 
     def write_to(
-        self, fd: Optional[types_path_and_file_like] = None, **write_args
+        self,
+        fd: Optional[types_path_and_file_like] = None,
+        array_inline_threshold=None,
+        **write_args,
     ) -> Optional[types_path_and_file_like]:
         """Write current contents to given file name or file type.
 
@@ -769,6 +802,10 @@ class WeldxFile(_ProtectedViewDict):
             object. If a string path, the file is automatically
             closed after writing. If `None` is given, write to a new buffer.
 
+        array_inline_threshold :
+            arrays below this threshold will be serialized as string, if
+            larger as binary block.
+
         write_args :
             Allowed parameters:
 
@@ -792,7 +829,8 @@ class WeldxFile(_ProtectedViewDict):
         if not write_args:
             write_args = self._write_kwargs
 
-        self._asdf_handle.write_to(fd, **write_args)
+        with self._config_context(array_inline_threshold=array_inline_threshold):
+            self._asdf_handle.write_to(fd, **write_args)
 
         if isinstance(fd, types_file_like.__args__):
             fd.seek(0)


Test patch:
diff --git a/weldx/tests/asdf_tests/test_weldx_file.py b/weldx/tests/asdf_tests/test_weldx_file.py
index 76491df..ce168be 100644
--- a/weldx/tests/asdf_tests/test_weldx_file.py
+++ b/weldx/tests/asdf_tests/test_weldx_file.py
@@ -14,7 +14,7 @@ from jsonschema import ValidationError
 
 from weldx import WeldxFile
 from weldx.asdf.cli.welding_schema import single_pass_weld_example
-from weldx.asdf.file import _PROTECTED_KEYS
+from weldx.asdf.file import _PROTECTED_KEYS, DEFAULT_ARRAY_INLINE_THRESHOLD
 from weldx.asdf.util import get_schema_path
 from weldx.types import SupportsFileReadWrite
 from weldx.util import WeldxDeprecationWarning, compare_nested
@@ -502,17 +502,18 @@ class TestWeldXFile:
             f.update()
             f.close()
 
-        # compare data
-        assert (
-            pathlib.Path("test.asdf").stat().st_size
-            == pathlib.Path("test.wx").stat().st_size
-        )
+        # file sizes should be almost equal (array inlining in wxfile).
+        a = pathlib.Path("test.asdf").stat().st_size
+        b = pathlib.Path("test.wx").stat().st_size
+        assert a >= b
+
+        if a == b:
 
-        def _read(fn):
-            with open(fn, "br") as fh:
-                return fh.read()
+            def _read(fn):
+                with open(fn, "br") as fh:
+                    return fh.read()
 
-        assert _read("test.asdf") == _read("test.wx")
+            assert _read("test.asdf") == _read("test.wx")
 
     @pytest.mark.filterwarnings("ignore:You tried to manipulate an ASDF internal")
     @pytest.mark.filterwarnings("ignore:Call to deprecated function data.")
@@ -553,3 +554,41 @@ class TestWeldXFile:
 
         for x in iter(self.fh):
             assert x not in _PROTECTED_KEYS
+
+    @staticmethod
+    def test_array_inline_threshold():
+        """Test array inlining threshold."""
+        x = np.arange(7)
+        buff = WeldxFile(
+            tree=dict(x=x), mode="rw", array_inline_threshold=len(x) + 1
+        ).file_handle
+        buff.seek(0)
+        assert str(list(x)).encode("utf-8") in buff.read()
+
+        # now we create an array longer than the default threshold
+        y = np.arange(DEFAULT_ARRAY_INLINE_THRESHOLD + 3)
+        buff2 = WeldxFile(tree=dict(x=y), mode="rw").file_handle
+        buff2.seek(0)
+        data = buff2.read()
+        assert b"BLOCK" in data
+        assert str(list(y)).encode("utf-8") not in data
+
+    @staticmethod
+    def test_array_inline_threshold_sync():
+        x = np.arange(DEFAULT_ARRAY_INLINE_THRESHOLD + 3)
+
+        with WeldxFile(mode="rw") as wx:
+            wx.update(x=x)
+            wx.sync(array_inline_threshold=len(x) + 1)
+            buff3 = wx.file_handle
+            buff3.seek(0)
+            data3 = buff3.read()
+            assert b"BLOCK" not in data3
+
+    @staticmethod
+    def test_array_inline_threshold_write_to():
+        x = np.arange(DEFAULT_ARRAY_INLINE_THRESHOLD + 3)
+        buff = WeldxFile(tree=dict(x=x), mode="rw").write_to(
+            array_inline_threshold=len(x) + 1
+        )
+        assert b"BLOCK" not in buff.read()