Repository: bluesky/tiled
Pull request #901: Client-side flattening of a container
Author: login: genematx
Created: 2025-02-28T15:10:00Z | Merged: 2025-03-13T19:43:24Z

A third attempt at implementing a flat-namespaced container, this time (mostly) on the client-side. While the data is stored in a new `Composite` container, the flat namespace is implemented by the python client itself: the client keeps a mapping of table column names to their full paths. The distinct structure family serves a role akin to a spec here, but its functionality is thought to be expanded in the future.

Related: #668
Issue: #824

### Checklist
- [x] Add a Changelog entry
- [x] Add the ticket number which this PR closes to the comment section



Diff:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 280f20505..94bc96493 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -3,6 +3,13 @@ Write the date in place of the "Unreleased" in the case a new version is release
 # Changelog
 
 
+## Unreleased
+
+### Added
+
+- `Composite` structure family to enable direct access to table columns in a single namespace.
+
+
 ## 0.1.0-b20 (2025-03-07)
 
 ### Added
diff --git a/docs/source/explanations/structures.md b/docs/source/explanations/structures.md
index 65ce55deb..5b0a99fe4 100644
--- a/docs/source/explanations/structures.md
+++ b/docs/source/explanations/structures.md
@@ -11,6 +11,7 @@ The structure families are:
 
 * array --- a strided array, like a [numpy](https://numpy.org) array
 * awkward --- nested, variable-sized data (as implemented by [AwkwardArray](https://awkward-array.org/))
+* composite --- a container-like structure to combine table columns and arrays in a common namespace
 * container --- a of other structures, akin to a dictionary or a directory
 * sparse --- a sparse array (i.e. an array which is mostly zeros)
 * table --- tabular data, as in [Apache Arrow](https://arrow.apache.org) or
@@ -575,3 +576,12 @@ response.
   "count": 5
 }
 ```
+
+### Composite
+
+This is a specialized container-like structure designed to link together multiple tables and arrays that store
+related scientific data. It does not support nesting but provides a common namespace across all columns of the
+contained tables along with the arrays (thus, name collisions are forbidden). This allows to further abstract out
+the disparate internal storage mechanisms (e.g. Parquet for tables and zarr for arrays) and present the user with a
+smooth homogeneous interface for data access. Composite structures do not support pagination and are not
+recommended for "wide" datasets with more than ~1000 items (columns and arrays) in the namespace.
diff --git a/docs/source/how-to/register.md b/docs/source/how-to/register.md
index 345de187e..6462915b6 100644
--- a/docs/source/how-to/register.md
+++ b/docs/source/how-to/register.md
@@ -112,3 +112,25 @@ client.new(
     specs=[],
 )
 ```
+
+#### Writing a composite structure
+
+A Composite structure allows the user to acess the columns of contained tables in
+a flat namespace along with other sibling arrays. Writing new data to a Composite container
+is analoguous to the usual containers, however exceptions will be raised if there are
+any name collisions.
+
+```python
+import pandas
+
+rng = numpy.random.default_rng(12345)
+arr = rng.random(size=(3, 5), dtype="float64")
+df = pandas.DataFrame({"A": ["one", "two", "three"], "B": [1, 2, 3]})
+
+# Create a Composite node
+node = client.create_composite(key="x")
+
+# Write the data
+node.write_array(arr, key="C")
+node.write_dataframe(df, key="table1")
+```
diff --git a/tiled/_tests/test_composite.py b/tiled/_tests/test_composite.py
new file mode 100644
index 000000000..72b16970b
--- /dev/null
+++ b/tiled/_tests/test_composite.py
@@ -0,0 +1,243 @@
+from pathlib import Path
+
+import awkward
+import numpy
+import pandas
+import pytest
+import sparse
+import tifffile as tf
+
+from ..catalog import in_memory
+from ..client import Context, from_context
+from ..server.app import build_app
+from ..structures.array import ArrayStructure, BuiltinDtype
+from ..structures.core import StructureFamily
+from ..structures.data_source import Asset, DataSource, Management
+from ..structures.table import TableStructure
+
+rng = numpy.random.default_rng(12345)
+
+df1 = pandas.DataFrame({"A": ["one", "two", "three"], "B": [1, 2, 3]})
+df2 = pandas.DataFrame(
+    {
+        "C": ["red", "green", "blue", "white"],
+        "D": [10.0, 20.0, 30.0, 40.0],
+        "E": [0, 0, 0, 0],
+    }
+)
+df3 = pandas.DataFrame(
+    {
+        "col1": ["one", "two", "three", "four", "five"],
+        "col2": [1.0, 2.0, 3.0, 4.0, 5.0],
+    }
+)
+arr1 = rng.random(size=(3, 5), dtype="float64")
+arr2 = rng.integers(0, 255, size=(5, 7, 3), dtype="uint8")
+img_data = rng.integers(0, 255, size=(5, 13, 17, 3), dtype="uint8")
+
+# An awkward array
+awk_arr = awkward.Array(
+    [
+        [{"x": 1.1, "y": [1]}, {"x": 2.2, "y": [1, 2]}],
+        [],
+        [{"x": 3.3, "y": [1, 2, 3]}],
+    ]
+)
+awk_packed = awkward.to_packed(awk_arr)
+awk_form, awk_length, awk_container = awkward.to_buffers(awk_packed)
+
+# A sparse array
+arr = rng.random(size=(10, 20, 30), dtype="float64")
+arr[arr < 0.95] = 0  # Fill half of the array with zeros.
+sps_arr = sparse.COO(arr)
+
+md = {"md_key1": "md_val1", "md_key2": 2}
+
+
+@pytest.fixture(scope="module")
+def tree(tmp_path_factory):
+    return in_memory(writable_storage=tmp_path_factory.getbasetemp())
+
+
+@pytest.fixture(scope="module")
+def context(tree):
+    with Context.from_app(build_app(tree)) as context:
+        client = from_context(context)
+        x = client.create_composite(key="x", metadata=md)
+        x.write_array(arr1, key="arr1", metadata={"md_key": "md_for_arr1"})
+        x.write_array(arr2, key="arr2", metadata={"md_key": "md_for_arr2"})
+        x.write_dataframe(
+            df1,
+            key="df1",
+            metadata={
+                "md_key": "md_for_df1",
+                "A": {"md_key": "md_for_A"},
+                "B": {"md_key": "md_for_B"},
+            },
+        )
+        x.write_dataframe(
+            df2,
+            key="df2",
+            metadata={
+                "md_key": "md_for_df2",
+                "C": {"md_key": "md_for_C"},
+                "D": {"md_key": "md_for_D"},
+                "E": {"md_key": "md_for_E"},
+            },
+        )
+        x.write_awkward(awk_arr, key="awk", metadata={"md_key": "md_for_awk"})
+        x.write_sparse(
+            coords=sps_arr.coords,
+            data=sps_arr.data,
+            shape=sps_arr.shape,
+            key="sps",
+            metadata={"md_key": "md_for_sps"},
+        )
+
+        yield context
+
+
+@pytest.fixture
+def tiff_sequence(tmpdir):
+    sequence_directory = Path(tmpdir, "sequence")
+    sequence_directory.mkdir()
+    filepaths = []
+    for i in range(img_data.shape[0]):
+        fpath = sequence_directory / f"temp{i:05}.tif"
+        tf.imwrite(fpath, img_data[i, ...])
+        filepaths.append(fpath)
+
+    yield filepaths
+
+
+@pytest.fixture
+def csv_file(tmpdir):
+    fpath = Path(tmpdir, "test.csv")
+    df3.to_csv(fpath, index=False)
+
+    yield fpath
+
+
+@pytest.mark.parametrize(
+    "name, expected",
+    [
+        ("A", df1["A"]),
+        ("B", df1["B"]),
+        ("C", df2["C"]),
+        ("D", df2["D"]),
+        ("E", df2["E"]),
+        ("arr1", arr1),
+        ("arr2", arr2),
+        ("awk", awk_arr),
+        ("sps", sps_arr.todense()),
+    ],
+)
+def test_reading(context, name, expected):
+    client = from_context(context)
+    actual = client["x"][name].read()
+    if name == "sps":
+        actual = actual.todense()
+    assert numpy.array_equal(actual, expected)
+
+
+def test_iterate_parts(context):
+    client = from_context(context)
+    for part in client["x"].parts:
+        client["x"].parts[part].read()
+
+
+def test_iterate_columns(context):
+    client = from_context(context)
+    for col, _client in client["x"].items():
+        read_from_client = _client.read()
+        read_from_column = client["x"][col].read()
+        read_from_full_path = client[f"x/{col}"].read()
+        if col == "sps":
+            read_from_client = read_from_client.todense()
+            read_from_column = read_from_column.todense()
+            read_from_full_path = read_from_full_path.todense()
+        assert numpy.array_equal(read_from_client, read_from_column)
+        assert numpy.array_equal(read_from_client, read_from_full_path)
+        assert numpy.array_equal(read_from_full_path, read_from_column)
+
+
+def test_metadata(context):
+    client = from_context(context)
+    assert client["x"].metadata == md
+
+    # Check metadata for each part
+    for part in client["x"].parts:
+        c = client["x"].parts[part]
+        assert c.metadata["md_key"] == f"md_for_{part}"
+
+
+def test_parts_not_direclty_accessible(context):
+    client = from_context(context)
+    client["x"].parts["df1"].read()
+    client["x"].parts["df1"]["A"].read()
+    client["x"]["A"].read()
+    with pytest.raises(KeyError):
+        client["x"]["df1"].read()
+    with pytest.raises(KeyError):
+        client["x/df1"].read()
+
+
+def test_external_assets(context, tiff_sequence, csv_file):
+    client = from_context(context)
+    tiff_assets = [
+        Asset(
+            data_uri=f"file://localhost/{fpath}",
+            is_directory=False,
+            parameter="data_uris",
+            num=i + 1,
+        )
+        for i, fpath in enumerate(tiff_sequence)
+    ]
+    tiff_structure_0 = ArrayStructure(
+        data_type=BuiltinDtype.from_numpy_dtype(numpy.dtype("uint8")),
+        shape=(5, 13, 17, 3),
+        chunks=((1, 1, 1, 1, 1), (13,), (17,), (3,)),
+    )
+    tiff_data_source = DataSource(
+        mimetype="multipart/related;type=image/tiff",
+        assets=tiff_assets,
+        structure_family=StructureFamily.array,
+        structure=tiff_structure_0,
+        management=Management.external,
+    )
+
+    csv_assets = [
+        Asset(
+            data_uri=f"file://localhost/{csv_file}",
+            is_directory=False,
+            parameter="data_uris",
+        )
+    ]
+    csv_data_source = DataSource(
+        mimetype="text/csv",
+        assets=csv_assets,
+        structure_family=StructureFamily.table,
+        structure=TableStructure.from_pandas(df3),
+        management=Management.external,
+    )
+
+    y = client.create_composite(key="y")
+    y.new(
+        structure_family=StructureFamily.array,
+        data_sources=[tiff_data_source],
+        key="image",
+    )
+    y.new(
+        structure_family=StructureFamily.table,
+        data_sources=[csv_data_source],
+        key="table",
+    )
+
+    arr = y.parts["image"].read()
+    assert numpy.array_equal(arr, img_data)
+
+    df = y.parts["table"].read()
+    for col in df.columns:
+        assert numpy.array_equal(df[col], df3[col])
+
+    assert set(y.keys()) == {"image", "col1", "col2"}
diff --git a/tiled/_tests/test_inlined_contents.py b/tiled/_tests/test_inlined_contents.py
index 5fc6224e2..ac70e9538 100644
--- a/tiled/_tests/test_inlined_contents.py
+++ b/tiled/_tests/test_inlined_contents.py
@@ -117,10 +117,6 @@ def test_too_wide_for_inline():
     It is fetched in pages on demand, as usual with nodes.
     """
 
-    class Adapter(MapAdapter):
-        def inlined_contents_enabled(self):
-            return True
-
     a = numpy.array([1])
     tree = MapAdapter(
         {
diff --git a/tiled/_tests/test_writing.py b/tiled/_tests/test_writing.py
index dcc9aed91..ce0a7a109 100644
--- a/tiled/_tests/test_writing.py
+++ b/tiled/_tests/test_writing.py
@@ -18,6 +18,7 @@
 from starlette.status import (
     HTTP_404_NOT_FOUND,
     HTTP_409_CONFLICT,
+    HTTP_415_UNSUPPORTED_MEDIA_TYPE,
     HTTP_422_UNPROCESSABLE_ENTITY,
 )
 
@@ -627,7 +628,7 @@ def test_write_with_specified_mimetype(tree):
             assert x.data_sources()[0].mimetype == mimetype
 
         # Specifying unsupported mimetype raises expected error.
-        with fail_with_status_code(415):
+        with fail_with_status_code(HTTP_415_UNSUPPORTED_MEDIA_TYPE):
             client.new(
                 "table",
                 [
@@ -678,3 +679,89 @@ def test_append_partition(
         x.append_partition(orig_table, 0)
         x.append_partition(table_to_append, 0)
         assert_frame_equal(x.read(), pandas.DataFrame(expected_file), check_dtype=False)
+
+
+def test_composite_one_table(tree):
+    with Context.from_app(build_app(tree)) as context:
+        client = from_context(context)
+        df = pandas.DataFrame({"A": [], "B": []})
+        client.create_composite(key="x")
+        client["x"].write_dataframe(df)
+        assert len(client["x"].parts) == 1
+
+
+def test_composite_two_tables(tree):
+    with Context.from_app(build_app(tree)) as context:
+        client = from_context(context)
+        df1 = pandas.DataFrame({"A": [], "B": []})
+        df2 = pandas.DataFrame({"C": [], "D": [], "E": []})
+        x = client.create_composite(key="x")
+        x.write_dataframe(df1, key="table1")
+        x.write_dataframe(df2, key="table2")
+        x.parts["table1"].read()
+        x.parts["table2"].read()
+
+
+def test_composite_two_tables_colliding_names(tree):
+    with Context.from_app(build_app(tree)) as context:
+        client = from_context(context)
+        df1 = pandas.DataFrame({"A": [], "B": []})
+        df2 = pandas.DataFrame({"C": [], "D": [], "E": []})
+        x = client.create_composite(key="x")
+        x.write_dataframe(df1, key="table1")
+        with fail_with_status_code(HTTP_409_CONFLICT):
+            x.write_dataframe(df2, key="table1")
+
+
+def test_composite_two_tables_colliding_keys(tree):
+    with Context.from_app(build_app(tree)) as context:
+        client = from_context(context)
+        df1 = pandas.DataFrame({"A": [], "B": []})
+        df2 = pandas.DataFrame({"A": [], "C": [], "D": []})
+        x = client.create_composite(key="x")
+        x.write_dataframe(df1, key="table1")
+        with fail_with_status_code(HTTP_409_CONFLICT):
+            x.write_dataframe(df2, key="table2")
+
+
+def test_composite_two_tables_two_arrays(tree):
+    with Context.from_app(build_app(tree)) as context:
+        client = from_context(context)
+        df1 = pandas.DataFrame({"A": [], "B": []})
+        df2 = pandas.DataFrame({"C": [], "D": [], "E": []})
+        arr1 = numpy.ones((5, 5), dtype=numpy.float64)
+        arr2 = 2 * numpy.ones((5, 5), dtype=numpy.int8)
+        x = client.create_composite(key="x")
+
+        # Write by data source.
+        x.write_dataframe(df1, key="table1")
+        x.write_dataframe(df2, key="table2")
+        x.write_array(arr1, key="F")
+        x.write_array(arr2, key="G")
+
+        # Read by data source.
+        x.parts["table1"].read()
+        x.parts["table2"].read()
+        x.parts["F"].read()
+        x.parts["G"].read()
+
+        # Read by column.
+        for column in ["A", "B", "C", "D", "E", "F", "G"]:
+            x[column].read()
+
+
+def test_composite_table_column_array_key_collision(tree):
+    with Context.from_app(build_app(tree)) as context:
+        client = from_context(context)
+        df = pandas.DataFrame({"A": [], "B": []})
+        arr = numpy.array([1, 2, 3], dtype=numpy.float64)
+
+        x = client.create_composite(key="x")
+        x.write_dataframe(df, key="table1")
+        with fail_with_status_code(HTTP_409_CONFLICT):
+            x.write_array(arr, key="A")
+
+        y = client.create_composite(key="y")
+        y.write_array(arr, key="A")
+        with fail_with_status_code(HTTP_409_CONFLICT):
+            y.write_dataframe(df, key="table1")
diff --git a/tiled/adapters/parquet.py b/tiled/adapters/parquet.py
index 44603acc6..6c30bce4d 100644
--- a/tiled/adapters/parquet.py
+++ b/tiled/adapters/parquet.py
@@ -68,7 +68,9 @@ def dataframe_adapter(self) -> DataFrameAdapter:
             else:
                 partition = dask.dataframe.read_parquet(path)
             partitions.append(partition)
-        return DataFrameAdapter(partitions, self._structure)
+        return DataFrameAdapter(
+            partitions, self._structure, specs=self.specs, metadata=self.metadata()
+        )
 
     @classmethod
     def init_storage(
diff --git a/tiled/adapters/sequence.py b/tiled/adapters/sequence.py
index 4f2871e57..8875f1a45 100644
--- a/tiled/adapters/sequence.py
+++ b/tiled/adapters/sequence.py
@@ -195,7 +195,6 @@ def read(self, slice: Optional[NDSlice] = ...) -> NDArray[Any]:
                 arr = np.atleast_1d(arr[tuple(the_rest)])
         else:
             raise RuntimeError(f"Unsupported slice type, {type(slice)} in {slice}")
-
         sliced_shape = ndindex(slice).newshape(self.structure().shape)
         return force_reshape(arr, sliced_shape)
 
diff --git a/tiled/adapters/sql.py b/tiled/adapters/sql.py
index 159427b26..fac99f167 100644
--- a/tiled/adapters/sql.py
+++ b/tiled/adapters/sql.py
@@ -73,8 +73,8 @@ def close(self) -> None:
         self.conn.close()
 
     def metadata(self) -> JSON:
-        """
-        The metadata representing the actual data.
+        """The metadata representing the actual data.
+
         Returns
         -------
         The metadata representing the actual data.
@@ -195,46 +195,43 @@ def structure(self) -> TableStructure:
         return self._structure
 
     def get(self, key: str) -> Union[ArrayAdapter, None]:
-        """
-        Get the data for a specific key
+        """Get the data for a specific key
 
         Parameters
         ----------
         key : a string to indicate which column to be retrieved
+
         Returns
         -------
         The column for the associated key.
         """
         if key not in self.structure().columns:
             return None
-        return ArrayAdapter.from_array(self.read([key])[key].values)
+        return self[key]
 
     def __getitem__(self, key: str) -> ArrayAdapter:
-        """
-        Get the data for a specific key.
+        """Get the data for a specific key.
 
         Parameters
         ----------
         key : a string to indicate which column to be retrieved
+
         Returns
         -------
         The column for the associated key.
         """
+
         # Must compute to determine shape.
         return ArrayAdapter.from_array(self.read([key])[key].values)
 
     def items(self) -> Iterator[Tuple[str, ArrayAdapter]]:
-        """
-        The function to iterate over the SQLAdapter data.
+        """Iterate over the SQLAdapter data.
 
         Returns
         -------
         An iterator for the data in the associated database.
         """
-        yield from (
-            (key, ArrayAdapter.from_array(self.read([key])[key].values))
-            for key in self._structure.columns
-        )
+        yield from ((key, self[key]) for key in self._structure.columns)
 
     def append_partition(
         self,
diff --git a/tiled/catalog/adapter.py b/tiled/catalog/adapter.py
index 1286deab3..4f81c003f 100644
--- a/tiled/catalog/adapter.py
+++ b/tiled/catalog/adapter.py
@@ -452,7 +452,15 @@ async def lookup_adapter(
                         if adapter is None:
                             break
                     return adapter
+                elif (
+                    isinstance(catalog_adapter, CatalogCompositeAdapter)
+                    and len(segments[i:]) == 1
+                ):
+                    # Trying to access a table column or an array in a composite node
+                    _segm = await catalog_adapter.resolve_flat_key(segments[i])
+                    return await catalog_adapter.lookup_adapter(_segm.split("/"))
             return None
+
         return STRUCTURES[node.structure_family](self.context, node)
 
     async def get_adapter(self):
@@ -634,7 +642,10 @@ async def create_node(
             await db.refresh(node)
             for data_source in data_sources:
                 if data_source.management != Management.external:
-                    if structure_family == StructureFamily.container:
+                    if structure_family in {
+                        StructureFamily.container,
+                        StructureFamily.composite,
+                    }:
                         raise NotImplementedError(structure_family)
                     if data_source.mimetype is None:
                         data_source.mimetype = DEFAULT_CREATION_MIMETYPE[
@@ -1000,7 +1011,9 @@ async def items_range(self, offset, limit):
             return [
                 (
                     node.key,
-                    type(self)(self.context, node, access_policy=self.access_policy),
+                    STRUCTURES[node.structure_family](
+                        self.context, node, access_policy=self.access_policy
+                    ),
                 )
                 for node in nodes
             ]
@@ -1014,6 +1027,63 @@ async def read(self, *args, **kwargs):
         return await ensure_awaitable((await self.get_adapter()).read, *args, **kwargs)
 
 
+class CatalogCompositeAdapter(CatalogContainerAdapter):
+    async def resolve_flat_key(self, key):
+        for _key, item in await self.items_range(offset=0, limit=None):
+            if key == _key:
+                return _key
+            if item.structure_family == StructureFamily.table:
+                if key in item.structure().columns:
+                    return f"{_key}/{key}"
+        raise KeyError(key)
+
+    async def create_node(
+        self,
+        structure_family,
+        metadata,
+        key=None,
+        specs=None,
+        data_sources=None,
+    ):
+        key = key or self.context.key_maker()
+
+        # List all new keys that would be added to the flattened namespace
+        assert len(data_sources) == 1
+        if data_sources[0].structure_family == StructureFamily.table:
+            new_keys = data_sources[0].structure.columns
+        elif data_sources[0].structure_family in {
+            StructureFamily.array,
+            StructureFamily.awkward,
+            StructureFamily.sparse,
+        }:
+            new_keys = [key]
+        else:
+            raise ValueError(
+                f"Unsupported structure family: {data_sources[0].structure_family}"
+            )
+
+        # Get all keys and columns names already in the Composite node
+        flat_keys = []
+        for _key, item in await self.items_range(offset=0, limit=None):
+            flat_keys.append(_key)
+            if item.structure_family == StructureFamily.table:
+                flat_keys.extend(item.structure().columns)
+
+        # Check for column name collisions
+        key_conflicts = set(new_keys) & set(flat_keys)
+        if key_conflicts:
+            raise Collision(f"Column name collision: {key_conflicts}")
+
+        # Ensure that child tables are marked with the 'flattened' spec
+        if structure_family == StructureFamily.table:
+            if not specs or "flattened" not in (s.name for s in specs):
+                specs = [Spec(name="flattened")] + (specs or [])
+
+        return await super().create_node(
+            structure_family, metadata, key=key, specs=specs, data_sources=data_sources
+        )
+
+
 class CatalogArrayAdapter(CatalogNodeAdapter):
     async def read(self, *args, **kwargs):
         if not self.data_sources:
@@ -1500,6 +1570,7 @@ def specs_array_to_json(specs):
 STRUCTURES = {
     StructureFamily.array: CatalogArrayAdapter,
     StructureFamily.awkward: CatalogAwkwardAdapter,
+    StructureFamily.composite: CatalogCompositeAdapter,
     StructureFamily.container: CatalogContainerAdapter,
     StructureFamily.sparse: CatalogSparseAdapter,
     StructureFamily.table: CatalogTableAdapter,
diff --git a/tiled/catalog/core.py b/tiled/catalog/core.py
index a57c88a6f..80144531f 100644
--- a/tiled/catalog/core.py
+++ b/tiled/catalog/core.py
@@ -5,6 +5,7 @@
 
 # This is list of all valid revisions (from current to oldest).
 ALL_REVISIONS = [
+    "45a702586b2a",
     "ed3a4223a600",
     "e756b9381c14",
     "2ca16566d692",
diff --git a/tiled/catalog/migrations/versions/45a702586b2a_add_composite_structure_family.py b/tiled/catalog/migrations/versions/45a702586b2a_add_composite_structure_family.py
new file mode 100644
index 000000000..6b52cbf26
--- /dev/null
+++ b/tiled/catalog/migrations/versions/45a702586b2a_add_composite_structure_family.py
@@ -0,0 +1,35 @@
+"""Add 'composite' structure family to the structurefamily enum.
+
+Revision ID: 45a702586b2a
+Revises: ed3a4223a600
+Create Date: 2025-03-12 15:05:25.335010
+
+"""
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "45a702586b2a"
+down_revision = "ed3a4223a600"
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    connection = op.get_bind()
+
+    if connection.engine.dialect.name == "postgresql":
+        with op.get_context().autocommit_block():
+            op.execute(
+                sa.text(
+                    "ALTER TYPE structurefamily ADD VALUE IF NOT EXISTS 'composite' AFTER 'awkward'"
+                )
+            )
+
+
+def downgrade():
+    # PostgreSQL does not support dropping values from enums.
+    # (The enum would need to be destroyed and recreated, migrating
+    # the data in the process.) Fortunately, an extra unused values does
+    # not interfere with operation of older versions of Tiled.
+    pass
diff --git a/tiled/client/base.py b/tiled/client/base.py
index a45f87f70..53d863973 100644
--- a/tiled/client/base.py
+++ b/tiled/client/base.py
@@ -133,7 +133,7 @@ def __init__(
             # Allow the caller to optionally hand us a structure that is already
             # parsed from a dict into a structure dataclass.
             self._structure = structure
-        elif structure_family == StructureFamily.container:
+        elif structure_family in {StructureFamily.container, StructureFamily.composite}:
             self._structure = None
         else:
             structure_type = STRUCTURE_TYPES[attributes["structure_family"]]
@@ -202,7 +202,10 @@ def refresh(self):
             )
         ).json()
         self._item = content["data"]
-        if self.structure_family != StructureFamily.container:
+        if self.structure_family not in {
+            StructureFamily.container,
+            StructureFamily.composite,
+        }:
             structure_type = STRUCTURE_TYPES[self.structure_family]
             self._structure = structure_type.from_json(
                 self._item["attributes"]["structure"]
diff --git a/tiled/client/container.py b/tiled/client/container.py
index 2b39b8c8e..e93875faf 100644
--- a/tiled/client/container.py
+++ b/tiled/client/container.py
@@ -258,7 +258,8 @@ def __getitem__(self, keys, _ignore_inlined_contents=False):
             keys = (keys,)
         for key in keys:
             if not isinstance(key, str):
-                raise TypeError("Containers can only be indexed strings")
+                raise TypeError("Containers can only be indexed by strings")
+        keys = tuple("/".join(keys).strip("/").split("/"))  # Remove any slashes
         if self._queries:
             # Lookup this key *within the search results* of this Node.
             key, *tail = keys
@@ -287,7 +288,7 @@ def __getitem__(self, keys, _ignore_inlined_contents=False):
                 raise KeyError(key)
             assert (
                 len(data) == 1
-            ), "The key lookup query must never result more than one result."
+            ), "The key lookup query must never return more than one result."
             (item,) = data
             result = client_for_item(
                 self.context,
@@ -319,9 +320,7 @@ def __getitem__(self, keys, _ignore_inlined_contents=False):
                     # or because it was added after we fetched the inlined contents.
                     # Make a request for it.
                     try:
-                        self_link = self.item["links"]["self"]
-                        if self_link.endswith("/"):
-                            self_link = self_link[:-1]
+                        self_link = self.item["links"]["self"].rstrip("/")
                         params = {}
                         if self._include_data_sources:
                             params["include_data_sources"] = True
@@ -342,6 +341,17 @@ def __getitem__(self, keys, _ignore_inlined_contents=False):
                             raise KeyError(err_arg)
                         raise
                     item = content["data"]
+
+                    # Tables that belong to composite nodes cannot be addressed directly
+                    if (
+                        item["attributes"]["structure_family"] == StructureFamily.table
+                    ) and (
+                        "flattened" in (s["name"] for s in item["attributes"]["specs"])
+                    ):
+                        raise KeyError(
+                            f"Attempting to access a table in a composite container; use .parts['{keys[-1]}'] instead."  # noqa
+                        )
+
                     break
             result = client_for_item(
                 self.context,
@@ -648,7 +658,7 @@ def new(
             )
         ).json()
 
-        if structure_family == StructureFamily.container:
+        if structure_family in {StructureFamily.container, StructureFamily.composite}:
             structure = {"contents": None, "count": None}
         else:
             # Only containers can have multiple data_sources right now.
@@ -671,9 +681,10 @@ def new(
         # Ensure this is a dataclass, not a dict.
         # When we apply type hints and mypy to the client it should be possible
         # to dispense with this.
-        if (structure_family != StructureFamily.container) and isinstance(
-            structure, dict
-        ):
+        if (
+            structure_family
+            not in {StructureFamily.container, StructureFamily.composite}
+        ) and isinstance(structure, dict):
             structure_type = STRUCTURE_TYPES[structure_family]
             structure = structure_type.from_json(structure)
 
@@ -689,9 +700,31 @@ def new(
     # to attempt to avoid bumping into size limits.
     _SUGGESTED_MAX_UPLOAD_SIZE = 100_000_000  # 100 MB
 
-    def create_container(self, key=None, *, metadata=None, dims=None, specs=None):
+    def create_composite(self, key=None, *, metadata=None, specs=None):
+        """Create a new, empty composite container.
+
+        Parameters
+        ----------
+        key : str, optional
+            Key (name) for this new node. If None, the server will provide a unique key.
+        metadata : dict, optional
+            User metadata. May be nested. Must contain only basic types
+            (e.g. numbers, strings, lists, dicts) that are JSON-serializable.
+        specs : List[Spec], optional
+            List of names that are used to label that the data and/or metadata
+            conform to some named standard specification.
+
         """
-        EXPERIMENTAL: Create a new, empty container.
+        return self.new(
+            StructureFamily.composite,
+            [],
+            key=key,
+            metadata=metadata,
+            specs=specs,
+        )
+
+    def create_container(self, key=None, *, metadata=None, specs=None):
+        """Create a new, empty container.
 
         Parameters
         ----------
@@ -700,8 +733,6 @@ def create_container(self, key=None, *, metadata=None, dims=None, specs=None):
         metadata : dict, optional
             User metadata. May be nested. Must contain only basic types
             (e.g. numbers, strings, lists, dicts) that are JSON-serializable.
-        dims : List[str], optional
-            A label for each dimension of the array.
         specs : List[Spec], optional
             List of names that are used to label that the data and/or metadata
             conform to some named standard specification.
@@ -1051,6 +1082,125 @@ def write_dataframe(
         return client
 
 
+class Composite(Container):
+    def get_contents(self, maxlen=None, include_metadata=False):
+        result = {}
+        next_page_url = f"{self.item['links']['search']}"
+        while (next_page_url is not None) or (
+            maxlen is not None and len(result) < maxlen
+        ):
+            content = handle_error(
+                self.context.http_client.get(
+                    next_page_url,
+                    headers={"Accept": MSGPACK_MIME_TYPE},
+                    params={
+                        **parse_qs(urlparse(next_page_url).query),
+                        **self._queries_as_params,
+                    }
+                    | ({} if include_metadata else {"select_metadata": False}),
+                )
+            ).json()
+            result.update({item["id"]: item for item in content["data"]})
+
+            next_page_url = content["links"]["next"]
+
+        return result
+
+    @property
+    def _flat_keys_mapping(self):
+        result = {}
+        for key, item in self.get_contents().items():
+            if item["attributes"]["structure_family"] == StructureFamily.table:
+                for col in item["attributes"]["structure"]["columns"]:
+                    result[col] = item["id"] + "/" + col
+            else:
+                result[item["id"]] = item["id"]
+
+        self._cached_len = (len(result), time.monotonic() + LENGTH_CACHE_TTL)
+
+        return result
+
+    @property
+    def parts(self):
+        return CompositeParts(self)
+
+    def _keys_slice(self, start, stop, direction, _ignore_inlined_contents=False):
+        yield from self._flat_keys_mapping.keys()
+
+    def _items_slice(self, start, stop, direction, _ignore_inlined_contents=False):
+        for key in self._flat_keys_mapping.keys():
+            yield key, self[key]
+
+    def __len__(self):
+        if self._cached_len is not None:
+            length, deadline = self._cached_len
+            if time.monotonic() < deadline:
+                # Used the cached value and do not make any request.
+                return length
+
+        return len(self._flat_keys_mapping)
+
+    def __getitem__(self, key: str, _ignore_inlined_contents=False):
+        if isinstance(key, tuple):
+            key = "/".join(key)
+        if key in self._flat_keys_mapping:
+            key = self._flat_keys_mapping[key]
+        else:
+            raise KeyError(
+                f"Key '{key}' not found. If it refers to a table, use .parts['{key}'] instead."
+            )
+
+        return super().__getitem__(key, _ignore_inlined_contents)
+
+    def create_container(self, key=None, *, metadata=None, specs=None):
+        """Composite nodes can not include nested containers by design."""
+        raise NotImplementedError("Cannot create a container within a composite node.")
+
+    def create_composite(self, key=None, *, metadata=None, specs=None):
+        """Ccomposite nodes can not include nested composites by design."""
+        raise NotImplementedError("Cannot create a composite within a composite node.")
+
+
+class CompositeParts:
+    def __init__(self, node):
+        self.contents = node.get_contents(include_metadata=True)
+        self.context = node.context
+        self.structure_clients = node.structure_clients
+        self._include_data_sources = node._include_data_sources
+
+    def __repr__(self):
+        return (
+            f"<{type(self).__name__} {{"
+            + ", ".join(f"'{item}'" for item in self.contents)
+            + "}>"
+        )
+
+    def __getitem__(self, key):
+        key, *tail = key.split("/")
+
+        if key not in self.contents:
+            raise KeyError(key)
+
+        client = client_for_item(
+            self.context,
+            self.structure_clients,
+            self.contents[key],
+            include_data_sources=self._include_data_sources,
+        )
+
+        if tail:
+            return client["/".join(tail)]
+        else:
+            return client
+
+    def __iter__(self):
+        for key in self.contents:
+            yield key
+
+    def __len__(self) -> int:
+        return len(self.contents)
+
+
 def _queries_to_params(*queries):
     "Compute GET params from the queries."
     params = collections.defaultdict(list)
@@ -1115,6 +1265,7 @@ def _write_partition(x, partition_info, client):
     "numpy": OneShotCachedMap(
         {
             "container": _Wrap(Container),
+            "composite": _Wrap(Composite),
             "array": _LazyLoad(("..array", Container.__module__), "ArrayClient"),
             "awkward": _LazyLoad(("..awkward", Container.__module__), "AwkwardClient"),
             "dataframe": _LazyLoad(
@@ -1132,6 +1283,7 @@ def _write_partition(x, partition_info, client):
     "dask": OneShotCachedMap(
         {
             "container": _Wrap(Container),
+            "composite": _Wrap(Composite),
             "array": _LazyLoad(("..array", Container.__module__), "DaskArrayClient"),
             # TODO Create DaskAwkwardClient
             # "awkward": _LazyLoad(("..awkward", Container.__module__), "DaskAwkwardClient"),
diff --git a/tiled/client/smoke.py b/tiled/client/smoke.py
index 8454fd1d5..2362d6f3d 100644
--- a/tiled/client/smoke.py
+++ b/tiled/client/smoke.py
@@ -24,7 +24,7 @@ def read(node, verbose=False, strict=False):
     """
 
     faulty_entries = []
-    if node.structure_family == StructureFamily.container:
+    if node.structure_family in {StructureFamily.container, StructureFamily.composite}:
         for key, child_node in node.items():
             fault_result = read(child_node, verbose=verbose, strict=strict)
             faulty_entries.extend(fault_result)
diff --git a/tiled/client/sync.py b/tiled/client/sync.py
index b595dd2a1..34fa6ee4c 100644
--- a/tiled/client/sync.py
+++ b/tiled/client/sync.py
@@ -90,7 +90,10 @@ def _copy_container(source, dest, on_conflict):
         if num_data_sources == 0:
             # A container with no data sources is just an organizational
             # entity in the database.
-            if child_node.structure_family == StructureFamily.container:
+            if child_node.structure_family in {
+                StructureFamily.container,
+                StructureFamily.composite,
+            }:
                 data_sources = []
             else:
                 raise ValueError(
@@ -102,7 +105,10 @@ def _copy_container(source, dest, on_conflict):
             if original_data_source.management == Management.external:
                 data_sources = [original_data_source]
             else:
-                if child_node.structure_family == StructureFamily.container:
+                if child_node.structure_family in {
+                    StructureFamily.container,
+                    StructureFamily.composite,
+                }:
                     data_sources = []
                 else:
                     data_sources = [
@@ -140,7 +146,8 @@ def _copy_container(source, dest, on_conflict):
             original_data_sources
             and (original_data_sources[0].management != Management.external)
         ) or (
-            child_node.structure_family == StructureFamily.container
+            child_node.structure_family
+            in [StructureFamily.container, StructureFamily.composite]
             and (not original_data_sources)
         ):
             _DISPATCH[child_node.structure_family](
@@ -152,6 +159,7 @@ def _copy_container(source, dest, on_conflict):
     StructureFamily.array: _copy_array,
     StructureFamily.awkward: _copy_awkward,
     StructureFamily.container: _copy_container,
+    StructureFamily.composite: _copy_container,
     StructureFamily.sparse: _copy_sparse,
     StructureFamily.table: _copy_table,
 }
diff --git a/tiled/serialization/container.py b/tiled/serialization/container.py
index 5b82074c7..694b60363 100644
--- a/tiled/serialization/container.py
+++ b/tiled/serialization/container.py
@@ -30,6 +30,9 @@ async def walk(node, filter_for_access, pre=None):
             ).items_range(0, None):
                 async for d in walk(value, filter_for_access, pre + [key]):
                     yield d
+        elif node.structure_family == StructureFamily.table:
+            for key in node.structure().columns:
+                yield (pre + [key], await filter_for_access(node, path_parts=[key]))
         else:
             for key, value in (await filter_for_access(node, path_parts=[])).items():
                 async for d in walk(value, filter_for_access, pre + [key]):
@@ -81,6 +84,9 @@ async def serialize_hdf5(node, metadata, filter_for_access):
     serialization_registry.register(
         StructureFamily.container, "application/x-hdf5", serialize_hdf5
     )
+    serialization_registry.register(
+        StructureFamily.composite, "application/x-hdf5", serialize_hdf5
+    )
 
 if modules_available("orjson"):
 
@@ -104,3 +110,6 @@ async def serialize_json(node, metadata, filter_for_access):
     serialization_registry.register(
         StructureFamily.container, "application/json", serialize_json
     )
+    serialization_registry.register(
+        StructureFamily.composite, "application/json", serialize_json
+    )
diff --git a/tiled/server/core.py b/tiled/server/core.py
index 65be00c5a..214b81420 100644
--- a/tiled/server/core.py
+++ b/tiled/server/core.py
@@ -260,6 +260,7 @@ async def construct_entries_response(
     StructureFamily.awkward: {"*/*": "application/zip"},
     StructureFamily.table: {"*/*": APACHE_ARROW_FILE_MIME_TYPE},
     StructureFamily.container: {"*/*": "application/x-hdf5"},
+    StructureFamily.composite: {"*/*": "application/x-hdf5"},
     StructureFamily.sparse: {"*/*": APACHE_ARROW_FILE_MIME_TYPE},
 }
 
@@ -431,8 +432,11 @@ async def construct_resource(
             # for ease of going between dataclass and pydantic.
             specs.append(schemas.Spec(**spec.model_dump()))
         attributes["specs"] = specs
-    if (entry is not None) and entry.structure_family == StructureFamily.container:
-        attributes["structure_family"] = StructureFamily.container
+    if (entry is not None) and entry.structure_family in {
+        StructureFamily.container,
+        StructureFamily.composite,
+    }:
+        attributes["structure_family"] = entry.structure_family
 
         if schemas.EntryFields.structure in fields:
             if (
diff --git a/tiled/server/links.py b/tiled/server/links.py
index 76bf2616f..2120c3741 100644
--- a/tiled/server/links.py
+++ b/tiled/server/links.py
@@ -48,6 +48,7 @@ def links_for_table(structure_family, structure, base_url, path_str):
     StructureFamily.array: links_for_array,
     StructureFamily.awkward: links_for_awkward,
     StructureFamily.container: links_for_container,
+    StructureFamily.composite: links_for_container,
     StructureFamily.sparse: links_for_array,  # spare and array are the same
     StructureFamily.table: links_for_table,
 }
diff --git a/tiled/server/router.py b/tiled/server/router.py
index d5f613920..87b891589 100644
--- a/tiled/server/router.py
+++ b/tiled/server/router.py
@@ -191,7 +191,10 @@ async def search(
     **filters,
 ):
     request.state.endpoint = "search"
-    if entry.structure_family != StructureFamily.container:
+    if entry.structure_family not in {
+        StructureFamily.container,
+        StructureFamily.composite,
+    }:
         raise WrongTypeForRoute("This is not a Node; it cannot be searched or listed.")
     try:
         resource, metadata_stale_at, must_revalidate = await construct_entries_response(
@@ -760,7 +763,8 @@ async def table_full(
 async def get_container_full(
     request: Request,
     entry=SecureEntry(
-        scopes=["read:data"], structure_families={StructureFamily.container}
+        scopes=["read:data"],
+        structure_families={StructureFamily.container, StructureFamily.composite},
     ),
     principal: str = Depends(get_current_principal),
     field: Optional[List[str]] = Query(None, min_length=1),
@@ -790,7 +794,8 @@ async def get_container_full(
 async def post_container_full(
     request: Request,
     entry=SecureEntry(
-        scopes=["read:data"], structure_families={StructureFamily.container}
+        scopes=["read:data"],
+        structure_families={StructureFamily.container, StructureFamily.composite},
     ),
     principal: str = Depends(get_current_principal),
     field: Optional[List[str]] = Body(None, min_length=1),
@@ -867,7 +872,11 @@ async def node_full(
     request: Request,
     entry=SecureEntry(
         scopes=["read:data"],
-        structure_families={StructureFamily.table, StructureFamily.container},
+        structure_families={
+            StructureFamily.table,
+            StructureFamily.container,
+            StructureFamily.composite,
+        },
     ),
     principal: str = Depends(get_current_principal),
     field: Optional[List[str]] = Query(None, min_length=1),
@@ -898,7 +907,7 @@ async def node_full(
                 "request a smaller chunks."
             ),
         )
-    if entry.structure_family == StructureFamily.container:
+    if entry.structure_family in {StructureFamily.container, StructureFamily.composite}:
         curried_filter = partial(
             filter_for_access,
             principal=principal,
@@ -1160,7 +1169,7 @@ async def _create_node(
         body.structure_family,
         body.specs,
     )
-    if structure_family == StructureFamily.container:
+    if structure_family in {StructureFamily.container, StructureFamily.composite}:
         structure = None
     else:
         if len(body.data_sources) != 1:
diff --git a/tiled/server/schemas.py b/tiled/server/schemas.py
index 4bc4ede2a..cf4100383 100644
--- a/tiled/server/schemas.py
+++ b/tiled/server/schemas.py
@@ -232,6 +232,7 @@ class SparseLinks(pydantic.BaseModel):
 resource_links_type_by_structure_family = {
     StructureFamily.array: ArrayLinks,
     StructureFamily.awkward: AwkwardLinks,
+    StructureFamily.composite: ContainerLinks,
     StructureFamily.container: ContainerLinks,
     StructureFamily.sparse: SparseLinks,
     StructureFamily.table: DataFrameLinks,
@@ -422,7 +423,10 @@ def specs_uniqueness_validator(cls, v):
     def narrow_strucutre_type(self):
         "Convert the structure on each data_source from a dict to the appropriate pydantic model."
         for data_source in self.data_sources:
-            if self.structure_family != StructureFamily.container:
+            if self.structure_family not in {
+                StructureFamily.container,
+                StructureFamily.composite,
+            }:
                 structure_cls = STRUCTURE_TYPES[self.structure_family]
                 if data_source.structure is not None:
                     data_source.structure = structure_cls.from_json(
diff --git a/tiled/structures/core.py b/tiled/structures/core.py
index 309ada6e8..e812a5e17 100644
--- a/tiled/structures/core.py
+++ b/tiled/structures/core.py
@@ -15,6 +15,7 @@
 class StructureFamily(str, enum.Enum):
     array = "array"
     awkward = "awkward"
+    composite = "composite"
     container = "container"
     sparse = "sparse"
     table = "table"


Reviews:
user: login: genematx
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-28T15:13:15Z
user: login: danielballan
body: Awesome, nice to see this converging.

I'd like to test-drive it a bit, which I haven't yet done, but I gave a close read.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-11T15:30:43Z
user: login: genematx
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T15:37:40Z
user: login: genematx
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T16:22:36Z
user: login: genematx
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T16:32:25Z
user: login: genematx
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T16:32:38Z
user: login: genematx
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T16:35:39Z
user: login: danielballan
body: I took one more detailed read through. 
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-13T17:00:45Z
user: login: genematx
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T17:26:11Z
user: login: genematx
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T17:26:28Z
user: login: genematx
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T17:27:17Z
user: login: genematx
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T17:41:40Z
user: login: genematx
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-13T17:42:23Z

Review comments:
diff_hunk: @@ -1051,6 +1050,57 @@ def write_dataframe(
         return client
 
 
+class Composite(Container):
+
+    @property
path: tiled/client/container.py
user: login: genematx
body: possibly cached?
created_at: 2025-02-28T15:13:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -689,9 +688,8 @@ def new(
     # to attempt to avoid bumping into size limits.
     _SUGGESTED_MAX_UPLOAD_SIZE = 100_000_000  # 100 MB
 
-    def create_container(self, key=None, *, metadata=None, dims=None, specs=None):
-        """
-        EXPERIMENTAL: Create a new, empty container.
+    def create_container(self, key=None, *, metadata=None, specs=None, flat=False):
path: tiled/client/container.py
user: login: danielballan
body: I lean toward, "This should be a new method, `create_composite`." We can always add this `flat=True` alias later. What do you think?
created_at: 2025-03-11T15:20:09Z
author_association: MEMBER
diff_hunk: @@ -634,7 +634,10 @@ async def create_node(
             await db.refresh(node)
             for data_source in data_sources:
                 if data_source.management != Management.external:
-                    if structure_family == StructureFamily.container:
+                    if structure_family in [
path: tiled/catalog/adapter.py
user: login: danielballan
body: Here and elsewhere below, it may be slightly better to make this a `set`. Negligible performance improvement, but a bit cleaner IMO.
created_at: 2025-03-11T15:22:54Z
author_association: MEMBER
diff_hunk: @@ -16,6 +16,7 @@ class StructureFamily(str, enum.Enum):
     array = "array"
     awkward = "awkward"
     container = "container"
+    composite = "composite"
path: tiled/structures/core.py
user: login: danielballan
body: Bump this up one row to maintain alphabetical sort order.
created_at: 2025-03-11T15:26:37Z
author_association: MEMBER
diff_hunk: @@ -1051,6 +1049,106 @@ def write_dataframe(
         return client
 
 
+class Composite(Container):
path: tiled/client/container.py
user: login: danielballan
body: This should override `create_container` (and also `create_composite` if we add it) to raise an exception. The server would reject the request (unsupported structure family) but it's probably worth catching it client side and making it clear in the docstring that this never works.
created_at: 2025-03-11T15:29:46Z
author_association: MEMBER
diff_hunk: @@ -689,9 +688,8 @@ def new(
     # to attempt to avoid bumping into size limits.
     _SUGGESTED_MAX_UPLOAD_SIZE = 100_000_000  # 100 MB
 
-    def create_container(self, key=None, *, metadata=None, dims=None, specs=None):
-        """
-        EXPERIMENTAL: Create a new, empty container.
+    def create_container(self, key=None, *, metadata=None, specs=None, flat=False):
path: tiled/client/container.py
user: login: genematx
body: I wasn't sure. Thanks! Will fix it.
created_at: 2025-03-11T15:37:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -16,6 +16,7 @@ class StructureFamily(str, enum.Enum):
     array = "array"
     awkward = "awkward"
     container = "container"
+    composite = "composite"
path: tiled/structures/core.py
user: login: genematx
body: Done
created_at: 2025-03-11T16:22:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1051,6 +1049,106 @@ def write_dataframe(
         return client
 
 
+class Composite(Container):
path: tiled/client/container.py
user: login: genematx
body: Done
created_at: 2025-03-11T16:32:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -689,9 +688,8 @@ def new(
     # to attempt to avoid bumping into size limits.
     _SUGGESTED_MAX_UPLOAD_SIZE = 100_000_000  # 100 MB
 
-    def create_container(self, key=None, *, metadata=None, dims=None, specs=None):
-        """
-        EXPERIMENTAL: Create a new, empty container.
+    def create_container(self, key=None, *, metadata=None, specs=None, flat=False):
path: tiled/client/container.py
user: login: genematx
body: Done
created_at: 2025-03-11T16:32:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -634,7 +634,10 @@ async def create_node(
             await db.refresh(node)
             for data_source in data_sources:
                 if data_source.management != Management.external:
-                    if structure_family == StructureFamily.container:
+                    if structure_family in [
path: tiled/catalog/adapter.py
user: login: genematx
body: Done!
created_at: 2025-03-11T16:35:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -195,46 +195,55 @@ def structure(self) -> TableStructure:
         return self._structure
 
     def get(self, key: str) -> Union[ArrayAdapter, None]:
-        """
-        Get the data for a specific key
+        """Get the data for a specific key
 
         Parameters
         ----------
         key : a string to indicate which column to be retrieved
+
         Returns
         -------
         The column for the associated key.
         """
         if key not in self.structure().columns:
             return None
-        return ArrayAdapter.from_array(self.read([key])[key].values)
+        return self[key]
 
     def __getitem__(self, key: str) -> ArrayAdapter:
-        """
-        Get the data for a specific key.
+        """Get the data for a specific key.
 
         Parameters
         ----------
         key : a string to indicate which column to be retrieved
+
         Returns
         -------
         The column for the associated key.
         """
+
+        # Construct the metadata for the array: assume that relevant metadata is in the `key` field
path: tiled/adapters/sql.py
user: login: danielballan
body: I feel unsure about this. This imposes or at least suggests a kind of schema on the `metadata`. Up to now we have treat `metadata` as opaque user space.

Arrow semantics does not have any concept of per-column `metadata`; it only has per-table `metadata`.

One option is to leave this `metadata` _empty_: we can always adds keys in a future PR. Another option is to propagate the full `metadata` of the table onto this node.
created_at: 2025-03-13T16:53:58Z
author_association: MEMBER
diff_hunk: @@ -145,7 +145,19 @@ def __getitem__(self, key: str) -> ArrayAdapter:
             max_size = max((len(i) for i in array.ravel()))
             array = array.astype(dtype=numpy.dtype(f"<U{max_size}"))
 
-        return ArrayAdapter.from_array(array)
+        # Construct the metadata for the array: assume that relevant metadata is in the `key` field
path: tiled/adapters/table.py
user: login: danielballan
body: (Comment above applies here.)
created_at: 2025-03-13T16:54:16Z
author_association: MEMBER
diff_hunk: @@ -1503,4 +1573,5 @@ def specs_array_to_json(specs):
     StructureFamily.container: CatalogContainerAdapter,
     StructureFamily.sparse: CatalogSparseAdapter,
     StructureFamily.table: CatalogTableAdapter,
+    StructureFamily.composite: CatalogCompositeAdapter,
path: tiled/catalog/adapter.py
user: login: danielballan
body: Nitpick: sort order
created_at: 2025-03-13T16:56:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,31 @@
+"""Add 'composite' structure family to the structurefamily enum.
+
+Revision ID: 45a702586b2a
+Revises: ed3a4223a600
+Create Date: 2025-03-12 15:05:25.335010
+
+"""
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "45a702586b2a"
+down_revision = "ed3a4223a600"
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    connection = op.get_bind()
+
+    if connection.engine.dialect.name == "postgresql":
+        with op.get_context().autocommit_block():
+            op.execute(
+                sa.text(
+                    "ALTER TYPE structurefamily ADD VALUE IF NOT EXISTS 'composite' AFTER 'awkward'"
+                )
+            )
+
+
+def downgrade():
+    raise NotImplementedError
path: tiled/catalog/migrations/versions/45a702586b2a_add_composite_structure_family.py
user: login: danielballan
body: As of the beta releases, we have started implementing a `downgrade` path. I gather it shouldn't be too hard to do in this case?
created_at: 2025-03-13T16:58:04Z
author_association: MEMBER
diff_hunk: @@ -342,6 +341,17 @@ def __getitem__(self, keys, _ignore_inlined_contents=False):
                             raise KeyError(err_arg)
                         raise
                     item = content["data"]
+
+                    # Tables that belong to composite nodes can not be addressed directly
path: tiled/client/container.py
user: login: danielballan
body: ```suggestion
                    # Tables that belong to composite nodes cannot be addressed directly
```
created_at: 2025-03-13T16:58:29Z
author_association: MEMBER
diff_hunk: @@ -1051,6 +1082,125 @@ def write_dataframe(
         return client
 
 
+class Composite(Container):
+    def _get_contents(self, maxlen=None, include_metadata=False):
+        result = {}
+        next_page_url = f"{self.item['links']['search']}"
+        while (next_page_url is not None) or (
+            maxlen is not None and len(result) < maxlen
+        ):
+            content = handle_error(
+                self.context.http_client.get(
+                    next_page_url,
+                    headers={"Accept": MSGPACK_MIME_TYPE},
+                    params={
+                        **parse_qs(urlparse(next_page_url).query),
+                        **self._queries_as_params,
+                    }
+                    | ({} if include_metadata else {"select_metadata": False}),
+                )
+            ).json()
+            result.update({item["id"]: item for item in content["data"]})
+
+            next_page_url = content["links"]["next"]
+
+        return result
+
+    @property
+    def _flat_keys_mapping(self):
+        result = {}
+        for key, item in self._get_contents().items():
+            if item["attributes"]["structure_family"] == StructureFamily.table:
+                for col in item["attributes"]["structure"]["columns"]:
+                    result[col] = item["id"] + "/" + col
+            else:
+                result[item["id"]] = item["id"]
+
+        self._cached_len = (len(result), time.monotonic() + LENGTH_CACHE_TTL)
+
+        return result
+
+    @property
+    def parts(self):
+        return CompositeContents(self)
+
+    def _keys_slice(self, start, stop, direction, _ignore_inlined_contents=False):
+        yield from self._flat_keys_mapping.keys()
+
+    def _items_slice(self, start, stop, direction, _ignore_inlined_contents=False):
+        for key in self._flat_keys_mapping.keys():
+            yield key, self[key]
+
+    def __len__(self):
+        if self._cached_len is not None:
+            length, deadline = self._cached_len
+            if time.monotonic() < deadline:
+                # Used the cached value and do not make any request.
+                return length
+
+        return len(self._flat_keys_mapping)
+
+    def __getitem__(self, key: str, _ignore_inlined_contents=False):
+        if isinstance(key, tuple):
+            key = "/".join(key)
+        if key in self._flat_keys_mapping:
+            key = self._flat_keys_mapping[key]
+        else:
+            raise KeyError(
+                f"Key '{key}' not found. If it refers to a table, use .parts['{key}'] instead."
+            )
+
+        return super().__getitem__(key, _ignore_inlined_contents)
+
+    def create_container(self, key=None, *, metadata=None, specs=None):
+        """Composite nodes can not include nested containers by design."""
+        raise NotImplementedError("Cannot create a container within a composite node.")
+
+    def create_composite(self, key=None, *, metadata=None, specs=None):
+        """Ccomposite nodes can not include nested composites by design."""
+        raise NotImplementedError("Cannot create a composite within a composite node.")
+
+
+class CompositeContents:
path: tiled/client/container.py
user: login: danielballan
body: Should this be named `CompositeParts` for consistency?

```py
>>> x.parts
<CompositeParts ...>
```

The word `Contents` is used, as in `self._contents` to mean something slightly distinct, I think.

I wouldn't insist on it, though; if you prefer it as is, I'm good. :+1:
created_at: 2025-03-13T17:00:08Z
author_association: MEMBER
diff_hunk: @@ -195,46 +195,55 @@ def structure(self) -> TableStructure:
         return self._structure
 
     def get(self, key: str) -> Union[ArrayAdapter, None]:
-        """
-        Get the data for a specific key
+        """Get the data for a specific key
 
         Parameters
         ----------
         key : a string to indicate which column to be retrieved
+
         Returns
         -------
         The column for the associated key.
         """
         if key not in self.structure().columns:
             return None
-        return ArrayAdapter.from_array(self.read([key])[key].values)
+        return self[key]
 
     def __getitem__(self, key: str) -> ArrayAdapter:
-        """
-        Get the data for a specific key.
+        """Get the data for a specific key.
 
         Parameters
         ----------
         key : a string to indicate which column to be retrieved
+
         Returns
         -------
         The column for the associated key.
         """
+
+        # Construct the metadata for the array: assume that relevant metadata is in the `key` field
path: tiled/adapters/sql.py
user: login: genematx
body: Agree. Leaving the column metadata empty and potentially offloading this to specs and structure-clients if needed in the future.
created_at: 2025-03-13T17:26:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -145,7 +145,19 @@ def __getitem__(self, key: str) -> ArrayAdapter:
             max_size = max((len(i) for i in array.ravel()))
             array = array.astype(dtype=numpy.dtype(f"<U{max_size}"))
 
-        return ArrayAdapter.from_array(array)
+        # Construct the metadata for the array: assume that relevant metadata is in the `key` field
path: tiled/adapters/table.py
user: login: genematx
body: Fixed
created_at: 2025-03-13T17:26:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1503,4 +1573,5 @@ def specs_array_to_json(specs):
     StructureFamily.container: CatalogContainerAdapter,
     StructureFamily.sparse: CatalogSparseAdapter,
     StructureFamily.table: CatalogTableAdapter,
+    StructureFamily.composite: CatalogCompositeAdapter,
path: tiled/catalog/adapter.py
user: login: genematx
body: Done.
created_at: 2025-03-13T17:27:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1051,6 +1082,125 @@ def write_dataframe(
         return client
 
 
+class Composite(Container):
+    def _get_contents(self, maxlen=None, include_metadata=False):
+        result = {}
+        next_page_url = f"{self.item['links']['search']}"
+        while (next_page_url is not None) or (
+            maxlen is not None and len(result) < maxlen
+        ):
+            content = handle_error(
+                self.context.http_client.get(
+                    next_page_url,
+                    headers={"Accept": MSGPACK_MIME_TYPE},
+                    params={
+                        **parse_qs(urlparse(next_page_url).query),
+                        **self._queries_as_params,
+                    }
+                    | ({} if include_metadata else {"select_metadata": False}),
+                )
+            ).json()
+            result.update({item["id"]: item for item in content["data"]})
+
+            next_page_url = content["links"]["next"]
+
+        return result
+
+    @property
+    def _flat_keys_mapping(self):
+        result = {}
+        for key, item in self._get_contents().items():
+            if item["attributes"]["structure_family"] == StructureFamily.table:
+                for col in item["attributes"]["structure"]["columns"]:
+                    result[col] = item["id"] + "/" + col
+            else:
+                result[item["id"]] = item["id"]
+
+        self._cached_len = (len(result), time.monotonic() + LENGTH_CACHE_TTL)
+
+        return result
+
+    @property
+    def parts(self):
+        return CompositeContents(self)
+
+    def _keys_slice(self, start, stop, direction, _ignore_inlined_contents=False):
+        yield from self._flat_keys_mapping.keys()
+
+    def _items_slice(self, start, stop, direction, _ignore_inlined_contents=False):
+        for key in self._flat_keys_mapping.keys():
+            yield key, self[key]
+
+    def __len__(self):
+        if self._cached_len is not None:
+            length, deadline = self._cached_len
+            if time.monotonic() < deadline:
+                # Used the cached value and do not make any request.
+                return length
+
+        return len(self._flat_keys_mapping)
+
+    def __getitem__(self, key: str, _ignore_inlined_contents=False):
+        if isinstance(key, tuple):
+            key = "/".join(key)
+        if key in self._flat_keys_mapping:
+            key = self._flat_keys_mapping[key]
+        else:
+            raise KeyError(
+                f"Key '{key}' not found. If it refers to a table, use .parts['{key}'] instead."
+            )
+
+        return super().__getitem__(key, _ignore_inlined_contents)
+
+    def create_container(self, key=None, *, metadata=None, specs=None):
+        """Composite nodes can not include nested containers by design."""
+        raise NotImplementedError("Cannot create a container within a composite node.")
+
+    def create_composite(self, key=None, *, metadata=None, specs=None):
+        """Ccomposite nodes can not include nested composites by design."""
+        raise NotImplementedError("Cannot create a composite within a composite node.")
+
+
+class CompositeContents:
path: tiled/client/container.py
user: login: genematx
body: Totally, `CompositeParts` is much better. Changed!
Also, made `self.contents` and `self.get_contents` public attribute/method.
created_at: 2025-03-13T17:41:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,31 @@
+"""Add 'composite' structure family to the structurefamily enum.
+
+Revision ID: 45a702586b2a
+Revises: ed3a4223a600
+Create Date: 2025-03-12 15:05:25.335010
+
+"""
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "45a702586b2a"
+down_revision = "ed3a4223a600"
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+    connection = op.get_bind()
+
+    if connection.engine.dialect.name == "postgresql":
+        with op.get_context().autocommit_block():
+            op.execute(
+                sa.text(
+                    "ALTER TYPE structurefamily ADD VALUE IF NOT EXISTS 'composite' AFTER 'awkward'"
+                )
+            )
+
+
+def downgrade():
+    raise NotImplementedError
path: tiled/catalog/migrations/versions/45a702586b2a_add_composite_structure_family.py
user: login: genematx
body: Oh, nice! Added.
Please, doublecheck, @danielballan 
created_at: 2025-03-13T17:42:23Z
author_association: CONTRIBUTOR

PR comments:
user: login: danielballan
created_at: 2025-03-12T16:18:32Z
body: This will also need a database migration to extend the `StructureFamilies` enum in PostgreSQL. This command will create a new stub file with a random name.

```
python -m tiled.catalog revision -m "Add composite structure family"
```

Add content with reference to this example: https://github.com/bluesky/tiled/blob/main/tiled/catalog/migrations/versions/0b033e7fbe30_add_awkward_to_structurefamily_enum.py

Then update the list of revisions in `tiled.catalog.core.py`.
author_association: MEMBER
user: login: danielballan
created_at: 2025-03-13T01:42:57Z
body: Here is @genematx's test script from our interactive session today, for Future Us:

```py
import pandas as pd
import numpy as np
df = pd.DataFrame({"colA": np.random.randn(10), "colB": np.random.randint(0, 10, 10), "colC": np.random.choice(["a", "b", "c", "d", "e"], 10)})

Y = c.create_composite("test", metadata={"attrs": {"b": 2}})

Y.write_array(np.random.randn(10, ), key="arr1", metadata={"attrs": {"c": 3}})
Y.write_array(np.random.randn(10, ), key="arr2", metadata={"attrs": {"d": 4}})
Y.write_array(np.random.randn(20, ), key="arr3", metadata={"attrs": {"e": 5}})
Y.write_dataframe(df, key="tab1", metadata={"attrs": {"f": 6}})
```

---

I like how this turned out:

```py
In [35]: c['test']
Out[35]: <Composite {'arr1', 'arr2', 'arr3', 'colA', 'colB', 'colC'}>

In [36]: c['test']['tab1']
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[36], line 1
----> 1 c['test']['tab1']

File ~/Repos/bnl/tiled/tiled/client/container.py:1149, in Composite.__getitem__(self, key, _ignore_inlined_contents)
   1147     key = self._flat_keys_mapping[key]
   1148 else:
-> 1149     raise KeyError(
   1150         f"Key '{key}' not found. If it refers to a table, use .parts['{key}'] instead."
   1151     )
   1153 return super().__getitem__(key, _ignore_inlined_contents)

KeyError: "Key 'tab1' not found. If it refers to a table, use .parts['tab1'] instead."

In [37]: c['test'].parts['tab1']
Out[37]: <DataFrameClient ['colA', 'colB', 'colC']>
```

Perhaps we should make that error message even more generous by checking whether `'tab1'` is in fact in `parts`.

---

Soon we will need `c['test'].read()` (which I guess returns an `xarray.Dataset`) but since Composite is usable without this, I would be in favor making that a separate follow-up PR.

---

I haven't done line review yet, but conceptually I think this has landed.
author_association: MEMBER
user: login: genematx
created_at: 2025-03-13T14:11:23Z
body: > Perhaps we should make that error message even more generous by checking whether 'tab1' is in fact in parts.

I was thinking about this too, @danielballan. I'm afraid this would force us to make another API call (since we are not currently caching `.parts` and don't know what's in there). Do you think it would be worth caching the `contents` (similarly to how we do this with `__len__`)? Or I can use some heuristic, e.g. `_flat_keys_mapping = {'tab1': None, 'colA': 'tab1/colA'}`, so if `tab1` is not in Keys -- it's definitely not existing, and if the value is None, it's just not addressable.
author_association: CONTRIBUTOR
user: login: danielballan
created_at: 2025-03-13T15:00:35Z
body: I can think of other counterarguments, too. Like, maybe it's not in `parts` _yet_ but it will be a moment later. And the point you raise about caching would make that even more complicated.

Let's leave it as is, at least for now.
author_association: MEMBER