Repository: pydap/pydap

Issue:
add createDapType method to DatasetType 
<!-- Thank you very much for your input! -->

### Description

### Minimal example
Consider the case of creating a Dataset with the following tree hierarchy
```
.example dataset
├──SimpleGroup
│  ├──Temperature
│  ├──Salinity
│  ├──Y
│  └──X
├──time
└──time_bnds
```

To create this array with pydap's model types I do:

```python
RootGroup = DatasetType("example dataset", dimensions={"time":1, "nv":2})
RootGroup["SimpleGroup"] = GroupType("SimpleGroup", dimensions = {"Y": 4, "X": 4})
RootGroup["/SimpleGroup/Temperature"] = BaseType("Temperature", np.arange(10, 26, 1, dtype="f4").reshape(1, 4, 4), dims=("/time", "/SimpleGroup/Y", "/SimpleGroup/X"))
RootGroup["/SimpleGroup/Salinity"] = BaseType("Salinity", 30 * np.ones(16, dtype="f4").reshape(1, 4, 4), dims=("/time", "/SimpleGroup/Y", "/SimpleGroup/X"))
RootGroup["/SimpleGroup/Y"] = BaseType("Y", np.arange(4, dtype="i2"), dims=("/SimpleGroup/Y",))
RootGroup["/SimpleGroup/X"] = BaseType("X", np.arange(4, dtype="i2"), dims=("/SimpleGroup/X",))
RootGroup["/time"] = BaseType("time", np.array(0.5, dtype="f4"),dims=("/time",))
RootGroup["/time_bnds"] = BaseType("time_bnds", np.arange(2, dtype="f4"), dims=("/time", "/nv"))
```
And so the`pydap` user needs to know the `GroupType, BaseType, StructureType, SequenceType` types. Moreover, note the use of `Fully qualifying names` on the left hand side, which is different from the `name` used in`BaseType`, `GroupType` on the right hand side. Trying to use a `FQN` in `BaseType` or `GroupType` when these are NOT at the root level raises a KeyError.

### What I propose adding
```python
.createGroup(**args)
.createVariable(**args)
.createSequence(**args)
.createStructure(**args)
```
where args define a`name= FullyQualifyingName`, and relevant necessary attributes such as dimensions. 

With these additions, one can generate the dataset as follows:

```python
RootGroup = DatasetType("example dataset", dimensions={"time":1, "nv":2})
RootGroup.createGroup("SimpleGroup", dimensions = {"Y": 4, "X": 4})
RootGroup.createVariable("/SimpleGroup/Temperature", np.arange(10, 26, 1, dtype="f4").reshape(1, 4, 4), dims=("/time", "/SimpleGroup/Y", "/SimpleGroup/X"))
RootGroup.createVariable("/SimpleGroup/Salinity", 30 * np.ones(16, dtype="f4").reshape(1, 4, 4), dims=("/time", "/SimpleGroup/Y", "/SimpleGroup/X"))
RootGroup.createVariable("/SimpleGroup/Y", np.arange(4, dtype="i2"), dims=("/SimpleGroup/Y",))
RootGroup.createVariable("/SimpleGroup/X", np.arange(4, dtype="i2"), dims=("/SimpleGroup/X",))
RootGroup.createVariable("/time", np.array(0.5, dtype="f4"),dims=("/time",))
RootGroup.createVariable("/time_bnds", np.arange(2, dtype="f4"), dims=("/time", "/nv"))
```


Since there is a bit of ambiguity when between determining whether there is a`Sequence` or a `Structure` from the fully Qualifying name of a Variable (BaseType), this new method `.createDapType` should check, whenever there is a '.' or a '/' in the `FQN` and raise an appropriate error if there is None there. 

Thus, using this new method should follow the workflow above, that is

1. Create first a Group, Sequence or Structure with the appropriate attributes (dimensions, name, etc).
2. Create a Variable (BaseType) with a fully qualifying name that spans an existing hierarchy.



Discussion:
Mikejmnez: > Comment: at some point we might want to look for all the for loops that use range() and make sure they are, like ones here, small ranges. I bet that can get time consuming when the ranges are large. Nothing to do here though. Even huge datasets are shallow.

yeah - I need to optimize code here. But I also agree that we can do that in some other `refactoring PR`.



Mikejmnez: Fixed conflicts, merging now :) 

Patch:
diff --git a/src/pydap/model.py b/src/pydap/model.py
index 11bc0ba..a782ee8 100644
--- a/src/pydap/model.py
+++ b/src/pydap/model.py
@@ -169,6 +169,8 @@ therefore highly recommended.
 
 import copy
 import operator
+import re
+import warnings
 from collections import OrderedDict
 from collections.abc import Mapping
 from functools import reduce
@@ -517,6 +519,10 @@ class StructureType(DapType, Mapping):
             out[child.name] = copy.copy(child)
         return out
 
+    @property
+    def type(self):
+        return "Structure"
+
 
 class DatasetType(StructureType):
     """A root Dataset.
@@ -529,28 +535,29 @@ class DatasetType(StructureType):
         'B'
     """
 
-    def __setitem__(self, key, item):
-        # StructureType.__setitem__(self, key, item)
-
-        # is key a path-like?
-        N = len(key.split("/")[1:])
+    def __setitem__(self, key, item, **kwargs):
+        # key a path-like only in DAP4
+        parts = re.split(r"[/.]", key)[1:]
+        N = len(parts)
         if key[0] == "/" and N > 1:
-            parts = key.split("/")[1:]
-            # add parent group to dataset keys
+            # add parent container type if not there
             if parts[0] not in self._dict:
                 self._visible_keys.append(parts[0])
-            # iterate over all groups to reach DAP object
+            #  iterate over all groups to reach DAP object
             current = self._dict
             for j in range(N - 1):
                 if parts[j] not in current:
+                    # This current approach works when parsing a DMR
+                    # with only Groups and arrays. Need to enable
+                    # Sequences and Structures. This works with all
+                    # DAP4 when creating Dataset manally.
                     current[parts[j]] = GroupType(parts[j])
                 current = current[parts[j]]
             current[parts[-1]] = item
         else:
             if key[0] == "/":
                 key = key[1:]
-
-            key = _quote(key)  # should name be quoted?
+            key = _quote(key)
             if key != item.name:
                 raise KeyError(
                     'Key "%s" is different from variable name "%s"!' % (key, item.name)
@@ -563,8 +570,15 @@ class DatasetType(StructureType):
             # By default added keys are visible:
             self._visible_keys.append(key)
 
-        # The dataset name does not go into the children ids.
-        item.id = item.name
+        key = key.replace("%2E", ".")
+        if len(key.split(".")) == 1:
+            # The parent name does not go into the children ids.
+            item.id = item.name
+        else:
+            parts = key.split("/")[-1]
+            item.id = (".").join(parts.split("."))
+            # Set item id.
+            # item.id  = "%s.%s" % (parent_name, item.name)
 
     def _getitem_string(self, key):
         """Assume that key is a string type"""
@@ -630,6 +644,73 @@ class DatasetType(StructureType):
             nbytes += var.nbytes
         return nbytes
 
+    def createDapType(self, daptype, name, **attrs):
+        # Creates a temporal quasi FQN by replacing all `.` with `/`
+        # since these are safe to escape. This catches all cases
+        # much more cleanly. User specifies FQN but DAPtype is defined
+        # within the method so OK.
+        if name[0] != "/":
+            warnings.warn(
+                """name must start with `/` to have a Fully Qualifing Name.
+                Adding a `/` to the begining of the name."""
+            )
+            # makes sure DAP4 fqn name (even when DAP2)
+            name = "/" + name
+        parts = re.split(r"[/.]", name)[1:]
+        item = daptype(name=parts[-1], **attrs)
+        try:
+            for i in range(1, len(parts)):
+                self[("/").join(parts[:i])]
+        except KeyError:
+            warnings.warn(
+                """Failed to create `{}` because parent `{}` does not exist!
+                """.format(
+                    parts[-1], parts[-2]
+                )
+            )
+            return None
+        DatasetType.__setitem__(self, name, item)
+
+    def createGroup(self, name, **attrs):
+        """
+        Creates a Group from `root`, even in the presence of nested Groups.
+        Uses the `Fully Qualifying Name`. Parent `Group` must exist.
+
+        Also: https://docs.opendap.org/index.php/DAP4:_Specification_Volume_1
+        """
+        return self.createDapType(GroupType, name, **attrs)
+
+    def createVariable(self, name, **attrs):
+        """
+        Creates a Variable (`BaseType`) from `root`, even in the presence of nested
+        DAPTypes (i.e. `GroupType`, `SequenceType`, `StructureType`). Uses the
+        `Fully Qualifying Name`. Parent `DAPType` must exist!
+
+        Also: https://docs.opendap.org/index.php/DAP4:_Specification_Volume_1
+
+        """
+        return self.createDapType(BaseType, name, **attrs)
+
+    def createSequence(self, name, **attrs):
+        """
+        Creates a Sequence (`SequenceType`) from `root`, even in the presence of nested
+        DAPTypes (i.e. `GroupType`, `SequenceType`, `StructureType`).
+        Uses the `Fully Qualifying Name`. Parent `DAPType` must exist!
+
+        Also: https://docs.opendap.org/index.php/DAP4:_Specification_Volume_1
+        """
+        return self.createDapType(SequenceType, name, **attrs)
+
+    def createStructure(self, name, **attrs):
+        """
+        Creates a Structure (`StructureType`) from `root`, even in the presence of
+        nested DAPTypes (i.e. `GroupType`, `SequenceType`, `StructureType`).
+        Uses the `Fully Qualifying Name`. Parent `DAPType` must exist!
+
+        Also: https://docs.opendap.org/index.php/DAP4:_Specification_Volume_1
+        """
+        return self.createDapType(StructureType, name, **attrs)
+
 
 class SequenceType(StructureType):
     """A container that stores data in a Numpy array.
@@ -786,6 +867,10 @@ class SequenceType(StructureType):
         out.id = self.id
         return out
 
+    @property
+    def type(self):
+        return "Sequence"
+
 
 class GridType(StructureType):
     """A Grid container.
@@ -909,3 +994,7 @@ class GroupType(StructureType):
 
         for child in self.children():
             child.id = child.name
+
+    @property
+    def type(self):
+        return "Group"


Test patch:
diff --git a/src/pydap/tests/datasets.py b/src/pydap/tests/datasets.py
index a225116..24f9f51 100644
--- a/src/pydap/tests/datasets.py
+++ b/src/pydap/tests/datasets.py
@@ -12,14 +12,7 @@ import numpy as np
 
 from pydap.client import open_file
 from pydap.handlers.lib import IterData
-from pydap.model import (
-    BaseType,
-    DatasetType,
-    GridType,
-    GroupType,
-    SequenceType,
-    StructureType,
-)
+from pydap.model import BaseType, DatasetType, GridType
 
 # Note that DAP2 does not support signed bytes (signed 8bits integers).
 
@@ -27,11 +20,11 @@ from pydap.model import (
 # directly to a Numpy structured array, and can be easily encoded and decoded
 # in the DAP spec.
 VerySimpleSequence = DatasetType("VerySimpleSequence")
-VerySimpleSequence["sequence"] = SequenceType("sequence")
-VerySimpleSequence["sequence"]["byte"] = BaseType("byte")
-VerySimpleSequence["sequence"]["int"] = BaseType("int")
-VerySimpleSequence["sequence"]["float"] = BaseType("float")
-VerySimpleSequence["sequence"].data = np.array(
+VerySimpleSequence.createSequence("/sequence")
+VerySimpleSequence.createVariable("/sequence.byte")
+VerySimpleSequence.createVariable("/sequence.int")
+VerySimpleSequence.createVariable("/sequence.float")
+VerySimpleSequence["/sequence"].data = np.array(
     [
         (0, 1, 10.0),
         (1, 2, 20.0),
@@ -48,15 +41,15 @@ VerySimpleSequence["sequence"].data = np.array(
 
 # A nested sequence.
 NestedSequence = DatasetType("NestedSequence")
-NestedSequence["location"] = SequenceType("location")
-NestedSequence["location"]["lat"] = BaseType("lat")
-NestedSequence["location"]["lon"] = BaseType("lon")
-NestedSequence["location"]["elev"] = BaseType("elev")
-NestedSequence["location"]["time_series"] = SequenceType("time_series")
-NestedSequence["location"]["time_series"]["time"] = BaseType("time")
-NestedSequence["location"]["time_series"]["slp"] = BaseType("slp")
-NestedSequence["location"]["time_series"]["wind"] = BaseType("wind")
-NestedSequence["location"].data = IterData(
+NestedSequence.createSequence("/location")
+NestedSequence.createVariable("/location.lat")
+NestedSequence.createVariable("/location.lon")
+NestedSequence.createVariable("/location.elev")
+NestedSequence.createSequence("/location.time_series")
+NestedSequence.createVariable("/location.time_series.time")
+NestedSequence.createVariable("/location.time_series.slp")
+NestedSequence.createVariable("/location.time_series.wind")
+NestedSequence["/location"].data = IterData(
     [
         (1, 1, 1, [(10, 11, 12), (21, 22, 23)]),
         (2, 4, 4, [(15, 16, 17)]),
@@ -70,9 +63,9 @@ NestedSequence["location"].data = IterData(
 # A simple array with bytes, strings and shorts. These types require special
 # encoding for the DODS response.
 SimpleArray = DatasetType("SimpleArray")
-SimpleArray["byte"] = BaseType("byte", np.arange(5, dtype=np.ubyte))
-SimpleArray["string"] = BaseType("string", np.array(["one", "two"]))
-SimpleArray["short"] = BaseType("short", np.array(1, dtype="h"))
+SimpleArray.createVariable("byte", data=np.arange(5, dtype=np.ubyte))
+SimpleArray.createVariable("string", data=np.array(["one", "two"]))
+SimpleArray.createVariable("short", data=np.array(1, dtype="h"))
 
 
 DODS = os.path.join(os.path.dirname(__file__), "data/rainfall_time_malaysia.cdp.dods")
@@ -82,11 +75,11 @@ dapper = open_file(DODS, DAS)
 
 # dataset from http://test.opendap.org:8080/dods/dts/D1.asc
 D1 = DatasetType("EOSDB.DBO", type="Drifters")
-D1["Drifters"] = SequenceType("Drifters")
-D1["Drifters"]["instrument_id"] = BaseType("instrument_id")
-D1["Drifters"]["location"] = BaseType("location")
-D1["Drifters"]["latitude"] = BaseType("latitude")
-D1["Drifters"]["longitude"] = BaseType("longitude")
+D1.createSequence("/Drifters")
+D1.createVariable("/Drifters.instrument_id")
+D1.createVariable("/Drifters.location")
+D1.createVariable("/Drifters.latitude")
+D1.createVariable("/Drifters.longitude")
 D1.Drifters.data = np.array(
     np.rec.fromrecords(
         list(
@@ -110,8 +103,8 @@ D1.Drifters.data = np.array(
 
 # testing structures
 SimpleStructure = DatasetType("SimpleStructure")
-SimpleStructure["types"] = StructureType(
-    name="types",
+SimpleStructure.createStructure(
+    name="/types",
     key="value",
     nested=OrderedDict(
         [
@@ -122,19 +115,19 @@ SimpleStructure["types"] = StructureType(
         ]
     ),
 )
-SimpleStructure["types"]["b"] = BaseType("b", np.array(-10, np.byte))
-SimpleStructure["types"]["ub"] = BaseType("ub", np.array(10, np.ubyte))
-SimpleStructure["types"]["i32"] = BaseType("i32", np.array(-10, np.int32))
-SimpleStructure["types"]["ui32"] = BaseType("ui32", np.array(10, np.uint32))
-SimpleStructure["types"]["i16"] = BaseType("i16", np.array(-10, np.int16))
-SimpleStructure["types"]["ui16"] = BaseType("ui16", np.array(10, np.uint16))
-SimpleStructure["types"]["f32"] = BaseType("f32", np.array(100.0, np.float32))
-SimpleStructure["types"]["f64"] = BaseType("f64", np.array(1000.0, np.float64))
-SimpleStructure["types"]["s"] = BaseType(
-    "s", np.array("This is a data test string (pass 0).")
+SimpleStructure.createVariable(name="/types.b", data=np.array(-10, np.byte))
+SimpleStructure.createVariable(name="/types.ub", data=np.array(10, np.ubyte))
+SimpleStructure.createVariable(name="/types.i32", data=np.array(-10, np.int32))
+SimpleStructure.createVariable(name="/types.ui32", data=np.array(10, np.uint32))
+SimpleStructure.createVariable(name="/types.i16", data=np.array(-10, np.int16))
+SimpleStructure.createVariable(name="/types.ui16", data=np.array(10, np.uint16))
+SimpleStructure.createVariable(name="/types.f32", data=np.array(100.0, np.float32))
+SimpleStructure.createVariable(name="/types.f64", data=np.array(1000.0, np.float64))
+SimpleStructure.createVariable(
+    name="/types.s", data=np.array("This is a data test string (pass 0).")
 )
-SimpleStructure["types"]["u"] = BaseType("u", np.array("http://www.dods.org"))
-SimpleStructure["types"]["U"] = BaseType("U", np.array("test unicode", str))
+SimpleStructure.createVariable(name="/types.u", data=np.array("http://www.dods.org"))
+SimpleStructure.createVariable(name="/types.U", data=np.array("test unicode", str))
 
 
 # test grid
@@ -149,11 +142,11 @@ rain["rain"]["y"] = BaseType("y", np.arange(2), units="degrees_north")
 
 # test for ``bounds`` function
 bounds = DatasetType("test")
-bounds["sequence"] = SequenceType("sequence")
-bounds["sequence"]["lon"] = BaseType("lon", axis="X")
-bounds["sequence"]["lat"] = BaseType("lat", axis="Y")
-bounds["sequence"]["depth"] = BaseType("depth", axis="Z")
-bounds["sequence"]["time"] = BaseType("time", axis="T", units="days since 1970-01-01")
+bounds.createSequence("/sequence")
+bounds.createVariable("/sequence.lon", axis="X")
+bounds.createVariable("/sequence.lat", axis="Y")
+bounds.createVariable("/sequence.depth", axis="Z")
+bounds.createVariable("/sequence.time", axis="T", units="days since 1970-01-01")
 bounds["sequence"]["measurement"] = BaseType("measurement")
 bounds.sequence.data = np.array(
     np.rec.fromrecords(
@@ -168,10 +161,11 @@ bounds.sequence.data = np.array(
 
 # test for density
 ctd = DatasetType("ctd")
-ctd["cast"] = SequenceType("cast")
-ctd["cast"]["temperature"] = BaseType("temperature")
-ctd["cast"]["salinity"] = BaseType("salinity")
-ctd["cast"]["pressure"] = BaseType("pressure")
+ctd.createSequence("cast")
+ctd.createVariable("/cast.temperature")
+ctd.createVariable("/cast.salinity")
+ctd.createVariable("/cast.pressure")
+
 ctd.cast.data = np.array(
     np.rec.fromrecords(
         [
@@ -186,17 +180,16 @@ ctd.cast.data = np.array(
 SimpleSequence = DatasetType(
     "SimpleSequence", description="A simple sequence for testing.", nested={"value": 42}
 )
-SimpleSequence["cast"] = SequenceType("cast")
-SimpleSequence["cast"]["id"] = BaseType("id")
-SimpleSequence["cast"]["lon"] = BaseType("lon", axis="X")
-SimpleSequence["cast"]["lat"] = BaseType("lat", axis="Y")
-SimpleSequence["cast"]["depth"] = BaseType("depth", axis="Z")
-SimpleSequence["cast"]["time"] = BaseType(
-    "time", axis="T", units="days since 1970-01-01"
-)
-SimpleSequence["cast"]["temperature"] = BaseType("temperature")
-SimpleSequence["cast"]["salinity"] = BaseType("salinity")
-SimpleSequence["cast"]["pressure"] = BaseType("pressure")
+SimpleSequence.createSequence("/cast")
+SimpleSequence.createVariable("/cast.id")
+SimpleSequence.createVariable("/cast.lon", axis="X")
+SimpleSequence.createVariable("/cast.lat", axis="Y")
+SimpleSequence.createVariable("/cast.depth", axis="Z")
+SimpleSequence.createVariable("/cast.time", axis="T", units="days since 1970-01-01")
+SimpleSequence.createVariable("/cast.temperature")
+SimpleSequence.createVariable("/cast.salinity")
+SimpleSequence.createVariable("/cast.pressure")
+
 SimpleSequence["cast"].data = np.array(
     np.rec.fromrecords(
         [
@@ -229,38 +222,38 @@ SimpleGroup = DatasetType(
     description="A simple group for testing.",
     dimensions={"time": 1, "nv": 2},
 )
-SimpleGroup["SimpleGroup"] = GroupType("SimpleGroup", dimensions={"Y": 4, "X": 4})
-SimpleGroup["/SimpleGroup/Temperature"] = BaseType(
-    "Temperature",
-    np.arange(10, 26, 1, dtype="f4").reshape(1, 4, 4),
+SimpleGroup.createGroup("SimpleGroup", dimensions={"Y": 4, "X": 4})
+SimpleGroup.createVariable(
+    name="/SimpleGroup/Temperature",
+    data=np.arange(10, 26, 1, dtype="f4").reshape(1, 4, 4),
     units="degrees_celsius",
     dims=("/time", "/SimpleGroup/Y", "/SimpleGroup/X"),
     _FillValue=np.inf,
 )
-SimpleGroup["/SimpleGroup/Salinity"] = BaseType(
-    "Salinity",
-    30 * np.ones(16, dtype="f4").reshape(1, 4, 4),
+SimpleGroup.createVariable(
+    name="/SimpleGroup/Salinity",
+    data=30 * np.ones(16, dtype="f4").reshape(1, 4, 4),
     units="psu",
     dims=("/time", "/SimpleGroup/Y", "/SimpleGroup/X"),
     _FillValue=np.nan,
 )
-SimpleGroup["/SimpleGroup/Y"] = BaseType(
-    "Y", np.arange(4, dtype="i2"), dims=("/SimpleGroup/Y",)
+SimpleGroup.createVariable(
+    name="/SimpleGroup/Y", data=np.arange(4, dtype="i2"), dims=("/SimpleGroup/Y",)
 )
-SimpleGroup["/SimpleGroup/X"] = BaseType(
-    "X", np.arange(4, dtype="i2"), dims=("/SimpleGroup/X",)
+SimpleGroup.createVariable(
+    name="/SimpleGroup/X", data=np.arange(4, dtype="i2"), dims=("/SimpleGroup/X",)
 )
-SimpleGroup["/time"] = BaseType(
-    "time",
-    np.array(0.5, dtype="f4"),
+SimpleGroup.createVariable(
+    name="/time",
+    data=np.array(0.5, dtype="f4"),
     dims=("/time",),
     attributes={
         "standard_name": "time",
         "bounds": "time_bnds",
     },
 )
-SimpleGroup["/time_bnds"] = BaseType(
-    "time_bnds", np.arange(2, dtype="f4"), dims=("/time", "/nv")
+SimpleGroup.createVariable(
+    name="/time_bnds", data=np.arange(2, dtype="f4"), dims=("/time", "/nv")
 )
 
 
diff --git a/src/pydap/tests/test_model.py b/src/pydap/tests/test_model.py
index 202c742..ccf752a 100644
--- a/src/pydap/tests/test_model.py
+++ b/src/pydap/tests/test_model.py
@@ -477,8 +477,7 @@ def test_SequenceType_copy(sequence_example):
 
 def test_DatasetType_get_item_directory_path():
     dataset = DatasetType("dataset")
-    child = BaseType("child")
-    dataset["child"] = child
+    dataset.createVariable("/child")
     assert dataset[""] == dataset
     assert dataset["/"] == dataset
 
@@ -494,6 +493,16 @@ def test_DatasetType_set_item_directory_path(sequence_example):
     assert isinstance(dataset["Group1/Group2/example"], SequenceType)
 
 
+def test_DatasetType_fails_createDAPType():
+    dataset = DatasetType("dataset")
+    name = "/Group1/Group2"
+    warning = "Failed to create `Group2` because parent `Group1` does not exist!"
+    with pytest.warns(UserWarning, match=warning):
+        dataset.createGroup(name)
+    assert "Group1" not in dataset
+    assert "Group2" not in dataset
+
+
 # Test pydap grids.
 @pytest.fixture()
 def gridtype_example():