Repository: ckan/ckan

Issue:
Data Dictionary Update Bug
**CKAN version**
2.8

**Describe the bug**
Data dictionary not updating properly.


**Steps to reproduce**
Example of inputting labels and descriptions:
![image](https://user-images.githubusercontent.com/62943444/93161993-58ee3900-f746-11ea-982a-e3935d81d86a.png)

Output:
![image](https://user-images.githubusercontent.com/62943444/93162031-70c5bd00-f746-11ea-95b6-8236a70926a2.png)

The description from the first field is being inputted into the description in the second field.

Which code is the logic of the editing of the field inputs? The front end code seems fine...

In addition, where in the postgresql database are the label and description being stored? I can't seem to find them.
Correctly unflatten extras
Fixes #5537 . Some degree of luck required for getting this error

One need `convert_from_extras` in order to reproduce it. Imagine dataset, created with following extras:
```python
package_create({..., extras=[
  {'key': 'a', 'value': '1'},
  {'key': 'b', 'value': '2'},
  {'key': 'c', 'value': '3'},
], ...}).
```
After flattening it looks like 
```python
{ ...,
  ('extras', 0, 'key'): 'a', ('extras', 0, 'value'): 1,
  ('extras', 1, 'key'): 'b', ('extras', 1, 'value'): 2,
  ('extras', 2, 'key'): 'c', ('extras', 2, 'value'): 3,
...}
```

If **only** `b` field have `convert_from_extras`, it will be converted to the following form with "gap" between **0** and **2**:
```python
{ ...,
  ('b',): 2,
  ('extras', 0, 'key'): 'a', ('extras', 0, 'value'): 1,
  ('extras', 2, 'key'): 'c', ('extras', 2, 'value'): 3,
...}
```

Afterward, [because of these lines](https://github.com/ckan/ckan/blob/master/ckan/lib/navl/dictization_functions.py#L416-L419) extra **2** will be unflattened into a broken form:
```python
{..., 
  "b": 2, 
  "extras": [
    {'key': 'a', 'value': '1'},
    {'key': 'c'},
    {'value': '3'},
  ], 
...}
```

Fixing this issue during unflattening will turn into an unnecessary complex block of code with a number of contradictory places, so I suggest "realigning" extras when they are updated by `convert_from_extras`. The example above will be converted to the following form when **b** property extracted from extras(the gap is fixed by shifting all the extras that follow this gap one position back):
```python
{ ...,
  ('b',): 2,
  ('extras', 0, 'key'): 'a', ('extras', 0, 'value'): 1,
  ('extras', 1, 'key'): 'c', ('extras', 1, 'value'): 3,
...}
```


Discussion:
I found that its got to do with the way the data_dict is being stored. In /ckan/lib/default/src/ckan/ckanext/datastore/dictionary.html, under the dictionary method, I printed out the info, fields and data variables and found that the structure is slightly off.

![image](https://user-images.githubusercontent.com/62943444/93166047-3bbe6800-f750-11ea-9c35-f414164c51d9.png)
  ![image](https://user-images.githubusercontent.com/62943444/93166131-6e686080-f750-11ea-879e-40adc6c3d814.png)
![image](https://user-images.githubusercontent.com/62943444/93165995-15003180-f750-11ea-8de7-65233db56abf.png)

As you can see, for the first 2 elements in the tuple, it will always have 1 missing label/notes sub element. I think the first two elements should be merged. Where can I change the structure of this tuple?
I think I am nearing the solution... I found out its due to the extra variable _id that always appear in a data table. How do I remove it?
![image](https://user-images.githubusercontent.com/62943444/93183806-09236800-f76e-11ea-89c8-9822fe6a6d14.png)

This would help to fix the bug in Data Dictionary! Please advise :)
Resolved it. Added the following code lines to /ckan/lib/default/src/ckan/ckanext/datastore/controller.py

![image](https://user-images.githubusercontent.com/62943444/93191816-8bfcf080-f777-11ea-9867-2662dd127d04.png)

@debbielee1996 thank you for this, are you using 2.8.5? There was a change to unflatten in this release that might have caused this to break. we have a patch that works around the issue for extras https://github.com/ckan/ckan/pull/5546/files but this sounds like we should fix unflatten instead of chasing the bugs created in other parts of the code.
Hi wardi, I installed CKAN 2.8.5. Will the patch you mentioned resolve the unflatten issue? Additionally, do you know where I can view the stored data dictionary in the postgresql database? I can't seem to find it.


Resolving patch:
diff --git a/ckan/lib/navl/dictization_functions.py b/ckan/lib/navl/dictization_functions.py
--- a/ckan/lib/navl/dictization_functions.py
+++ b/ckan/lib/navl/dictization_functions.py
@@ -405,7 +405,7 @@ def unflatten(data):
     '''
 
     unflattened = {}
-    convert_to_list = []
+    clean_lists = {}
 
     for flattend_key in sorted(data.keys(), key=flattened_order_key):
         current_pos = unflattened
@@ -414,8 +414,13 @@ def unflatten(data):
             try:
                 current_pos = current_pos[key]
             except IndexError:
-                new_pos = {}
-                current_pos.append(new_pos)
+                while True:
+                    new_pos = {}
+                    current_pos.append(new_pos)
+                    if key < len(current_pos):
+                        break
+                    # skipped list indexes need to be removed before returning
+                    clean_lists[id(current_pos)] = current_pos
                 current_pos = new_pos
             except KeyError:
                 new_pos = []
@@ -423,6 +428,9 @@ def unflatten(data):
                 current_pos = new_pos
         current_pos[flattend_key[-1]] = data[flattend_key]
 
+    for cl in clean_lists.values():
+        cl[:] = [i for i in cl if i]
+
     return unflattened
 
 
diff --git a/ckan/logic/converters.py b/ckan/logic/converters.py
--- a/ckan/logic/converters.py
+++ b/ckan/logic/converters.py
@@ -26,24 +26,14 @@ def convert_to_extras(key, data, errors, context):
 
 def convert_from_extras(key, data, errors, context):
 
-    def remove_from_extras(data, idx):
-        for key in sorted(data):
-            if key[0] != 'extras' or key[1] < idx:
-                continue
-            if key[1] == idx:
-                del data[key]
-
-            # Following block required for unflattening extras with
-            # "gaps" created sometimes by `convert_from_extra`
-            # validator :
-            #
-            #   {
-            #     ('extras', 0, 'key'): 'x',
-            #     ('extras', 2, 'key): 'y'
-            #   }
-            if key[1] > idx:
-                new_key = (key[0], key[1] - 1) + key[2:]
-                data[new_key] = data.pop(key)
+    def remove_from_extras(data, key):
+        to_remove = []
+        for data_key, data_value in six.iteritems(data):
+            if (data_key[0] == 'extras'
+                and data_key[1] == key):
+                to_remove.append(data_key)
+        for item in to_remove:
+            del data[item]
 
     for data_key, data_value in six.iteritems(data):
         if (data_key[0] == 'extras'


Test patch:
diff --git a/ckan/tests/legacy/lib/test_navl.py b/ckan/tests/legacy/lib/test_navl.py
--- a/ckan/tests/legacy/lib/test_navl.py
+++ b/ckan/tests/legacy/lib/test_navl.py
@@ -272,6 +272,22 @@ def test_flatten_deeper():
         unflatten(flatten_dict(data)))
 
 
+def test_unflatten_regression():
+    fdata = {
+        (u"items", 0, u"name"): u"first",
+        (u"items", 0, u"value"): u"v1",
+        (u"items", 3, u"name"): u"second",
+        (u"items", 3, u"value"): u"v2",
+    }
+    expected = {
+        u"items": [
+            {u"name": u"first", u"value": u"v1"},
+            {u"name": u"second", u"value": u"v2"},
+        ],
+    }
+    assert unflatten(fdata) == expected, pformat(unflatten(fdata))
+
+
 def test_simple():
     schema = {
         "name": [not_empty],