Repository: scikit-hep/awkward

Issue:
`ak.packed` casts categorical to value type
### Version of Awkward Array

1.10.0.rc2

### Description and code to reproduce

`ak.packed` seems to get rid of categoricals

```python
import awkward._v2 as ak

a = ak.Array(["foo", "foo", "bar"])
a_cat = ak.to_categorical(a)

assert a_cat.typestr == ak.packed(a_cat).typestr, f"'{a_cat.typestr}' != '{ak.packed(a_cat).typestr}'"
```

```pytb
      3 a = ak.Array(["foo", "foo", "bar"])
      4 a_cat = ak.to_categorical(a)
----> 6 assert a_cat.typestr == ak.packed(a_cat).typestr, f"'{a_cat.typestr}' != '{ak.packed(a_cat).typestr}'"

AssertionError: '3 * categorical[type=string]' != '3 * string'
```

This makes sense in that the indexed array is projected, but I did not expect `ak.packed` to change the type of an array.

The docstring also says this:

> Returns an array with the same type and values as the input, but with packed inner structures


Discussion:
I suspect this is a manifestation of #1685 
No, actually this is just that projecting out categoricals is not the correct behaviour
Yeah, this

https://github.com/scikit-hep/awkward/blob/31f3afb6bd31949bb2014d3be4b86194a63a573c/src/awkward/_v2/contents/indexedarray.py#L1246-L1247

should only project itself if _not_ categorical. Otherwise, it should pass on the recursive packing to its `content`, but not change itself.

```python
    def packed(self): 
        if self.parameter("__array__") == "categorical":
            return self.copy(content=content.packed())
        else:
            return self.project().packed() 
```

Actually, @agoose77 just pointed out in Slack that we should also remove unreachable items from `content`, which is the purpose of `packed`. I stand corrected: it's more complicated than my suggested fix.
Would unreachable items include categories that are no longer present, or just values from the integer array that aren't referenced?
Yes; the simplest fix is just to pack the _content_ of the categorical indexed array, but if we want to do this properly, we should produce a new indexed type which only includes the items that are referenced in `index`. That corresponds to keeping only the categories that are referred to by at least one index.
> we should produce a new indexed type which only includes the items that are referenced in index. That corresponds to keeping only the categories that are referred to by at least one index.

I would argue against this.

I think the set of categories shouldn't be modified by this function. I may only have a subset of the categories present in one instance, but would want to combine it with another instance that has the full set. If I know my set of categories ahead of time, I would like to just have all my arrays refer to them so I don't have to re-map the codes. This could also cause the codes to lose meaning if their order was important.

I think this is why some categorical array implementations model the categories as being part of the type, e.g. pandas [`CategoricalDtype`](https://pandas.pydata.org/docs/user_guide/categorical.html#categoricaldtype).
In that case, the simple implementation in https://github.com/scikit-hep/awkward/issues/1688#issuecomment-1240737263 is all we need.

Resolving patch:
diff --git a/src/awkward/_v2/contents/indexedarray.py b/src/awkward/_v2/contents/indexedarray.py
--- a/src/awkward/_v2/contents/indexedarray.py
+++ b/src/awkward/_v2/contents/indexedarray.py
@@ -1244,7 +1244,16 @@ def continuation():
             raise ak._v2._util.error(AssertionError(result))
 
     def packed(self):
-        return self.project().packed()
+        if self.parameter("__array__") == "categorical":
+            return IndexedArray(
+                self._index,
+                self._content.packed(),
+                identifier=self._identifier,
+                parameters=self._parameters,
+                nplike=self._nplike,
+            )
+        else:
+            return self.project().packed()
 
     def _to_list(self, behavior, json_conversions):
         out = self._to_list_custom(behavior, json_conversions)
@@ -1261,8 +1270,8 @@ def _to_nplike(self, nplike):
         return IndexedArray(
             index,
             content,
-            identifier=self.identifier,
-            parameters=self.parameters,
+            identifier=self._identifier,
+            parameters=self._parameters,
             nplike=nplike,
         )
 
diff --git a/src/awkward/_v2/contents/indexedoptionarray.py b/src/awkward/_v2/contents/indexedoptionarray.py
--- a/src/awkward/_v2/contents/indexedoptionarray.py
+++ b/src/awkward/_v2/contents/indexedoptionarray.py
@@ -1684,10 +1684,20 @@ def continuation():
 
     def packed(self):
         original_index = self._index.raw(self._nplike)
-
         is_none = original_index < 0
         num_none = self._nplike.index_nplike.count_nonzero(is_none)
-        if self._content.length > len(original_index) - num_none:
+        if self.parameter("__array__") == "categorical" or self._content.length <= (
+            len(original_index) - num_none
+        ):
+            return ak._v2.contents.IndexedOptionArray(
+                self._index,
+                self._content.packed(),
+                self._identifier,
+                self._parameters,
+                self._nplike,
+            )
+
+        else:
             new_index = self._nplike.index_nplike.empty(
                 len(original_index), dtype=original_index.dtype
             )
@@ -1704,15 +1714,6 @@ def packed(self):
                 self._nplike,
             )
 
-        else:
-            return ak._v2.contents.IndexedOptionArray(
-                self._index,
-                self._content.packed(),
-                self._identifier,
-                self._parameters,
-                self._nplike,
-            )
-
     def _to_list(self, behavior, json_conversions):
         out = self._to_list_custom(behavior, json_conversions)
         if out is not None:


Test patch:
diff --git a/tests/v2/test_1688-pack-categorical.py b/tests/v2/test_1688-pack-categorical.py
new file mode 100644
--- /dev/null
+++ b/tests/v2/test_1688-pack-categorical.py
@@ -0,0 +1,24 @@
+# BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/main/LICENSE
+
+import pytest  # noqa: F401
+import awkward as ak  # noqa: F401
+
+
+numpy = ak.nplike.Numpy.instance()
+
+
+def test():
+    this = ak._v2.to_categorical(["one", "two", "one", "three", "one", "four"])
+    assert ak._v2.is_categorical(this)
+    # Ensure packing by itself doesn't change the type
+    this_packed = ak._v2.packed(this)
+    assert this_packed.type == this.type
+    # Ensure the categories match between the two
+    assert ak._v2.all(ak._v2.categories(this_packed) == ak._v2.categories(this))
+
+    # Ensure the inner types match (ignoring the length change)
+    this_subset_packed = ak._v2.packed(this[:-1])
+    assert ak._v2.is_categorical(this_subset_packed)
+    assert this_subset_packed.type.content == this.type.content
+    # Ensure the categories match between the two
+    assert ak._v2.all(ak._v2.categories(this_subset_packed) == ak._v2.categories(this))