Repository: scikit-hep/awkward

Issue:
Ravelling a string array drops string behavior
### Version of Awkward Array

1.8.0

### Description and code to reproduce

```python
>>> np.ravel(ak.Array(["this", "that"]))
<Array [116, 104, 105, 115, ... 104, 97, 116] type='8 * uint8'>
```


Discussion:
It should, though: `ak.ravel` changes the type, and so the old parameters are not necessarily going to be applicable to the new array. Parameters are used to determine whether a given `ak.Array` subclass applies, which has methods that assume some type, and parameters are also used to determine whether stand-alone operators (e.g. particular implementations of `+`) apply, which also assume some type.

In particular, `ak.ravel` turns the array into a NumpyArray with no other structure. `parameters={"__array__": "string"}` can only be applied to List-type arrays of uint8. After `ak.ravel`, it can't be strings because there are no List-type arrays.
I think this is a question of what to expect from `np.ravel` / `ak.ravel`. Fundamentally, I see a string as an atomic type, even though it is in reality composed of `uint8` characters. I don't see there being many uses for the average user in dropping the string abstraction, in the same way that I don't think dropping the record abstraction (#984) is often that useful.

If the user did want to drop this abstraction, then maybe we add an option to `ravel` to opt-in ignore the string behavior. 

So, perhaps a clearer summary of this issue and #984 is that I think we should be more conservative with what `ak.ravel` as an implementation of `np.ravel` does, namely that it preserves record-ness and string-ness. Of course, what that actually looks like for records is more complex, which is why I've not actually made a PR just yet ;)

@jpivarski in light of the recent work on strings and parameters during broadcasting, has your opinion on this changed? I.e., are you now in favour of treating strings as a mostly fundamental type by making it harder to lose string-ness?

I believe that users are much more likely to want to flatten `10 * var * var * string` to `10 * string` than to `N * uint8`. If users *do* want integers, it's still possible to do this by stripping off parameters:
```python
ak.ravel(ak.without_parameters(my_array_of_strings))
```
I guess there are a lot of exceptions for strings this is somewhat like having custom broadcasting for strings. A deeper question is whether we want to maintain the standard that all of the string type's customizations are things that one could do through `ak.behavior`. I think that's not currently the case; the question is whether we want to get back to that state.

So the proposal is for `ak.ravel` to stop when it encounters a string node, so that the output of `ak.ravel` maintains both a list structure and the identification of that list as a string? Then strings would be considered atomic, a thing that can't be split up, which is what users would intuitively expect. However, it would remove a property from `ak.ravel` that users might expect: right now, `ak.ravel` makes an array into something that could be a NumPy array (not just rectilinear, but completely flat). That would not be the case if it could output an array of strings. The question is which feature—not breaking strings or always outputting a NumPy-like array—is more important. We can't have both.
Coming back to this, 

> `ak.ravel` makes an array into something that could be a NumPy array (not just rectilinear, but completely flat). That would not be the case if it could output an array of strings.

`ak.to_numpy` already produces a flat array with `U` dtype for strings:
```python
>>> ak.to_numpy(ak.flatten(ak.Array([["this","that"]])))
array(['this', 'thast'], dtype='<U5')
```

I think we *can* have both! 
That would be to say that `ak.ravel` stops at strings and doesn't ravel within them, outputting a ListType array with string parameters (the first of the two choices).

But you're right, `ak.to_numpy` is ready to take it the rest of the way, padding those ListType arrays to make NumPy `dtype="U"`. (For this to be Unicode-aware enough to turn UTF-8 into UTF-32, it must be going through a Python for loop, but that can be fixed with the functions Martin is looking at.)

So yes, because of what `ak.to_numpy` does, I guess `ak.ravel` can be made to stop at strings and bytestrings.

What does `ak.flatten(ak.Array(["one", "two", "three"]))` do? That might also rip the strings apart because it considers `axis=1` to be the axis of lists-that-are-strings. Maybe they both need to be stopped. Or all functions? Does it need to be a codebase-wide `axis`-interpretation rule that lists-that-are-strings don't count as an axis?
Since 1.8.0, `ak.ravel` just flat-out refuses to operate on strings (which is clearly not what we want):
```pycon
>>> import awkward as ak
>>> ak.ravel(ak.Array(["hi", "bye"]))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/angus/Git/awkward/awkward/operations/ak_ravel.py", line 48, in ravel
    return _impl(array, highlevel, behavior)
  File "/home/angus/Git/awkward/awkward/operations/ak_ravel.py", line 61, in _impl
    out = ak.contents.NumpyArray(nplike.concatenate(out))
  File "/home/angus/Git/awkward/awkward/contents/numpyarray.py", line 49, in __init__
    ak.types.numpytype.dtype_to_primitive(self._data.dtype)
  File "/home/angus/Git/awkward/awkward/types/numpytype.py", line 47, in dtype_to_primitive
    raise ak._util.error(
TypeError: while calling (from <stdin>, line 1)

    ak.ravel(
        array = <Array ['hi', 'bye'] type='2 * string'>
        highlevel = True
        behavior = None
    )

Error details: unsupported dtype: dtype('<U3'). Must be one of

    bool, int8, uint8, int16, uint16, int32, uint32, int64, uint64, float32, float64, complex64, complex128, datetime64, timedelta64, float16, float128, complex256

or a datetime64/timedelta64 with units (e.g. 'datetime64[15us]')
```
This looks like a bug in which we produce the NumPy dtype, and then try to build a `NumpyArray` out of it (which is not correct; we don't handle strings this way).

Meanwhile, `ak.flatten` does behave as you expected:
```pycon
>>> import awkward as ak
>>> ak.flatten(ak.Array(["hi", "bye"]))
'hibye'
```

I think it _does_ need to be a rule that `axis=-1` means "the axis in which each element is a list of characters, i.e. a string". We already partially handle this. It would also help protect from attempting any ufunc operations on char-codes, which is no bad thing.

Maybe we need a task to track all of the string-related changes that need to be made.

Resolving patch:
diff --git a/src/awkward/contents/listarray.py b/src/awkward/contents/listarray.py
--- a/src/awkward/contents/listarray.py
+++ b/src/awkward/contents/listarray.py
@@ -1398,7 +1398,7 @@ def _completely_flatten(self, backend, options):
             self.parameter("__array__") == "string"
             or self.parameter("__array__") == "bytestring"
         ):
-            return [ak.operations.to_numpy(self)]
+            return [self]
         else:
             next = self.to_ListOffsetArray64(False)
             flat = next.content[next.offsets[0] : next.offsets[-1]]
diff --git a/src/awkward/contents/listoffsetarray.py b/src/awkward/contents/listoffsetarray.py
--- a/src/awkward/contents/listoffsetarray.py
+++ b/src/awkward/contents/listoffsetarray.py
@@ -1997,7 +1997,7 @@ def _completely_flatten(self, backend, options):
             self.parameter("__array__") == "string"
             or self.parameter("__array__") == "bytestring"
         ):
-            return [ak.operations.to_numpy(self)]
+            return [self]
         else:
             flat = self._content[self._offsets[0] : self._offsets[-1]]
             return flat._completely_flatten(backend, options)
diff --git a/src/awkward/contents/regulararray.py b/src/awkward/contents/regulararray.py
--- a/src/awkward/contents/regulararray.py
+++ b/src/awkward/contents/regulararray.py
@@ -1264,7 +1264,7 @@ def _completely_flatten(self, backend, options):
             self.parameter("__array__") == "string"
             or self.parameter("__array__") == "bytestring"
         ):
-            return [ak.operations.to_numpy(self)]
+            return [self]
         else:
             flat = self._content[: self._length * self._size]
             return flat._completely_flatten(backend, options)


Test patch:
diff --git a/tests/test_1377-ravel-string.py b/tests/test_1377-ravel-string.py
new file mode 100644
--- /dev/null
+++ b/tests/test_1377-ravel-string.py
@@ -0,0 +1,56 @@
+# BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/main/LICENSE
+
+import pytest  # noqa: F401
+
+import awkward as ak
+
+
+@pytest.mark.parametrize(
+    "data",
+    [
+        ["hello", "wo", "rld!", "what's", "occurring"],
+        [b"hello", b"wo", b"rld!", b"what's", b"occurring"],
+    ],
+)
+def test_ListOffsetArray(data):
+    layout = ak.from_iter(data, highlevel=False)
+    assert isinstance(layout, ak.contents.ListOffsetArray)
+    flattened = ak.ravel(layout, highlevel=False)
+    assert isinstance(flattened, ak.contents.Content)
+    assert flattened.to_list() == data
+
+
+@pytest.mark.parametrize(
+    "data",
+    [
+        ["hello", "wo", "rld!", "what's", "occurring"],
+        [b"hello", b"wo", b"rld!", b"what's", b"occurring"],
+    ],
+)
+def test_ListArray(data):
+    layout = ak.from_iter(data, highlevel=False)
+    assert isinstance(layout, ak.contents.ListOffsetArray)
+    as_list = ak.contents.ListArray(
+        layout.starts, layout.stops, layout.content, parameters=layout.parameters
+    )
+    assert isinstance(as_list, ak.contents.ListArray)
+    flattened = ak.ravel(as_list, highlevel=False)
+    assert isinstance(flattened, ak.contents.Content)
+    assert flattened.to_list() == data
+
+
+@pytest.mark.parametrize(
+    "data",
+    [
+        ["jack", "back", "tack", "rack", "sack"],
+        [b"jack", b"back", b"tack", b"rack", b"sack"],
+    ],
+)
+def test_RegularArray(data):
+    layout = ak.from_iter(data, highlevel=False)
+    assert isinstance(layout, ak.contents.ListOffsetArray)
+    as_regular = layout.to_RegularArray()
+    assert isinstance(as_regular, ak.contents.RegularArray)
+    flattened = ak.ravel(as_regular, highlevel=False)
+    assert isinstance(flattened, ak.contents.Content)
+    assert flattened.to_list() == data