Repository: scikit-hep/awkward

Issue:
string comparison of string typed array throws a UFuncTypeError
Minimal working example (hopefully better than the last one):
``` python
one = ak.Array(["uno", "dos", "tres"])
one_l = one.layout
two = ak.Array(["un", "deux", "trois", "quatre"])
two_l = two.layout
three = ak.Array(["onay", "ootay", "eethray"])
three_l = three.layout
assert ak.to_list(one_l.mergemany([two_l, three_l])) == ["uno", "dos", "tres", "un", "deux", "trois", "quatre", "onay", "ootay", "eethray"]
merged = one_l.mergemany([two_l, three_l])
```
Expected behavior:
``` python
one == "uno"
>>> <Array [True, False, False] type='3 * ?bool'>
merged == "longerstringthantyped"
>>> <Array [False, False, False, ... False, False] type='10 * ?bool'>
```
Bug:
``` python
ak.Array(merged) == "uno"
>>> UFuncTypeError: ufunc 'equal' did not contain a loop with signature matching types (dtype('S3'), dtype('S3')) -> dtype('bool')
```

Unfortunately, I haven't caught up with the changes lately. My preliminary check suggests that, this behavior was introduced with [v.0.2.37](https://github.com/scikit-hep/awkward-1.0/compare/0.2.36...0.2.37#diff-5c46d0b61253440d84e39816781d3277efccea7fb0a7997657c58856c5b21260) (more specifically with .mergemany() ([PR#449](https://github.com/scikit-hep/awkward-1.0/pull/449)), and quick guess somewhere around [here](https://github.com/scikit-hep/awkward-1.0/blob/5a90e58fab4d48f46dbc2ff3e14275eccb688349/src/libawkward/array/NumpyArray.cpp#L1596)).
I experienced this bug, when higher level functions like `from_arrow` (that call mergemany internally now) produced the same bug.



Discussion:
Generally, you should use high-level functions, like [ak.concatenate](https://awkward-array.readthedocs.io/en/latest/_auto/ak.concatenate.html) to concatenate:

```python
>>> merged = ak.concatenate([one, two, three])
```

but I reproduced the bug you reported. The difference between `{one, two, three}` and `merged` is that `merged` is a ListArray, rather than a ListOffsetArray, and it's likely that has never been tested.

It's somewhere in here:

https://github.com/scikit-hep/awkward-1.0/blob/40fdf1e6bf427dab7923934d6180e1621ea59e73/src/awkward1/behaviors/string.py#L100-L125

Note to self: also test it in Numba.
Ok, thanks for the quick reply.
I recently updated from pre-0.2.32 and experienced that bug with an array constructed via `from_arrow()` (same code and data that worked before).

The last working version was 0.2.36. Which is why I debugged 0.2.37 and tried to reproduce the bug with `mergemany()` directly (the main change in v0.2.37, and function that is called by `from_arrow`).

Interestingly, `merged` is a `NumpyArray` of `format='c'`. Not sure if [that](https://github.com/scikit-hep/awkward-1.0/blob/40fdf1e6bf427dab7923934d6180e1621ea59e73/src/libawkward/array/NumpyArray.cpp#L1593) string type is correct?
There are two format strings that correspond to raw bytes, and `"c"` is one of them. Within the C++ part of Awkward, I ignore the difference between them, but I think `"c"` is more appropriate for character data.

This format is probably the reason why NumPy raised the error it did, instead of a different one, but I think there's something structural going on...

You know, maybe not... I'm slow on the uptake sometimes—maybe this is exactly it. It will be the first thing I try, anyway. Thanks for the tip!

Resolving patch:
diff --git a/src/awkward1/nplike.py b/src/awkward1/nplike.py
--- a/src/awkward1/nplike.py
+++ b/src/awkward1/nplike.py
@@ -68,6 +68,8 @@ class NumpyMetadata(Singleton):
     float64 = numpy.float64
     complex64 = numpy.complex64
     complex128 = numpy.complex128
+    str_ = numpy.str_
+    bytes_ = numpy.bytes_
 
     intp = numpy.intp
     integer = numpy.integer
@@ -280,6 +282,10 @@ def __init__(self):
     def ma(self):
         return self._module.ma
 
+    @property
+    def char(self):
+        return self._module.char
+
 
 class Cupy(NumpyLike):
     def __init__(self):
@@ -305,6 +311,14 @@ def ma(self):
             + awkward1._util.exception_suffix(__file__)
         )
 
+    @property
+    def char(self):
+        raise ValueError(
+            "CUDA arrays cannot do string manipulations until CuPy implements "
+            "numpy.char"
+            + awkward1._util.exception_suffix(__file__)
+        )
+
     @property
     def ndarray(self):
         return self._module.ndarray
diff --git a/src/awkward1/operations/convert.py b/src/awkward1/operations/convert.py
--- a/src/awkward1/operations/convert.py
+++ b/src/awkward1/operations/convert.py
@@ -70,7 +70,38 @@ def recurse(array, mask):
             )
 
         if len(array.shape) == 0:
-            data = awkward1.layout.NumpyArray(array.reshape(1))
+            array = array.reshape(1)
+
+        if array.dtype.kind == "S":
+            asbytes = array.reshape(-1)
+            itemsize = array.dtype.itemsize
+            starts = numpy.arange(0, len(asbytes)*itemsize, itemsize, dtype=np.int64)
+            stops = starts + numpy.char.str_len(asbytes)
+            data = awkward1.layout.ListArray64(
+                awkward1.layout.Index64(starts),
+                awkward1.layout.Index64(stops),
+                awkward1.layout.NumpyArray(
+                    asbytes.view("u1"), parameters={"__array__": "byte"}
+                ),
+                parameters={"__array__": "bytestring"},
+            )
+            for size in array.shape[-1:0:-1]:
+                data = awkward1.layout.RegularArray(data, size)
+        elif array.dtype.kind == "U":
+            asbytes = numpy.char.encode(array.reshape(-1), "utf-8", "surrogateescape")
+            itemsize = array.dtype.itemsize // 4
+            starts = numpy.arange(0, len(asbytes)*itemsize, itemsize, dtype=np.int64)
+            stops = starts + numpy.char.str_len(asbytes)
+            data = awkward1.layout.ListArray64(
+                awkward1.layout.Index64(starts),
+                awkward1.layout.Index64(stops),
+                awkward1.layout.NumpyArray(
+                    asbytes.view("u1"), parameters={"__array__": "char"}
+                ),
+                parameters={"__array__": "string"},
+            )
+            for size in array.shape[-1:0:-1]:
+                data = awkward1.layout.RegularArray(data, size)
         else:
             data = awkward1.layout.NumpyArray(array)
 
@@ -1532,7 +1563,7 @@ def to_layout(
     array,
     allow_record=True,
     allow_other=False,
-    numpytype=(np.number, np.bool_, np.bool),
+    numpytype=(np.number, np.bool_, np.bool, np.str_, np.bytes_),
 ):
     """
     Args:
@@ -1574,32 +1605,13 @@ def to_layout(
     elif allow_record and isinstance(array, awkward1.layout.Record):
         return array
 
-    elif isinstance(array, numpy.ma.MaskedArray):
-        mask = numpy.ma.getmask(array)
-        data = numpy.ma.getdata(array)
-        if mask is False:
-            out = awkward1.layout.UnmaskedArray(
-                awkward1.layout.NumpyArray(data.reshape(-1))
-            )
-        else:
-            out = awkward1.layout.ByteMaskedArray(
-                awkward1.layout.Index8(mask.reshape(-1)),
-                awkward1.layout.NumpyArray(data.reshape(-1)),
-            )
-        for size in array.shape[:0:-1]:
-            out = awkward1.layout.RegularArray(out, size)
-        return out
-
-    elif isinstance(array, np.ndarray):
+    elif isinstance(array, (np.ndarray, numpy.ma.MaskedArray)):
         if not issubclass(array.dtype.type, numpytype):
             raise ValueError(
                 "NumPy {0} not allowed".format(repr(array.dtype))
                 + awkward1._util.exception_suffix(__file__)
             )
-        out = awkward1.layout.NumpyArray(array.reshape(-1))
-        for size in array.shape[:0:-1]:
-            out = awkward1.layout.RegularArray(out, size)
-        return out
+        return from_numpy(array, regulararray=True, recordarray=True, highlevel=False)
 
     elif isinstance(array, (str, bytes)) or (
         awkward1._util.py27 and isinstance(array, awkward1._util.unicode)


Test patch:
diff --git a/tests/test_0501-string-comparison-bug.py b/tests/test_0501-string-comparison-bug.py
new file mode 100644
--- /dev/null
+++ b/tests/test_0501-string-comparison-bug.py
@@ -0,0 +1,27 @@
+# BSD 3-Clause License; see https://github.com/scikit-hep/awkward-1.0/blob/master/LICENSE
+
+from __future__ import absolute_import
+
+import sys
+
+import pytest
+
+import numpy
+import awkward1
+
+
+def test():
+    one = awkward1.Array(["uno", "dos", "tres"])
+    two = awkward1.Array(["un", "deux", "trois", "quatre"])
+    three = awkward1.Array(["onay", "ootay", "eethray"])
+    merged = awkward1.concatenate([one, two, three])
+    assert awkward1.to_list(merged) == ["uno", "dos", "tres", "un", "deux", "trois", "quatre", "onay", "ootay", "eethray"]
+    assert awkward1.to_list(merged == "uno") == [True, False, False, False, False, False, False, False, False, False]
+    assert awkward1.to_list(one == numpy.array(["UNO", "dos", "tres"])) == [False, True, True]
+    assert awkward1.to_list(merged == numpy.array(["UNO", "dos", "tres", "one", "two", "three", "quatre", "onay", "two", "three"])) == [False, True, True, False, False, False, True, True, False, False]
+
+
+def test_fromnumpy():
+    assert awkward1.to_list(awkward1.from_numpy(numpy.array(["uno", "dos", "tres", "quatro"]))) == ["uno", "dos", "tres", "quatro"]
+    assert awkward1.to_list(awkward1.from_numpy(numpy.array([["uno", "dos"], ["tres", "quatro"]]))) == [["uno", "dos"], ["tres", "quatro"]]
+    assert awkward1.to_list(awkward1.from_numpy(numpy.array([["uno", "dos"], ["tres", "quatro"]]), regulararray=True)) == [["uno", "dos"], ["tres", "quatro"]]