Repository: vaexio/vaex

Issue:
head() method not displaying result after doing a replace() with regex


**Software information**
 - Vaex version (`import vaex; vaex.__version__)`: 4.7.0
 - Vaex was installed via: pip / conda-forge / from source : Vaex installed through conda-forge
 - OS: macOS
 - Conda  4.10.3

**Additional information**
Please state any supplementary information or provide additional context for the problem (e.g. screenshots, data, etc..).


Thanks for this great package. I'm not sure if this is a bug or how to classify it but I thought it was just unusual . So I ran the following code to do a replace on all empty strings in my data. The code ran quite well and fast without any error but when I run a head()  method to view the result, the process runs indefinitely. I figured it's because of the volume of data  (8 million + rows) but then I repeated the task on smaller data (5 rows) but the same thing occurred. However, trying this task on a pandas dataframe gave me the result I wanted and I was able to view the result. Thanks


```
%time var = sample.get_column_names() 

for i in var:
    if i == 'ibe8579_8579' or i == 'ibe8592_8592' or i == 'raw_create_timestamp' or i == 'raw_load_date':
        sample[i]
    else:
        sample[i] = sample[i].str.replace(pat = r'^\s*$', repl = 'empty', regex = True)


head(2)
```





Discussion:
Hi, can you make a reproducible example? Small with fake data will do. Thanks
This example works for me  (on latest master)

```python
import vaex

x = ['asad', 'asf adf ', '', '', 'asfasf']
df = vaex.from_arrays(x=x)
df['x'] = df['x'].str.replace(pat = r'^\s*$', repl = 'empty', regex = True)
print(df)
```
Ok I think I can reproduce it. I will make tests, thanks!


The issue happens only if an empty string appears as the first element of a column.  
Ok, as a workaround you can try using `str_pandas`:

```python
sample[i] = sample[i].str_pandas.replace(pat = r'^\s*$', repl = 'empty', regex = True)
```

This will not be as performant, but it should work until we fix the issue. The issue only happens if you have an empty string in the 1st element. So in principle you can adjust your loop to check for this, and if true, use `str_pandas.replace`, otherwise use `str.replace` for performance. 

Thanks for reporting this. Let's keep this open until solved.
How to reproduce the issue @maartenbreddels :

```python
import vaex
x = ['', '', 'please']
df = vaex.from_arrays(x=x,)
df['x_fill'] = df.x.str.replace(pat = r'^\s*$', repl = 'empty', regex = True)
```

Sorry for my late response. I'm not allowed to share such info. I'm not sure if I mentioned this before or if it's relevant but my data contains 8+ million rows and 417 columns hence the reason for the for loop. You mentioned that this happens when some of the columns has empty strings as the first value, this is true for my data. That being said, I'm not sure how to modify my code to work according to your suggestion. I'd appreciate it if you could help me with that. 

Would following your method retain the type of data as a Vaex dataframe and would I be able to call the head method after running the code? 


Thanks
When we ask for a reproducible example we don't mean with your original data necessarily. It can be any fake made up date that raises the same issue. See my examples above.  If we can't recreate your exact case, we can't help. This is probably true for any similar project. So make sure you include something we can run/test on our end. 

I meant (as a temporary solution) to use `str_pandas.replace` instead of `str.replace` in your original example. 
Everything will be the same, but the evaluation of the expressions will be significantly slower, as it will use pandas under the hood. You will still get lazy vaex expressions as before. 

You can also try the `where` statement. Something like `df[col] = df.func.where(df[col] == '', 'empty', df[col])`. [You can see the docs for more details](https://vaex.io/docs/api.html#expression-class). Not sure which one will be more performant. 
I tried using the where statement but it unfortunately couldn't catch all the variation of empty strings in the data , I was left with some stubborn ones but I was able to run the head() method, that was why I resulted to using regex but couldn't view the data with the head() method after running the aforementioned code. Nonetheless, I will try your suggestion. Thanks for responding so swiftly.
Goodnews!!! It worked like a charm. I didn't know I could use pandas methods with Vaex arguments on a Vaex dataframe.  I'm shocked or am I missing something? Just incase anyone might need this, this is how I modified my code. A huge THANKS to you. 



`%time var = bge.get_column_names()  # Ben

for i in var:
    if i == 'ibe8579_8579' or i == 'ibe8592_8592' or i == 'raw_create_timestamp' or i == 'raw_load_date':
        bge[i]
    else:
        bge[i] = bge[i].str_pandas.replace(pat = r'^\s*$', repl = 'empty',  n = -1, regex = True)`

Resolving patch:
diff --git a/packages/vaex-core/vaex/functions.py b/packages/vaex-core/vaex/functions.py
--- a/packages/vaex-core/vaex/functions.py
+++ b/packages/vaex-core/vaex/functions.py
@@ -1735,6 +1735,13 @@ def str_replace(x, pat, repl, n=-1, flags=0, regex=False):
     """
     sl = _to_string_sequence(x).replace(pat, repl, n, flags, regex)
     return column.ColumnStringArrow.from_string_sequence(sl)
+    # TODO: we should be using arrow compute functions (since MB rewrote them in arrow)
+    # except that they don't support flags yet (case insensitive)
+    # we could invoke it when flags=0 though
+    # if regex:
+    #     return pc.replace_substring_regex(x, pattern=pat, max_replacements=None if n==-1 else n, replacement=repl)
+    # else:
+    #     return pc.replace_substring(x, pattern=pat, max_replacements=None if n==-1 else n, replacement=repl)
 
 
 @register_function(scope='str')


Test patch:
diff --git a/tests/strings_test.py b/tests/strings_test.py
--- a/tests/strings_test.py
+++ b/tests/strings_test.py
@@ -283,6 +283,13 @@ def test_string_replace(dfs, pattern, replacement, n):
     assert dfs.s.str.replace(pattern, replacement, n).tolist() == dfs.s.str_pandas.replace(pattern, replacement, n).tolist()
 
 
+def test_string_replace_empty():
+    x = ['', '', 'please']
+    df = vaex.from_arrays(x=x,)
+    x =  df.x.str.replace(pat = r'^\s*$', repl = 'empty', regex = True)
+    assert x.tolist() == ['empty', 'empty', 'please']
+
+
 @pytest.mark.parametrize("pattern", ["v", " "])
 @pytest.mark.parametrize("replacement", ["?", unicode_compat("VæX")])
 @pytest.mark.parametrize("flags", [0, int(re.IGNORECASE)])