Repository: vaexio/vaex

Issue:
CSV to HDF5 conversion never finishes?
I'm trying to convert a large (5 627 353 rows) CSV file to HDF5:

```
>>> import vaex
>>> df = vaex.from_csv("feat.csv", convert=True, chunk_size=50_000, copy_index=False)
```

This produced 113 HDF5 chunks, but now it seems to have stopped working. `strace` shows this in a loop:

```
[pid 187414] epoll_wait(343, [], 1, 0)  = 0
[pid 187414] stat("/usr/lib/python3.8/asyncio/tasks.py", {st_mode=S_IFREG|0644, st_size=33076, ...}) = 0
[pid 187414] stat("/home/me/.local/lib/python3.8/site-packages/vaex/execution.py", {st_mode=S_IFREG|0644, st_size=9682, ...}) = 0
[pid 187414] stat("/home/me/.local/lib/python3.8/site-packages/nest_asyncio.py", {st_mode=S_IFREG|0644, st_size=5358, ...}) = 0
[pid 187414] stat("/home/me/.local/lib/python3.8/site-packages/vaex/column.py", {st_mode=S_IFREG|0644, st_size=18853, ...}) = 0
[pid 187414] stat("/home/me/.local/lib/python3.8/site-packages/vaex/__init__.py", {st_mode=S_IFREG|0644, st_size=29617, ...}) = 0
[pid 187414] stat("/home/me/.local/lib/python3.8/site-packages/vaex/expression.py", {st_mode=S_IFREG|0644, st_size=45326, ...}) = 0
[pid 187414] stat("/home/me/.local/lib/python3.8/site-packages/vaex/dataframe.py", {st_mode=S_IFREG|0644, st_size=277401, ...}) = 0
[pid 187414] stat("/home/me/.local/lib/python3.8/site-packages/vaex/asyncio.py", {st_mode=S_IFREG|0644, st_size=1345, ...}) = 0
[pid 187414] getpid()                   = 187414
[pid 187414] futex(0x55903bde9120, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 188725] <... futex resumed>)       = 0
[pid 187414] <... futex resumed>)       = 1
[pid 188725] futex(0x7fc2cf96b968, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=81999, tv_nsec=413799091}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 187414] futex(0x7fc2cf96b968, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 188725] <... futex resumed>)       = 0
[pid 187414] <... futex resumed>)       = 1
[pid 188725] futex(0x7fc2cf96b970, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 187414] futex(0x7fc2cf96b970, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 188725] <... futex resumed>)       = -1 EAGAIN (Resource temporarily unavailable)
[pid 187414] <... futex resumed>)       = 0
[pid 188725] futex(0x7fc2cf96b970, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 187414] epoll_wait(343,  <unfinished ...>
[pid 188725] <... futex resumed>)       = 0
[pid 188725] sendto(345, "\0", 1, 0, NULL, 0) = 1
[pid 187414] <... epoll_wait resumed>[{EPOLLIN, {u32=344, u64=140471200383320}}], 1, -1) = 1
[pid 188725] futex(0x7fc2cf96b96c, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 187414] futex(0x7fc2cf96b96c, FUTEX_WAIT_BITSET_PRIVATE, 0, {tv_sec=81999, tv_nsec=414188319}, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 188725] <... futex resumed>)       = 0
[pid 187414] <... futex resumed>)       = -1 EAGAIN (Resource temporarily unavailable)
[pid 188725] futex(0x55903bde9120, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...>
[pid 187414] futex(0x7fc2cf96b970, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 187414] recvfrom(344, "\0", 4096, 0, NULL, NULL) = 1
[pid 187414] recvfrom(344, 0x5590575138d0, 4096, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)
[pid 187414] getpid()                   = 187414
[pid 187414] getpid()                   = 187414
[pid 187414] getpid()                   = 187414
```

`perf top`:

![image](https://user-images.githubusercontent.com/308347/83648135-5d8b4900-a5be-11ea-87ae-ab3b3b4bbe60.png)



Discussion:
![image](https://user-images.githubusercontent.com/308347/83648728-13569780-a5bf-11ea-8617-b7869a212ca2.png)

Maybe there's something quadratic or worse in `add_column`? My CSV has 617 columns.
![image](https://user-images.githubusercontent.com/308347/83649567-05554680-a5c0-11ea-8c13-3baab876ae9b.png)

Hi Laurențiu,

Thanks for all the detail, there is a similar issue I think (which I have trouble finding), reporting that concatate takes a long time. Indeed, it's worse than quadratic iirc.
Can you confirm this by opening a few files, and concatenating them manually:
`vaex.concat([df1, df2])` and vaex.concat([df1, df2, df3])

cheers,

Maarten
> Can you confirm this by opening a few files, and concatenating them manually

That never finishes in a reasonable time either, presumably because they have the same large number of columns (I opened two of the chunks from earlier).
This is likely related to #604 

Looks like it, thanks.
Jovan; now with the benchmarking in place, this is a good time to optimize
this

(from mobile phone)

On Wed, 3 Jun 2020, 17:09 Laurențiu Nicola, <notifications@github.com>
wrote:

> Closed #788 <https://github.com/vaexio/vaex/issues/788>.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/vaexio/vaex/issues/788#event-3402652826>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AANPEPP6V7M5L7YK6PU3RLTRUZRTBANCNFSM4NRW2V4A>
> .
>

By the way, I commented out a couple of lines and it finally finished. But it's still quadratic, I think.
Which lines?

(from mobile phone)

On Wed, 3 Jun 2020, 18:41 Laurențiu Nicola, <notifications@github.com>
wrote:

> By the way, I commented out a couple of lines and it finally finished.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/vaexio/vaex/issues/788#issuecomment-638316069>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AANPEPO46BU2YNBRVEONUJLRUZ4MLANCNFSM4NRW2V4A>
> .
>

https://github.com/vaexio/vaex/blob/2eaba353cc2f330933d6e9a4392b792bfad98183/packages/vaex-core/vaex/dataframe.py#L2799-L2802 and I replaced https://github.com/vaexio/vaex/blob/2eaba353cc2f330933d6e9a4392b792bfad98183/packages/vaex-core/vaex/dataframe.py#L2818-L2820 with `valid_name = name`. I don't know if it's still called, but I also made `find_valid_name` return its argument unchanged.

https://github.com/vaexio/vaex/blob/2eaba353cc2f330933d6e9a4392b792bfad98183/packages/vaex-core/vaex/dataframe.py#L2832-L2833 is also quadratic because of the `valid_name not in self.column_names` check. At least the `insert` seems to happen mostly at the end, so it's probably fine.

Resolving patch:
diff --git a/packages/vaex-core/vaex/column.py b/packages/vaex-core/vaex/column.py
--- a/packages/vaex-core/vaex/column.py
+++ b/packages/vaex-core/vaex/column.py
@@ -272,10 +272,10 @@ def __init__(self, expressions, dtype=None, shape=None, fill_value=None):
         if shape is not None:
             self.shape = (len(self),) + shape
         else:
-            self.shape = (len(self), ) + self.expressions[0][0:1].to_numpy().shape[1:]
+            self.shape = (len(self), ) + self.expressions[0].evaluate(0, 1, array_type='numpy', parallel=False).shape[1:]
             for i in range(1, len(self.expressions)):
                 expression = self.expressions[i]
-                shape_i = (len(self), ) + expressions[i][0:1].to_numpy().shape[1:]
+                shape_i = (len(self), ) + expressions[i].evaluate(0, 1, array_type='numpy', parallel=False).shape[1:]
                 if self.shape != shape_i:
                     raise ValueError("shape of of expression %s, array index 0, is %r and is incompatible with the shape of the same column of array index %d, %r" % (self.expressions[0], self.shape, i, shape_i))
 
diff --git a/packages/vaex-core/vaex/expression.py b/packages/vaex-core/vaex/expression.py
--- a/packages/vaex-core/vaex/expression.py
+++ b/packages/vaex-core/vaex/expression.py
@@ -716,8 +716,8 @@ def countmissing(self):
         """Returns the number of missing values in the expression."""
         return self.ismissing().astype('int').sum().item()  # so the output is int, not array
 
-    def evaluate(self, i1=None, i2=None, out=None, selection=None, parallel=True):
-        return self.ds.evaluate(self, i1, i2, out=out, selection=selection, parallel=parallel)
+    def evaluate(self, i1=None, i2=None, out=None, selection=None, parallel=True, array_type=None):
+        return self.ds.evaluate(self, i1, i2, out=out, selection=selection, array_type=array_type, parallel=parallel)
 
     # TODO: it is not so elegant we need to have a custom version of this
     # it now also misses the docstring, reconsider how the the meta class auto


Test patch: