Repository: vaexio/vaex

Issue:
df.head error after join
df = df1.join(df2,how='left',on='col-name',allow_duplication=True)
then when i use df.head(3)  i see this error:
array is of length 5012, while the length of the DataFrame is 3
df1 and df2 are csv files and i created them with pandas after clustering algorithms. also df2  has null values but df2.head(3) works (and also works for df1) and after joining i got the error. i tried df.fillna(0).head(3) but still see that error.


Discussion:
Hi,

thanks for the report. It would help us a lot if you can provide a reproducible example!

Regards,

Maarten

thank you for your reply
example:
import pandas as pd 
import numpy as np
import vaex
df_test1 = pd.DataFrame(np.array([['a1', 2.0],
                                  ['a2', 1.0],
                                  ['a3', 3.0],
                                  ['a4', 5.0],
                                  ['a5', 6.0]]),columns =['a','b'])
df_test2 = pd.DataFrame(np.array([['a1',3],
                                  ['a2', 4],
                                  ['a2', 6],
                                  ['a1', None],
                                  ['a2', 6]]),columns =['a','c'])
df_test1.to_csv('test1.csv',header=True,index=False)
df_test2.to_csv('test2.csv',header=True,index=False)
df1 = vaex.from_csv('test1.csv', copy_index=False)
df2 = vaex.from_csv('test2.csv', copy_index=False)
df1.head():
	a	b
0	a1	2
1	a2	1
2	a3	3
3	a4	5
4	a5	6
df2.head():
       	a	c
0	a1	3
1	a2	4
2	a2	6
3	a1	nan
4	a2	6
df = df1.join(df2,on='a',how='left',allow_duplication=True)
df.head(3):
....ValueError: array is of length 5, while the length of the DataFrame is 3
Hi @alrz199 
Thanks for the reproducible example. I can also reproduce it and have already created a unit-test. We hope we can fix this soon. 

Thanks!

Resolving patch:
diff --git a/packages/vaex-core/vaex/column.py b/packages/vaex-core/vaex/column.py
--- a/packages/vaex-core/vaex/column.py
+++ b/packages/vaex-core/vaex/column.py
@@ -228,9 +228,11 @@ def trim(self, i1, i2):
         while start >= offset + len(self.expressions[i].df):
             offset += len(self.expressions[i].df)
             i += 1
-        expressions = [self.expressions[i][start-offset:]]
-        offset += len(self.expressions[i].df)
-        if stop > offset:
+        # do need more than 1?
+        if stop > offset + len(self.expressions[i].df):
+            # then add the first one
+            expressions = [self.expressions[i][start-offset:]]
+            offset += len(self.expressions[i].df)
             # we need more expression
             i += 1
             # keep adding complete expression till we find the it to be the last
@@ -240,6 +242,10 @@ def trim(self, i1, i2):
                 i += 1
             if stop > offset:  # add the tail part
                 expressions.append(self.expressions[i][:stop-offset])
+        else:
+            # otherwise we only need a slice of the first
+            expressions = [self.expressions[i][start-offset:stop-offset]]
+
         return ColumnConcatenatedLazy(expressions, self.dtype)
 
     def __getitem__(self, slice):


Test patch:
diff --git a/tests/concat_test.py b/tests/concat_test.py
--- a/tests/concat_test.py
+++ b/tests/concat_test.py
@@ -103,6 +103,7 @@ def test_sliced_concat(i1, length, df_concat):
     i2 = i1 + length
     x = df_concat.x.tolist()
     df = df_concat[i1:i2]
+    assert len(df_concat.columns['x'].trim(0, length)) == length
     assert df.x.tolist() == x[i1:i2]