Repository: vaexio/vaex

Issue:
join dataframes throws error in Vaex 4.0.0a5 upgrading from 3.0
**Description**
I recently upgraded the vaex version to 4.0.0a5 from 3.0.0. I experienced some differences when join dataframes.

The code : **A inner join B** works fine and took 61 seconds in V3, now throws error in V4.
       Error: Merging datasets with unequal row counts (83285687 != None)
 
I was able to resolve it by changing it to **B inner join A**, however after switching, the calculation time became 253 seconds from 61 seconds.



**Software information**
 - Vaex version (`import vaex; vaex.__version__)`: 4.0.0a5
 - Vaex was installed via: pip
 - OS:  Linux

**Additional information**

Here is information of my code

- Source df: a 400 million rows df (hd5f file)
- A df: 368 million rows filtered from the source df
- B df: 30 million rows filtered from the source df

Program has : read data, processing data + operation 1 + operation 2

  _operation 1_ :  join ( join A, B table on a long_col)
  _operation 2_ : group by the result from operation 1 and run below aggregation
              
	agg_dict = {'long_col': 'nunique',
                'numeric_col': 'sum',
                'numeric_col': 'sum'}

  The long_col is high cardinality categorical column

vaex version | join method | memory_rss | Total Program  | operation 1 | operation 2
-- | -- | -- | -- | -- | --
V3.0.0 | A inner join B | 30 GB | 61s | 31s | 5s
V3.0.0 | B inner jon A | 91 GB | 187s | 143s | 19s
V 4.0.0a5 | A inner join B | NA | Error: Merging datasets with unequal   row counts (83285687 != None)| NA| NA
V 4.0.0a5 | B inner jon A | 41 GB | 253s | 114s | 129s



Discussion:
Hi, can you please try the latest alpha:
`pip install vaex==4.0.0a13` and tell us if you experience the same thing?
I'm not the original poster but I encountered the same problem.
For me, installing vaex version 4.0.0a13 didn't work.
Hi, 

Can you please try with the latest version of vaex and let us know?

Also, is the data public? Would be nice to have such datasets so we can reproduce the issue locally. Otherwise hard to find the problem.

Thanks!
Closing this as stale. Please re-open if needed.
Also there have been a number of new versions with various performance improvements since this issue was created.
I am facing this problem, and the reproduction is very simple:

```
import numpy as np
import vaex

vaex.__version__

{'vaex': '4.8.0',
 'vaex-core': '4.8.0',
 'vaex-viz': '0.5.1',
 'vaex-hdf5': '0.12.0',
 'vaex-server': '0.8.1',
 'vaex-astro': '0.9.0',
 'vaex-jupyter': '0.7.0',
 'vaex-ml': '0.17.0'}

df1 = vaex.from_arrays(a=np.arange(1000).astype(np.int32) % 200,
                       b=np.random.random(1000))

df2 = vaex.from_arrays(a=np.array([1, 3, 10]).astype(np.int32))


df1.join(df2, on='a', how='inner')
# ValueError: Merging datasets with unequal row counts (3 != None)

df2.join(df1, on='a', how='inner')
# ValueError: This join will lead to duplication of rows which is disabled, pass allow_duplication=True

df2.join(df1, on='a', how='inner', allow_duplication=True)
# OK!
```

Am I doing something wrong, or does the join work only one way? I don't think it would make any difference on Pandas when inner join is used.

Ooh wait a second, why does this work:

```
df3 = vaex.from_arrays(a=np.array([1, 3, 10]).astype(np.int32),
                       c=[np.nan] * 3)

df1.join(df3, on='a', how='inner')
# Works, why???
```


Resolving patch:
diff --git a/packages/vaex-core/vaex/join.py b/packages/vaex-core/vaex/join.py
--- a/packages/vaex-core/vaex/join.py
+++ b/packages/vaex-core/vaex/join.py
@@ -1,4 +1,5 @@
 import numpy as np
+from vaex.dataframe import DataFrame
 
 import vaex.dataset
 from vaex.utils import _ensure_string_from_expression
@@ -124,8 +125,8 @@ def _decode(cls, encoding, spec):
 def join(df, other, on=None, left_on=None, right_on=None, lprefix='', rprefix='', lsuffix='', rsuffix='', how='left', allow_duplication=False, prime_growth=False, cardinality_other=None, inplace=False):
     # implementation of DataFrameLocal.join
     inner = False
-    left = df
-    right = other
+    left: DataFrame = df
+    right: DataFrame = other
     left_original = left.copy()
     right_original = right.copy()
     rprefix_original, lprefix_original = rprefix, lprefix
@@ -276,13 +277,16 @@ def mangle_name(prefix, name, suffix):
             left.column_names.append(name)
             left._initialize_column(name)
     # merge the two datasets
-    right_dataset = right.dataset.project(*right_columns)
-    if lookup is not None:
-        # if lookup is None, we do a row based join
-        # and we only need to merge.
-        # if we have an array of lookup indices, we 'take' those
-        right_dataset = right_dataset.take(lookup, masked=any(lookup_masked))
-    dataset = left.dataset.merged(right_dataset)
+    if right_columns:
+        right_dataset = right.dataset.project(*right_columns)
+        if lookup is not None:
+            # if lookup is None, we do a row based join
+            # and we only need to merge.
+            # if we have an array of lookup indices, we 'take' those
+            right_dataset = right_dataset.take(lookup, masked=any(lookup_masked))
+        dataset = left.dataset.merged(right_dataset)
+    else:
+        dataset = left.dataset
     # row number etc should not have changed, we only append new columns
     # so no need to reset caches
     left._dataset = DatasetJoin(dataset, left_original, right_original,


Test patch:
diff --git a/tests/join_test.py b/tests/join_test.py
--- a/tests/join_test.py
+++ b/tests/join_test.py
@@ -384,3 +384,11 @@ def transform(x):
     df2["y2"] = df2["y"].apply(transform, multiprocessing=False)
     joined = df1.join(df2, left_on="x2", right_on="y2", allow_duplication=True)
     assert joined.y2.tolist() == [2, 99]
+
+
+def test_join_no_right_columns_left():
+    df1 = vaex.from_arrays(a=[1, 2, 3])
+    # df2 only contains 'a', so we don't do the join for real
+    df2 = vaex.from_arrays(a=[1, 10])
+    df = df1.join(df2, on="a", how="inner")
+    assert df["a"].tolist() == [1]