Repository: vaexio/vaex

Issue:
Allow lazy file opening for parquet formatted files
When I run `open` on an HDF5 file, the operation is done lazily – I believe that file handlers are opened but nothing is loaded into RAM. However running `open` on a parquet file results in eager evaluation, where data is immediately loaded into RAM. (Same goes for `open_many`, FWIW). This means that it's impossible to use vaex on large-than-RAM datasets if they're stored as parquet.

I've put a minimal working example below, although note it'll use like 3-6GB of RAM and dump like 5 GB to your HD:

```python
import numpy as np
import pandas as pd
import time
import vaex

# Creates a roughly 2.4 GB dataframe
pandas_df = pd.DataFrame({
    "numeric_1": np.random.rand(100000000),
    "numeric_2": np.random.rand(100000000),
    "numeric_3": np.random.rand(100000000)
})

# Saves to parquet and vaex-formatted hdf5
pandas_df.to_parquet("temp_df.parquet", index=False)
vaex_df = vaex.from_pandas(pandas_df)
vaex_df.export_hdf5("temp_df.hdf5")

del vaex_df
del pandas_df

start = time.time()
vaex_df_hdf5 = vaex.open("temp_df.hdf5")
print(f"Took {time.time() - start:.2f} seconds to open the hdf5 file")

start = time.time()
vaex_df_parquet = vaex.open("temp_df.parquet")
print(f"Took {time.time() - start:.2f} seconds to open the parquet file")
```

Output on my machine:
```
Took 0.01 seconds to open the hdf5 file
Took 7.89 seconds to open the parquet file
```

(You can also see that RAM usage is up, although printing that to screen from python is trickier 😄.)

I have no idea if it's possible, but it would be really awesome to get the same lazy opening for parquet files – A fair amount of the files I interact with are stored as parquet, and converting to HDF5 would be time consuming and annoying. 


Discussion:
This has some overlap with #366, albeit with a much more focused use case. 
And [this](https://stackoverflow.com/a/56481636/1373664) makes me think this ask is flat out not possible 😞. 
Thanks for the code, I'll leave this open till I get a chance to look at it.

I wonder if there is a way to read the data that is needed and discard it afterwards. This means that using parquet would always read it from disk when accessing it. Vaex is designed to touch the data as little as possible, and do as little passes over the data as possible, so this may work out, what do you think?
Yeah, maybe, although I have no clue whether that would be anywhere near performant enough to be useful. It might push people to use the `delayed` decorator more, push the computation to happen all at once _a la_ dask. 
Do you have single parquet files, or do you have them in multiple files, like row groups?
I would like to +1 this issue as well. I'm trying to add a column to a parquet file inside a docker container, which has limited memory resources. Pandas was having problems because it loads the whole thing in memory, so I thought I'd try vaex. I couldn't figure out why I couldn't even open the parquet file until I found this ticket.

In my case, I'm reading in a single parquet file, but it's about 1 GB.
How did you create the file, and does it have multiple row groups?

(from mobile phone)

On Tue, 11 Feb 2020, 22:11 Christine Smit, <notifications@github.com> wrote:

> I would like to +1 this issue as well. I'm trying to add a column to a
> parquet file inside a docker container, which has limited memory resources.
> Pandas was having problems because it loads the whole thing in memory, so I
> thought I'd try vaex. I couldn't figure out why I couldn't even open the
> parquet file until I found this ticket.
>
> In my case, I'm reading in a single parquet file, but it's about 1 GB.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/vaexio/vaex/issues/477?email_source=notifications&email_token=AANPEPLG44J2QYSHKP7THELRCMH6VA5CNFSM4JRZ6CZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELOCUBA#issuecomment-584854020>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AANPEPLAJQY2BH4KN4DUEFDRCMH6VANCNFSM4JRZ6CZQ>
> .
>

I just talked with the person who created the file. He says he used `fastparquet` and was not in a docker container. Opening the file outside the docker container, it appears to have 30 row groups:

```
Python 3.8.1 | packaged by conda-forge | (default, Jan 29 2020, 15:06:10) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.12.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import fastparquet as fp                                                                                                                                                             

In [2]: large = fp.ParquetFile("large.parquet")                                                                                                                                              

In [3]: len(large.row_groups)                                                                                                                                                                
Out[3]: 30
```

@maartenbreddels a mix of single files and multiple ones partitioned on rows, but probably more of the latter type. 
@maartenbreddels  Wondering about the status of lazy loading parquet files? If it is possible, I think it will make Vaex very usable to many industry scenarios where we are working with Hive tables and parquet files on HDFS. 

Also, a big thank you for your work on Vaex, I am excited to try it out on our use case (feature generation + lightgbm training on datasets running into several hundred million rows) :) 
I would say low priority right now because we have no client that actually needs it, but there are things in the pipeline that bring it closer for sure!
Thanks for clarifying! Do you have any suggestions then on the best way to read data that is stored as parquet into Vaex? Is there a way to convert the data into HDF5 first? Or do you have any other suggestions?
Does the parquet file not fit into RAM? Otherwise you can open them one by one with vaex, write them to hdf5 or arrow, and then concat all the dataframes.
Hmm, I can get a significant chunk of the parquet files into RAM, but not all of them together. I will try reading them in chunks and producing hdf5 from them.
https://docs.vaex.io/en/latest/example_io.html might be good to take a look at.

Resolving patch:
diff --git a/packages/vaex-core/vaex/arrow/dataset.py b/packages/vaex-core/vaex/arrow/dataset.py
--- a/packages/vaex-core/vaex/arrow/dataset.py
+++ b/packages/vaex-core/vaex/arrow/dataset.py
@@ -30,8 +30,9 @@ def get_main_io_pool():
 
 
 class DatasetArrow(vaex.dataset.Dataset):
-    def __init__(self, ds):
+    def __init__(self, ds, max_rows_read=1024**2*10):
         super().__init__()
+        self.max_rows_read = max_rows_read
         self._arrow_ds = ds
         row_count = 0
         for fragment in self._arrow_ds.get_fragments():
@@ -70,18 +71,15 @@ def close(self):
         pass
 
     def _chunk_producer(self, columns, chunk_size=None, reverse=False, start=0, end=None):
+        import pyarrow.parquet
         pool = get_main_io_pool()
         offset = 0
         for fragment_large in self._arrow_ds.get_fragments():
             fragment_large_rows = sum([rg.num_rows for rg in fragment_large.row_groups])
-            # when do we want to split up? File size? max chunk size?
-            # if fragment_large_rows > chunk_size:
-            #     fragments = fragment_large.split_by_row_group()
-            # else:
-            #     # or not
-            #     fragments = [fragment_large]
-            import pyarrow.parquet
             fragments = [fragment_large]
+            # when do we want to split up? File size? max chunk size?
+            if fragment_large_rows > self.max_rows_read:
+                fragments = fragment_large.split_by_row_group()
             for fragment in fragments:
                 rows = sum([rg.num_rows for rg in fragment.row_groups])
                 chunk_start = offset


Test patch: