Package: parquet-loader
ParquetLoaderParquet file Load and Read from minio & S3 or Local
This repository help read parquet file, When you train model or Analysis using bigdata.1. Installation1.1. Install from pippipinstallparquet-loader1.2 Install from source codesgitclonehttps://github.com/Keunyoung-Jung/ParquetLoadercdParquetLoader
pipinstall-e.2. IntroduceParquetLoadercan help to read large parquet files.ParquetLoaderis built on base of pandas and fastparquet, which helps in situations where Spark clusters are not available.It proceeds to load data into memory based on chuck size.Then return it as a pandas dataframe.3. Quick Start3.1. Local PathIf your file is locatedlocal, you can load the data this way.fromParquetLoaderimportDataLoaderdl=DataLoader(folder='parquet_data'shuffle=False)fordfindl:print(df.head())3.2. S3 PathIf your file is locatedS3orMinio, you have to set
environment variable.exportAWS_ACCESS_KEY_ID=my-access-keyexportAWS_SECRET_ACCESS_KEY=my-secret-keyexportAWS_DEFAULT_REGION=ap-northeast-2When you have finished setting, you can load data this way.fromParquetLoaderimportS3Loadersl=S3Loader(bucket='mysterico-feature-store',folder="mongo-sentence-token-feature",depth=2)fordfinsl:print(df.head())4. ParametersParquetLoadercan control reading data using parameters.
The only difference betweenS3LoaderandDataLoaderis theroot_pathparameter.dl=DataLoader(chunk_size:int=100000,root_path:str='.',# S3Loader using "bucket"folder:str='data',shuffle:bool=True,random_seed:int=int((time()-int(time()))*100000),columns:list=None,depth:int=0,std_out:bool=True,filters:list=None)chunk_sizedefault : 100,000 rowThis parameter controls the number of rows loaded into memory when reading data.root_pathorbucketdefault : current pathThis parameter is used to specify the project path or datastore path.folderdefault : "data"This parameter specifies the actual folder in which the parquet is clustered.shuffledefault : TrueWhether to shuffle the data.random_seeddefault :int((time() - int(time()))*100000)You can fix the order of the shuffled data by giving it a random seed.columnsdefault : NoneWhen reading data, you can select columns.depthdefault : 0It is used when the parquet in the folder is partitioned and there is depth.std_outdefault : TrueYou can turn off output.filtersIt is used when you want get filtered dataframe, It must use 2 dim listexample :[[("column","==",10)]]4.1. Select Columnscolumnsparam is taken as a list.dl=DataLoader(folder='parquet_data',colums=['name','age','gender'])4.2. Setting depthUse if your parquet file is partitioned and depth exists.Example📦 data    
 ┣ 📦 Year=2020     
    ┣ 📜 part-0000-example.snappy.parquet   
    ┗ 📜 part-0001-example.snappy.parquet  
 ┣ 📦 Year=2021     
    ┣ 📜 part-0002-example.snappy.parquet   
    ┗ 📜 part-0003-example.snappy.parquetThe data path in this example has adepth 1.dl=DataLoader(folder='parquet_data',depth=1)5. Get MetadataDataLoaderObject can get metadata your parquetprint(data_loader.schema)# get data schemaprint(data_loader.columns)# get data columnsprint(data_loader.count)# get total countprint(data_loader.info)# get metadata infomation6. Customize S3 PathIf you use minio or other obejct storage,you will use s3 parametersdl=S3Loader(s3_endpoint_url:str='',s3_access_key:str='',s3_secret_key:str='',bucket:str='.',folder:str='data',)s3_endpoint_urlStorage endpoint url you want to useexample : "http://mino-service.kubeflow:9000"s3_access_keyands3_secret_keyyou can set s3_access_key and s3_secret_key, but I don't recommend using itit is recommended to use environment variables.7. Get Filtered DataframeIt is used when you want get filtered dataframe, It must use 2 dim list
It is built with a two-dimensional list construction. (Equal fastparquet filter)dl=S3Loader(bucket='test',folder='data',filters=[[[("col1",">",10)]]])The first list consists of an OR operation.# col > 10 or col2 in ["children","kids"]filters=[[("col1",">",10)],["col2","in",["children","kids"]]]The second list consists of an AND operation.# col > 10 and col2 == "male"filters=[[("col1",">",10),("col2","==","male")]]You can also mix the two to make a filter.# (col > 10 and col2 == "male") or col3 in ["children","kids"]filters=[[("col1",">",10),("col2","==","male")],["col3","in",["children","kids"]]]