Package: pandasglue
DEPRECATION: This project was only created for a Proof of Concept purpose. The production ready version of this project received the name ofAWS Data Wrangler(pip install awswrangler).Please consider move forward to:https://pypi.org/project/awswrangler/https://github.com/awslabs/aws-data-wranglerPandasGlueA Python library for creating lite ETLs with the widely used Pandas library and the power of AWS Glue Catalog.WithPandasGLueyou will be able to write/read to/from an AWS Data Lake with one single line of code. With its minimalist naturePandasGLuehas an interface with only 2 functions:functionFromTowrite_glue()Pandas DataFrameAWS Glue Tableread_glue()AWS GlueTablePandas DataFrameOnce your data is mapped toAWS Glue Catalogit will be accessible to many other tools likeAWS Redshift Spectrum,AWS Athena,AWS Glue Jobs,AWS EMR(Spark,Hive,PrestoDB), etc.Amazon Glueis anAWSsimple, flexible, and cost-effective ETL service andPandasis a Python library which provides high-performance, easy-to-use data structures and data analysis tools.The goal of this package is help data engineers in the usage of cost efficient serverless compute services (Lambda,Glue,Athena) in order to provide an easy way to integrate Pandas with  AWS Glue,  allowing load (appending, overwriting or only overwriting the partitions with data) the content of a DataFrame (Write function) directly in a table (parquet/csv format) in theGlue Data Catalogand also execute Athena queries (Read function) returning the result directly in a Pandas DataFrame.Use casesThis package is recommended for ETL purposes which loads and transforms small to medium size datasets without requiring to create Spark jobs, helping reduce infrastructure costs.It could be used withinLambda functions,Glue scripts,EC2instances or any other infrastucture resources.PrerequisitesPython 2.7, 3.6, 3.7pip install pandas
pip install boto3
pip install pyarrowInstalling the package...pip install pandasglueOr you can download direct the artifacts forAWS Lambda Layer/AWS Glue Jobfrom ourrelease page. Then you only will need to upload it in your AWS account.UsageRead method:read_glue()To retrieve the result of an Athena Query in a Pandas DataFrame.Quick example:importpandasaspdimportpandasglueaspg#Parameterssql_query="SELECT * FROM table_name LIMIT 20"db_name="DB_NAME"s3_output_bucket="s3://bucket-url/"df=pg.read_glue(sql_query,db_name,s3_output_bucket)print(df)Parameters list:query:the SQL statement on Athenadb:database name.s3_output:path of the S3 output folder (optional)region:id of the AWS region, e.g: us-west-1 (optional)key:AWS Access key (optional)secret:AWS secret key (optional)profile_name:AWS IAM profile (optional)Write method:write_glue()Convert a given Pandas Dataframe to a Glue Parquet table.Quick example:importpandasaspdimportpandasglueaspg#Parametersdatabase="DB_NAME"table_name="TB_NAME"s3_path="s3://bucket-url/"#Sample DFsource_data={'name':['Sarah','Renata','Erika','Fernanda','Diana'],'city':['Seattle','Sao Paulo','Seattle','Santiago','Lima'],'test_score':[82,52,56,234,254]}df=pd.DataFrame(source_data,columns=['name','city','test_score'])pg.write_glue(df,database,table_name,s3_path,partition_cols=['city'])Parameters list:df:variable containing the Pandas DataFramedatabase:database name.path:Path of the target S3 buckettable:table name (optional)partition_cols:list of columns to partition (optional)preserve_index:boolean, if you want to preserve the index on the table (optional)file_format:parquet|csv(optional)mode:append|overwrite|overwrite_partitions(optional)region:ID of the AWS region (optional)key:AWS Access key (optional)secret:AWS secret key (optional)profile_name:AWS IAM profile (optional)Built WithBoto3- (AWS) SDK for Python, which allows Python developers to write software that makes use of Amazon services like S3 and EC2.PyArrow- Python package to interoperate Arrow with Python allowing to convert text files format to parquet files among other functions.ContributingPlease readCONTRIBUTING.mdfor details on our code of conduct, and the process for submitting pull requests to us.AuthorsIgor Tavares-Profile linkRicardo Serafim-Profile linkAndres Palacios-Profile linkSee also the list ofcontributorswho participated in this project.LicenseThis project is licensed under the MIT License - see theLICENSE.mdfile for details