Package: h10awswrnglr
AWS SDK for pandas (awswrangler)AWS Data Wrangler is nowAWS SDK for pandas (awswrangler).  We’re changing the name we use when we talk about the library, but everything else will stay the same.  You’ll still be able to install usingpip install awswranglerand you won’t need to change any of your code.  As part of this change, we’ve moved the library from AWS Labs to the main AWS GitHub organisation but, thanks to the GitHub’s redirect feature, you’ll still be able to access the project by its old URLs until you update your bookmarks.  Our documentation has also moved toaws-sdk-pandas.readthedocs.io, but old bookmarks will redirect to the new site.Pandas on AWSEasy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).AnAWS Professional Serviceopen source initiative |aws-proserve-opensource@amazon.comSourceDownloadsInstallation CommandPyPipip install awswranglerCondaconda install -c conda-forge awswrangler⚠️For platforms without PyArrow 3 support (e.g.EMR,Glue PySpark Job, MWAA):➡️pip install pyarrow==2 awswranglerPowered ByTable of contentsQuick StartRead The DocsGetting HelpCommunity ResourcesLoggingWho uses AWS SDK for pandas?Quick StartInstallation command:pip install awswrangler⚠️For platforms without PyArrow 3 support (e.g.EMR,Glue PySpark Job, MWAA):➡️pip install pyarrow==2 awswranglerimportawswrangleraswrimportpandasaspdfromdatetimeimportdatetimedf=pd.DataFrame({"id":[1,2],"value":["foo","boo"]})# Storing data on Data Lakewr.s3.to_parquet(df=df,path="s3://bucket/dataset/",dataset=True,database="my_db",table="my_table")# Retrieving the data directly from Amazon S3df=wr.s3.read_parquet("s3://bucket/dataset/",dataset=True)# Retrieving the data from Amazon Athenadf=wr.athena.read_sql_query("SELECT * FROM my_table",database="my_db")# Get a Redshift connection from Glue Catalog and retrieving data from Redshift Spectrumcon=wr.redshift.connect("my-glue-connection")df=wr.redshift.read_sql_query("SELECT * FROM external_schema.my_table",con=con)con.close()# Amazon Timestream Writedf=pd.DataFrame({"time":[datetime.now(),datetime.now()],"my_dimension":["foo","boo"],"measure":[1.0,1.1],})rejected_records=wr.timestream.write(df,database="sampleDB",table="sampleTable",time_col="time",measure_col="measure",dimensions_cols=["my_dimension"],)# Amazon Timestream Querywr.timestream.query("""SELECT time, measure_value::double, my_dimensionFROM "sampleDB"."sampleTable" ORDER BY time DESC LIMIT 3""")Read The DocsWhat is AWS SDK for pandas?InstallPyPi (pip)CondaAWS Lambda LayerAWS Glue Python Shell JobsAWS Glue PySpark JobsAmazon SageMaker NotebookAmazon SageMaker Notebook LifecycleEMRFrom sourceTutorials001 - Introduction002 - Sessions003 - Amazon S3004 - Parquet Datasets005 - Glue Catalog006 - Amazon Athena007 - Databases (Redshift, MySQL, PostgreSQL, SQL Server and Oracle)008 - Redshift - Copy & Unload.ipynb009 - Redshift - Append, Overwrite and Upsert010 - Parquet Crawler011 - CSV Datasets012 - CSV Crawler013 - Merging Datasets on S3014 - Schema Evolution015 - EMR016 - EMR & Docker017 - Partition Projection018 - QuickSight019 - Athena Cache020 - Spark Table Interoperability021 - Global Configurations022 - Writing Partitions Concurrently023 - Flexible Partitions Filter024 - Athena Query Metadata025 - Redshift - Loading Parquet files with Spectrum026 - Amazon Timestream027 - Amazon Timestream 2028 - Amazon DynamoDB029 - S3 Select030 - Data Api031 - OpenSearch032 - Lake Formation Governed Tables033 - Amazon Neptune034 - Glue Data QualityAPI ReferenceAmazon S3AWS Glue CatalogAmazon AthenaAWS Lake FormationAmazon RedshiftPostgreSQLMySQLSQL ServerOracleData API RedshiftData API RDSOpenSearchAmazon NeptuneDynamoDBAmazon TimestreamAmazon EMRAmazon CloudWatch LogsAmazon ChimeAmazon QuickSightAWS STSAWS Secrets ManagerGlobal ConfigurationsLicenseContributingGetting HelpThe best way to interact with our team is through GitHub. You can open anissueand choose from one of our templates for bug reports, feature requests...
You may also find help on these community resources:The #aws-sdk-pandas SlackchannelAsk a question onStack Overflowand tag it withawswranglerCommunity ResourcesPleasesend a Pull Requestwith your resource reference and @githubhandle.Optimize Python ETL by extending Pandas with AWS SDK for pandas[@igorborgest]Reading Parquet Files With AWS Lambda[@anand086]Transform AWS CloudTrail data using AWS SDK for pandas[@anand086]Rename Glue Tables using AWS SDK for pandas[@anand086]Getting started on AWS SDK for pandas and Athena[@dheerajsharma21]Simplifying Pandas integration with AWS data related services[@bvsubhash]Build an ETL pipeline using AWS S3, Glue and Athena[@taupirho]LoggingEnabling internal logging examples:importlogginglogging.basicConfig(level=logging.INFO,format="[%(name)s][%(funcName)s]%(message)s")logging.getLogger("awswrangler").setLevel(logging.DEBUG)logging.getLogger("botocore.credentials").setLevel(logging.CRITICAL)Into AWS lambda:importlogginglogging.getLogger("awswrangler").setLevel(logging.DEBUG)Who uses AWS SDK for pandas?Knowing which companies are using this library is important to help prioritize the project internally.
If you would like us to include your company’s name and/or logo in the README file to indicate that your company is using the AWS SDK for pandas, please raise a "Support Us" issue. If you would like us to display your company’s logo, please raise a linked pull request to provide an image file for the logo. Note that by raising a Support Us issue (and related pull request), you are granting AWS permission to use your company’s name (and logo) for the limited purpose described here and you are confirming that you have authority to grant such permission.AmazonAWSCepsa[@alvaropc]Cognitivo[@msantino]Digio[@afonsomy]DNX[@DNXLabs]Fortescue Future Industries[@spencervoorend]Funcional Health Tech[@webysther]Funding Circle[@pfig]InfomachInforma Markets[@mateusmorato]LINE TV[@bryanyang0528]Magnataur[@brianmingus2]M4U[@Thiago-Dantas]NBCUniversal[@vibe]nrd.io[@mrtns]OKRA Technologies[@JPFrancoia,@schot]Pier[@flaviomax]Pismo[@msantino]ringDNA[@msropp]Serasa Experian[@andre-marcos-perez]Shipwell[@zacharycarter]strongDM[@mrtns]Thinkbumblebee[@dheerajsharma21]VTEX[@igorborgest]Zillow[@nicholas-miles]