Package: awswrangler
AWS SDK for pandas (awswrangler)AWS Data Wrangler is nowAWS SDK for pandas (awswrangler).  We’re changing the name we use when we talk about the library, but everything else will stay the same.  You’ll still be able to install usingpip install awswranglerand you won’t need to change any of your code.  As part of this change, we’ve moved the library from AWS Labs to the main AWS GitHub organisation but, thanks to the GitHub’s redirect feature, you’ll still be able to access the project by its old URLs until you update your bookmarks.  Our documentation has also moved toaws-sdk-pandas.readthedocs.io, but old bookmarks will redirect to the new site.Pandas on AWSEasy integration with Athena, Glue, Redshift, Timestream, OpenSearch, Neptune, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).AnAWS Professional Serviceopen source initiative |aws-proserve-opensource@amazon.comSourceDownloadsInstallation CommandPyPipip install awswranglerCondaconda install -c conda-forge awswrangler⚠️Starting version 3.0, optional modules must be installed explicitly:➡️pip install 'awswrangler[redshift]'Powered ByTable of contentsQuick StartAt ScaleRead The DocsGetting HelpCommunity ResourcesLoggingWho uses AWS SDK for pandas?Quick StartInstallation command:pip install awswrangler⚠️Starting version 3.0, optional modules must be installed explicitly:➡️pip install 'awswrangler[redshift]'importawswrangleraswrimportpandasaspdfromdatetimeimportdatetimedf=pd.DataFrame({"id":[1,2],"value":["foo","boo"]})# Storing data on Data Lakewr.s3.to_parquet(df=df,path="s3://bucket/dataset/",dataset=True,database="my_db",table="my_table")# Retrieving the data directly from Amazon S3df=wr.s3.read_parquet("s3://bucket/dataset/",dataset=True)# Retrieving the data from Amazon Athenadf=wr.athena.read_sql_query("SELECT * FROM my_table",database="my_db")# Get a Redshift connection from Glue Catalog and retrieving data from Redshift Spectrumcon=wr.redshift.connect("my-glue-connection")df=wr.redshift.read_sql_query("SELECT * FROM external_schema.my_table",con=con)con.close()# Amazon Timestream Writedf=pd.DataFrame({"time":[datetime.now(),datetime.now()],"my_dimension":["foo","boo"],"measure":[1.0,1.1],})rejected_records=wr.timestream.write(df,database="sampleDB",table="sampleTable",time_col="time",measure_col="measure",dimensions_cols=["my_dimension"],)# Amazon Timestream Querywr.timestream.query("""SELECT time, measure_value::double, my_dimensionFROM "sampleDB"."sampleTable" ORDER BY time DESC LIMIT 3""")At scaleAWS SDK for pandas can also run your workflows at scale by leveragingModinandRay. Both projects aim to speed up data workloads by distributing processing over a cluster of workers.The quickest way to get started is to use AWS Glue with Ray. Read ourdocs, our blogs (1/2), or head to our latesttutorialsto discover even more features.⚠️Ray is currently not available for Python 3.12. While AWS SDK for pandas supports Python 3.12, it cannot be used at scale.Read The DocsWhat is AWS SDK for pandas?InstallPyPi (pip)CondaAWS Lambda LayerAWS Glue Python Shell JobsAWS Glue PySpark JobsAmazon SageMaker NotebookAmazon SageMaker Notebook LifecycleEMRFrom sourceAt scaleGetting StartedSupported APIsResourcesTutorials001 - Introduction002 - Sessions003 - Amazon S3004 - Parquet Datasets005 - Glue Catalog006 - Amazon Athena007 - Databases (Redshift, MySQL, PostgreSQL, SQL Server and Oracle)008 - Redshift - Copy & Unload.ipynb009 - Redshift - Append, Overwrite and Upsert010 - Parquet Crawler011 - CSV Datasets012 - CSV Crawler013 - Merging Datasets on S3014 - Schema Evolution015 - EMR016 - EMR & Docker017 - Partition Projection018 - QuickSight019 - Athena Cache020 - Spark Table Interoperability021 - Global Configurations022 - Writing Partitions Concurrently023 - Flexible Partitions Filter024 - Athena Query Metadata025 - Redshift - Loading Parquet files with Spectrum026 - Amazon Timestream027 - Amazon Timestream 2028 - Amazon DynamoDB029 - S3 Select030 - Data Api031 - OpenSearch032 - Lake Formation Governed Tables033 - Amazon Neptune034 - Distributing Calls Using Ray035 - Distributing Calls on Ray Remote Cluster036 - Distributing Calls with Glue Interactive Sessions on Ray037 - Glue Data Quality038 - OpenSearch Serverless039 - Athena Iceberg040 - EMR Serverless041 - Apache Spark on Amazon AthenaAPI ReferenceAmazon S3AWS Glue CatalogAmazon AthenaAWS Lake FormationAmazon RedshiftPostgreSQLMySQLSQL ServerOracleData API RedshiftData API RDSOpenSearchAWS Glue Data QualityAmazon NeptuneDynamoDBAmazon TimestreamAmazon EMRAmazon CloudWatch LogsAmazon ChimeAmazon QuickSightAWS STSAWS Secrets ManagerGlobal ConfigurationsDistributed - RayLicenseContributingGetting HelpThe best way to interact with our team is through GitHub. You can open anissueand choose from one of our templates for bug reports, feature requests...
You may also find help on these community resources:The #aws-sdk-pandas SlackchannelAsk a question onStack Overflowand tag it withawswranglerRunbookfor AWS SDK for pandas with RayCommunity ResourcesPleasesend a Pull Requestwith your resource reference and @githubhandle.YouTube channel[@AdrianoNicolucci]Optimize Python ETL by extending Pandas with AWS SDK for pandas[@igorborgest]Reading Parquet Files With AWS Lambda[@anand086]Transform AWS CloudTrail data using AWS SDK for pandas[@anand086]Rename Glue Tables using AWS SDK for pandas[@anand086]Getting started on AWS SDK for pandas and Athena[@dheerajsharma21]Simplifying Pandas integration with AWS data related services[@bvsubhash]Build an ETL pipeline using AWS S3, Glue and Athena[@taupirho]LoggingEnabling internal logging examples:importlogginglogging.basicConfig(level=logging.INFO,format="[%(name)s][%(funcName)s]%(message)s")logging.getLogger("awswrangler").setLevel(logging.DEBUG)logging.getLogger("botocore.credentials").setLevel(logging.CRITICAL)Into AWS lambda:importlogginglogging.getLogger("awswrangler").setLevel(logging.DEBUG)Who uses AWS SDK for pandas?Knowing which companies are using this library is important to help prioritize the project internally.
If you would like us to include your company’s name and/or logo in the README file to indicate that your company is using the AWS SDK for pandas, please raise a "Support Us" issue. If you would like us to display your company’s logo, please raise a linked pull request to provide an image file for the logo. Note that by raising a Support Us issue (and related pull request), you are granting AWS permission to use your company’s name (and logo) for the limited purpose described here and you are confirming that you have authority to grant such permission.AmazonAWSCepsa[@alvaropc]Cognitivo[@msantino]Digio[@afonsomy]DNX[@DNXLabs]Fortescue Future Industries[@spencervoorend]Funcional Health Tech[@webysther]Funding Circle[@pfig]InfomachInforma Markets[@mateusmorato]LINE TV[@bryanyang0528]LogicalCube[@zolabud]Magnataur[@brianmingus2]M4U[@Thiago-Dantas]NBCUniversal[@vibe]nrd.io[@mrtns]OKRA Technologies[@JPFrancoia,@schot]Pier[@flaviomax]Pismo[@msantino]ringDNA[@msropp]Serasa Experian[@andre-marcos-perez]Shipwell[@zacharycarter]strongDM[@mrtns]Thinkbumblebee[@dheerajsharma21]VTEX[@igorborgest]Zillow[@nicholas-miles]