Package: redshift-connector
redshift_connectoris the Amazon Redshift connector for
Python. Easy integration withpandasandnumpy, as well as support for numerous Amazon Redshift specific features help you get the most out of your dataSupported Amazon Redshift features include:IAM authenticationIdentity provider (IdP) authenticationRedshift specific data typesThis pure Python connector implementsPython Database API Specification 2.0.Getting StartedInstall from BinaryPackage ManagerDownloadsInstallation CommandPyPipip install redshift_connectorCondaconda install-cconda-forgeredshift_connectorInstall from SourceYou may install from source by cloning this repository.$gitclonehttps://github.com/aws/amazon-redshift-python-driver.git$cdredshift_connector$pipinstall.Tutorials001 - Connecting to Amazon Redshift002 - Data Science Library Integrations003 - Amazon Redshift Feature Support004 - Amazon Redshift DatatypesWe are working to add more documentation and would love your feedback. Please reach out to the team byopening an issueorstarting a discussionto help us fill in the gaps in our documentation.Integrationsredshift_connectorintegrates with various open source projects to provide an interface to Amazon Redshift. Pleaseopen an issuewith our project to request new integrations or get support for aredshift_connectorissue seen in an existing integration.apache-airflowquerybooksqlalchemy-redshiftBasic Exampleimportredshift_connector# Connects to Redshift cluster using AWS credentialsconn=redshift_connector.connect(host='examplecluster.abc123xyz789.us-west-1.redshift.amazonaws.com',database='dev',user='awsuser',password='my_password')cursor:redshift_connector.Cursor=conn.cursor()cursor.execute("create Temp table book(bookname varchar,author varchar)")cursor.executemany("insert into book (bookname, author) values (%s,%s)",[('One Hundred Years of Solitude','Gabriel García Márquez'),('A Brief History of Time','Stephen Hawking')])cursor.execute("select * from book")result:tuple=cursor.fetchall()print(result)>>(['One Hundred Years of Solitude','Gabriel García Márquez'],['A Brief History of Time','Stephen Hawking'])Enabling autocommitFollowing the DB-API specification, autocommit is off by default. It can be turned on by using the autocommit property of the connection.# Make sure we're not in a transactionconn.rollback()conn.autocommit=Trueconn.run("VACUUM")conn.autocommit=FalseConfiguring paramstyleParamstyle can be set on both a module and cursor level. When paramstyle is set on a module level e.g.redshift_connector.paramstyle = 'qmark', the user specified paramstyle is used for all subsequent cursors unless set on the cursor.
When paramstyle is set on the cursor e.g.`cursor.paramstyle = 'qmark'the user specified paramstyle is only used for that cursor object.# setting paramstyle to qmark on a module levelredshift_connector.paramstyle='qmark'withredshift_connector.connect()asconn1:withconn1.cursor()ascursor1:# this cursor will use qmark paramstyle as it's been set on the module levelpasswithconn1.cursor()ascursor2:# setting paramstyle to numeric on the cursor level only this cursor will use numeric paramstylecursor.paramstyle='numeric'withconn1.cursor()ascursor3:# this cursor will use qmark paramstyle as it's been set on the module levelpasswithredshift_connector.connect()asconn2:withconn2.cursor()ascursor1:# this cursor will use qmark paramstyle as it's been set on the module levelpassThe module level default paramstyle used isformat. Valid values forparamstyleincludeqmark, numeric, named, format, pyformat. The below example shows how to use various paramstyles after the paramstyle is set on the cursor.When paramstyle is set tonamedorpyformat, parameters must be passed as a Python dictionary to theexecute()method. Other paramstyles require parameters to be passed as a Python tuple or list.# qmarkcursor.paramstyle='qmark'sql='insert into foo(bar, jar) VALUES(?, ?)'cursor.execute(sql,(1,"hello world"))# numericcursor.paramstyle='numeric'sql='insert into foo(bar, jar) VALUES(:1, :2)'cursor.execute(sql,(1,"hello world"))# namedcursor.paramstyle='named'sql='insert into foo(bar, jar) VALUES(:p1, :p2)'cursor.execute(sql,{"p1":1,"p2":"hello world"})# formatcursor.paramstyle='format'sql='insert into foo(bar, jar) VALUES(%s,%s)'cursor.execute(sql,(1,"hello world"))# pyformatcursor.paramstyle='pyformat'sql='insert into foo(bar, jar) VALUES(%(bar)s,%(jar)s)'cursor.execute(sql,{"bar":1,"jar":"hello world"})Exception Handlingredshift_connectoruses the guideline for exception handling specified in thePython DB-API. For exception definitions, please seeredshift_connector/error.pyExample using IAM CredentialsIAM Credentials can be supplied directly toconnect(...)using an AWS profile as shown below:importredshift_connector# Connects to Redshift cluster using IAM credentials from default profile defined in ~/.aws/credentialsconn=redshift_connector.connect(iam=True,database='dev',db_user='awsuser',password='',user='',cluster_identifier='examplecluster',profile='default')# ~/.aws/credentials[default]aws_access_key_id="my_aws_access_key_id"aws_secret_access_key="my_aws_secret_access_key"aws_session_token="my_aws_session_token"# ~/.aws/config[default]region=us-west-2If a region is not provided in~/.aws/configor you would like to override its value,regionmay be passed toconnect(...).Alternatively, IAM credentials can be supplied directly toconnect(...)using AWS credentials as shown below:importredshift_connector# Connects to Redshift cluster using IAM credentials from default profile defined in ~/.aws/credentialsconn=redshift_connector.connect(iam=True,database='dev',db_user='awsuser',password='',user='',cluster_identifier='examplecluster',access_key_id="my_aws_access_key_id",secret_access_key="my_aws_secret_access_key",session_token="my_aws_session_token",region="us-east-2")Integration with pandasRetrieving query results as apandas.DataFrameimportpandascursor.execute("create Temp table book(bookname varchar,author varchar)")cursor.executemany("insert into book (bookname, author) values (%s,%s)",[('One Hundred Years of Solitude','Gabriel García Márquez'),('A Brief History of Time','Stephen Hawking')])cursor.execute("select * from book")result:pandas.DataFrame=cursor.fetch_dataframe()print(result)>>booknameauthor>>0OneHundredYearsofSolitudeGabrielGarcíaMárquez>>1ABriefHistoryofTimeStephenHawkingInsert data stored in apandas.DataFrameinto an Amazon Redshift tableimportnumpyasnpimportpandasaspddf=pd.DataFrame(np.array([["One Hundred Years of Solitude","Gabriel García Márquez"],["A Brief History of Time","Stephen Hawking"],]),columns=["bookname","author‎"],)withconn.cursor()ascursor:cursor.write_dataframe(df,"book")cursor.execute("select * from book; ")result=cursor.fetchall()Integration with numpyimportnumpycursor.execute("select * from book")result:numpy.ndarray=cursor.fetch_numpy_array()print(result)>>[['One Hundred Years of Solitude''Gabriel García Márquez']>>['A Brief History of Time''Stephen Hawking']]Query using functionscursor.execute("SELECT CURRENT_TIMESTAMP")print(cursor.fetchone())>>[datetime.datetime(2020,10,26,23,3,54,756497,tzinfo=datetime.timezone.utc)]Connection ParametersNameTypeDescriptionDefault ValueRequiredaccess_key_idstrThe access key for the IAM role or IAM user configured for IAM database authenticationNoneNoallow_db_user_overrideboolTrue specifies the driver uses the DbUser value from the SAML assertion while False indicates the value in the DbUser connection parameter is usedFALSENoapp_namestrThe name of the IdP application used for authenticationNoneNoauth_profilestrThe name of an Amazon Redshift Authentication profile having connection properties as JSON. See the RedshiftProperty class to learn how connection parameters should be named.NoneNoauto_createboolIndicates whether the user should be created if they do not existFALSENoclient_idstrThe client id from Azure IdPNoneNoclient_secretstrThe client secret from Azure IdPNoneNocluster_identifierstrThe cluster identifier of the Amazon Redshift ClusterNoneNocredentials_providerstrThe IdP that will be used for authenticating with Amazon Redshift.NoneNodatabasestrThe name of the database to connect toNoneNodatabase_metadata_current_db_onlyboolIndicates if application supports multi-database datashare catalogs. Default value of  True indicates application does not support multi-database datashare catalogs for backwards compatibilityTRUENodb_groupslistA comma-separated list of existing database group names that the DbUser joins for the current sessionNoneNodb_userstrThe user ID to use with Amazon RedshiftNoneNoendpoint_urlstrThe Amazon Redshift endpoint url. This option is only used by AWS internal teams.NoneNogroup_federationboolUse the IdP Groups in the Redshift. Default value False.FalseNohoststrThe hostname of Amazon Redshift clusterNoneNoiamboolIf IAM Authentication is enabledFALSENoiam_disable_cacheboolThis option specifies whether the IAM credentials are cached. By default the IAM credentials are cached. This improves performance when requests to the API gateway are throttled.FALSENoidentity_namespacestrThe identity namespace to be used for the IdC browser auth plugin and IdP token auth plugin. It is an optional value if there is only one IdC instance existing or if default identity namespace is set on the cluster - else it is required.NoneNoidp_response_timeoutintThe timeout for retrieving SAML assertion from IdP120Noidp_tenantstrThe IdP tenantNoneNolisten_portintThe listen port IdP will send the SAML assertion to7890Nologin_to_rpstrOnly for AdfsCredentialsProvider. Used to specify the loginToRp when performing IdpInitiatedSignOn as apart of form based authentication.urn:amazon:webservicesNologin_urlstrThe SSO Url for the IdPNoneNomax_prepared_statementsintThe maximum number of prepared statements that can be open at once1000Nonumeric_to_floatboolSpecifies if NUMERIC datatype values will be converted from decimal.Decimal to float. By default NUMERIC values are received as decimal.Decimal. Enabling this option is not recommended for use cases which prefer the most precision as results may be rounded. Please reference the Python docs on decimal.Decimal to see the tradeoffs between decimal.Decimal and float before enabling this option.FalseNopartner_sp_idstrThe Partner SP Id used for authentication with PingNoneNopasswordstrThe password to use for authenticationNoneNoportIntThe port number of the Amazon Redshift cluster5439Nopreferred_rolestrThe IAM role preferred for the current connectionNoneNoprincipal_arnstrThe ARN of the IAM entity (user or role) for which you are generating a policyNoneNoprofilestrThe name of a profile in a AWS credentials file that contains AWS credentials.NoneNoprovider_namestrThe name of the Redshift Native Auth Provider.NoneNoregionstrThe AWS region where the cluster is locatedNoneNorole_arnstrThe Amazon Resource Name (ARN) of the role that the caller is assuming. This parameter is used by JwtCredentialsProvider. For this provider, this is a mandatory parameter.NoneNorole_session_namestrAn identifier for the assumed role session. Typically, you pass the name or identifier that is associated with the user who is using your application. That way, the temporary security credentials that your application will use are associated with that user. This parameter is used by JwtCredentialsProvider. For this provider, this is an optional parameter.jwt_redshift_sessionNoscopestrScope for BrowserAzureOauth2CredentialsProvider authentication.“”Nosecret_access_key_idstrThe secret access key for the IAM role or IAM user configured for IAM database authenticationNoneNoserverless_acct_idstrThe account ID of the serverless. Default value NoneNoneNoserverless_work_groupstrThe name of work group for serverless end point. Default value None.NoneNosession_tokenstrThe access key for the IAM role or IAM user configured for IAM database authentication. Not required unless temporary AWS credentials are being used.NoneNosslboolIf SSL is enabledTRUENossl_insecureboolSpecifies if IDP hosts server certificate will be verifiedTRUENosslmodestrThe security of the connection to Amazon Redshift. verify-ca and verify-full are supported.verify_caNotimeoutintThe number of seconds before the connection to the server will timeout.NoneNotokenstrThe access token required for the IdP token auth plugin.NoneNotoken_typestrThe token type required for the IdP token auth plugin.ACCESS_TOKENNouserstrThe username to use for authenticationNoneNoweb_identity_tokenstrThe OAuth 2.0 access token or OpenID Connect ID token that is provided by the identity provider. Your application must get this token by authenticating the user who is using your application with a web identity provider. This parameter is used by JwtCredentialsProvider. For this provider, this is a mandatory parameter.NoneNoSupported Datatypesredshift_connectorsupports the following Amazon Redshift datatypes.redshift_connectorwill attempt to treat unsupported datatypes as strings.
Incoming data from Amazon Redshift is treated as follows:Amazon Redshift DatatypePython DatatypeACLITEMstrBOOLEANboolINT8intINT4intINT2intVARCHARstrOIDintREGPROCintXIDintFLOAT4floatFLOAT8floatTEXTstrCHARstrDATEdatetime.dateTIMEdatetime.timeTIMETZdatetime.timeTIMESTAMPdatetime.datetimeTIMESTAMPTZdatetime.datetimeNUMERICdecimal.DecimalGEOMETRYstrSUPERstrVARBYTEbytesGEOGRAPHYstrLoggingredshift_connectoruses logging for providing detailed error messages regarding IdP authentication. A do-nothing handler is enabled by default as to prevent logs from being output tosys.stderr.Enable logging in your application to view logs output byredshift_connectoras described in
thedocumentation for Python logging module.Client Transfer Protocolredshift_connectorrequests the Amazon Redshift server use the  highest transfer protocol version supported. As of v2.0.879 binary transfer protocol is requested by default. If necessary, the requested transfer protocol can be modified via theclient_protocol_versionparameter ofredshift_connector.connect(...). Please see the Connection Parameters table for more details.Getting HelpAsk a question onStack Overflowand tag it with redshift_connectorOpen a support ticket withAWS SupportIf you may have found a bug, pleaseopen an issueContributingWe look forward to collaborating with you! Please read throughCONTRIBUTINGbefore submitting any issues or pull requests.Changelog GenerationAn entry in the changelog is generated upon release usinggitchangelog. Please use the configuration file,.gitchangelog.rcwhen generating the changelog.Running TestsYou can run tests by usingpytest test/unit. This will run all unit tests. Integration tests require providing credentials for an Amazon Redshift cluster as well as IdP attributes intest/config.ini.Additional ResourcesLICENSEPython Database API Specification v2.0 (PEP 249)PostgreSQL Frontend/Backend Protocol