Package: aws-cdk.aws-glue
AWS Glue Construct Library---AWS CDK v1 has reached End-of-Support on 2023-06-01.
This package is no longer being updated, and users should migrate to AWS CDK v2.For more information on how to migrate, see theMigrating to AWS CDK v2guide.This module is part of theAWS Cloud Development Kitproject.JobAJobencapsulates a script that connects to data sources, processes them, and then writes output to a data target.There are 3 types of jobs supported by AWS Glue: Spark ETL, Spark Streaming, and Python Shell jobs.Theglue.JobExecutableallows you to specify the type of job, the language to use and the code assets required by the job.glue.Codeallows you to refer to the different code assets required by the job, either from an existing S3 location or from a local file path.Spark JobsThese jobs run in an Apache Spark environment managed by AWS Glue.ETL JobsAn ETL job processes data in batches using Apache Spark.# bucket: s3.Bucketglue.Job(self,"ScalaSparkEtlJob",executable=glue.JobExecutable.scala_etl(glue_version=glue.GlueVersion.V2_0,script=glue.Code.from_bucket(bucket,"src/com/example/HelloWorld.scala"),class_name="com.example.HelloWorld",extra_jars=[glue.Code.from_bucket(bucket,"jars/HelloWorld.jar")]),description="an example Scala ETL job")Streaming JobsA Streaming job is similar to an ETL job, except that it performs ETL on data streams. It uses the Apache Spark Structured Streaming framework. Some Spark job features are not available to streaming ETL jobs.glue.Job(self,"PythonSparkStreamingJob",executable=glue.JobExecutable.python_streaming(glue_version=glue.GlueVersion.V2_0,python_version=glue.PythonVersion.THREE,script=glue.Code.from_asset(path.join(__dirname,"job-script/hello_world.py"))),description="an example Python Streaming job")Python Shell JobsA Python shell job runs Python scripts as a shell and supports a Python version that depends on the AWS Glue version you are using.
This can be used to schedule and run tasks that don't require an Apache Spark environment.# bucket: s3.Bucketglue.Job(self,"PythonShellJob",executable=glue.JobExecutable.python_shell(glue_version=glue.GlueVersion.V1_0,python_version=glue.PythonVersion.THREE,script=glue.Code.from_bucket(bucket,"script.py")),description="an example Python Shell job")Seedocumentationfor more information on adding jobs in Glue.ConnectionAConnectionallows Glue jobs, crawlers and development endpoints to access certain types of data stores. For example, to create a network connection to connect to a data source within a VPC:# security_group: ec2.SecurityGroup# subnet: ec2.Subnetglue.Connection(self,"MyConnection",type=glue.ConnectionType.NETWORK,# The security groups granting AWS Glue inbound access to the data source within the VPCsecurity_groups=[security_group],# The VPC subnet which contains the data sourcesubnet=subnet)If you need to use a connection type that doesn't exist as a static member onConnectionType, you can instantiate aConnectionTypeobject, e.g:new glue.ConnectionType('NEW_TYPE').SeeAdding a Connection to Your Data StoreandConnection Structuredocumentation for more information on the supported data stores and their configurations.SecurityConfigurationASecurityConfigurationis a set of security properties that can be used by AWS Glue to encrypt data at rest.glue.SecurityConfiguration(self,"MySecurityConfiguration",security_configuration_name="name",cloud_watch_encryption=glue.CloudWatchEncryption(mode=glue.CloudWatchEncryptionMode.KMS),job_bookmarks_encryption=glue.JobBookmarksEncryption(mode=glue.JobBookmarksEncryptionMode.CLIENT_SIDE_KMS),s3_encryption=glue.S3Encryption(mode=glue.S3EncryptionMode.KMS))By default, a shared KMS key is created for use with the encryption configurations that require one. You can also supply your own key for each encryption config, for example, for CloudWatch encryption:# key: kms.Keyglue.SecurityConfiguration(self,"MySecurityConfiguration",security_configuration_name="name",cloud_watch_encryption=glue.CloudWatchEncryption(mode=glue.CloudWatchEncryptionMode.KMS,kms_key=key))Seedocumentationfor more info for Glue encrypting data written by Crawlers, Jobs, and Development Endpoints.DatabaseADatabaseis a logical grouping ofTablesin the Glue Catalog.glue.Database(self,"MyDatabase",database_name="my_database")TableA Glue table describes a table of data in S3: its structure (column names and types), location of data (S3 objects with a common prefix in a S3 bucket), and format for the files (Json, Avro, Parquet, etc.):# my_database: glue.Databaseglue.Table(self,"MyTable",database=my_database,table_name="my_table",columns=[glue.Column(name="col1",type=glue.Schema.STRING),glue.Column(name="col2",type=glue.Schema.array(glue.Schema.STRING),comment="col2 is an array of strings")],data_format=glue.DataFormat.JSON)By default, a S3 bucket will be created to store the table's data but you can manually pass thebucketands3Prefix:# my_bucket: s3.Bucket# my_database: glue.Databaseglue.Table(self,"MyTable",bucket=my_bucket,s3_prefix="my-table/",# ...database=my_database,table_name="my_table",columns=[glue.Column(name="col1",type=glue.Schema.STRING)],data_format=glue.DataFormat.JSON)By default, an S3 bucket will be created to store the table's data and stored in the bucket root. You can also manually pass thebucketands3Prefix:Partition KeysTo improve query performance, a table can specifypartitionKeyson which data is stored and queried separately. For example, you might partition a table byyearandmonthto optimize queries based on a time window:# my_database: glue.Databaseglue.Table(self,"MyTable",database=my_database,table_name="my_table",columns=[glue.Column(name="col1",type=glue.Schema.STRING)],partition_keys=[glue.Column(name="year",type=glue.Schema.SMALL_INT),glue.Column(name="month",type=glue.Schema.SMALL_INT)],data_format=glue.DataFormat.JSON)Partition IndexesAnother way to improve query performance is to specify partition indexes. If no partition indexes are
present on the table, AWS Glue loads all partitions of the table and filters the loaded partitions using
the query expression. The query takes more time to run as the number of partitions increase. With an
index, the query will try to fetch a subset of the partitions instead of loading all partitions of the
table.The keys of a partition index must be a subset of the partition keys of the table. You can have a
maximum of 3 partition indexes per table. To specify a partition index, you can use thepartitionIndexesproperty:# my_database: glue.Databaseglue.Table(self,"MyTable",database=my_database,table_name="my_table",columns=[glue.Column(name="col1",type=glue.Schema.STRING)],partition_keys=[glue.Column(name="year",type=glue.Schema.SMALL_INT),glue.Column(name="month",type=glue.Schema.SMALL_INT)],partition_indexes=[glue.PartitionIndex(index_name="my-index",# optionalkey_names=["year"])],# supply up to 3 indexesdata_format=glue.DataFormat.JSON)Alternatively, you can call theaddPartitionIndex()function on a table:# my_table: glue.Tablemy_table.add_partition_index(index_name="my-index",key_names=["year"])EncryptionYou can enable encryption on a Table's data:Unencrypted- files are not encrypted. The default encryption setting.S3Managed- Server side encryption (SSE-S3) with an Amazon S3-managed key.# my_database: glue.Databaseglue.Table(self,"MyTable",encryption=glue.TableEncryption.S3_MANAGED,# ...database=my_database,table_name="my_table",columns=[glue.Column(name="col1",type=glue.Schema.STRING)],data_format=glue.DataFormat.JSON)Kms- Server-side encryption (SSE-KMS) with an AWS KMS Key managed by the account owner.# my_database: glue.Database# KMS key is created automaticallyglue.Table(self,"MyTable",encryption=glue.TableEncryption.KMS,# ...database=my_database,table_name="my_table",columns=[glue.Column(name="col1",type=glue.Schema.STRING)],data_format=glue.DataFormat.JSON)# with an explicit KMS keyglue.Table(self,"MyTable",encryption=glue.TableEncryption.KMS,encryption_key=kms.Key(self,"MyKey"),# ...database=my_database,table_name="my_table",columns=[glue.Column(name="col1",type=glue.Schema.STRING)],data_format=glue.DataFormat.JSON)KmsManaged- Server-side encryption (SSE-KMS), likeKms, except with an AWS KMS Key managed by the AWS Key Management Service.# my_database: glue.Databaseglue.Table(self,"MyTable",encryption=glue.TableEncryption.KMS_MANAGED,# ...database=my_database,table_name="my_table",columns=[glue.Column(name="col1",type=glue.Schema.STRING)],data_format=glue.DataFormat.JSON)ClientSideKms- Client-side encryption (CSE-KMS) with an AWS KMS Key managed by the account owner.# my_database: glue.Database# KMS key is created automaticallyglue.Table(self,"MyTable",encryption=glue.TableEncryption.CLIENT_SIDE_KMS,# ...database=my_database,table_name="my_table",columns=[glue.Column(name="col1",type=glue.Schema.STRING)],data_format=glue.DataFormat.JSON)# with an explicit KMS keyglue.Table(self,"MyTable",encryption=glue.TableEncryption.CLIENT_SIDE_KMS,encryption_key=kms.Key(self,"MyKey"),# ...database=my_database,table_name="my_table",columns=[glue.Column(name="col1",type=glue.Schema.STRING)],data_format=glue.DataFormat.JSON)Note: you cannot provide aBucketwhen creating theTableif you wish to use server-side encryption (KMS,KMS_MANAGEDorS3_MANAGED).TypesA table's schema is a collection of columns, each of which have anameand atype. Types are recursive structures, consisting of primitive and complex types:# my_database: glue.Databaseglue.Table(self,"MyTable",columns=[glue.Column(name="primitive_column",type=glue.Schema.STRING),glue.Column(name="array_column",type=glue.Schema.array(glue.Schema.INTEGER),comment="array<integer>"),glue.Column(name="map_column",type=glue.Schema.map(glue.Schema.STRING,glue.Schema.TIMESTAMP),comment="map<string,string>"),glue.Column(name="struct_column",type=glue.Schema.struct([name="nested_column",type=glue.Schema.DATE,comment="nested comment"]),comment="struct<nested_column:date COMMENT 'nested comment'>")],# ...database=my_database,table_name="my_table",data_format=glue.DataFormat.JSON)PrimitivesNumericNameTypeCommentsFLOATConstantA 32-bit single-precision floating point numberINTEGERConstantA 32-bit signed value in two's complement format, with a minimum value of -2^31 and a maximum value of 2^31-1DOUBLEConstantA 64-bit double-precision floating point numberBIG_INTConstantA 64-bit signed INTEGER in two’s complement format, with a minimum value of -2^63 and a maximum value of 2^63 -1SMALL_INTConstantA 16-bit signed INTEGER in two’s complement format, with a minimum value of -2^15 and a maximum value of 2^15-1TINY_INTConstantA 8-bit signed INTEGER in two’s complement format, with a minimum value of -2^7 and a maximum value of 2^7-1Date and timeNameTypeCommentsDATEConstantA date in UNIX format, such as YYYY-MM-DD.TIMESTAMPConstantDate and time instant in the UNiX format, such as yyyy-mm-dd hh:mm:ss[.f...]. For example, TIMESTAMP '2008-09-15 03:04:05.324'. This format uses the session time zone.StringNameTypeCommentsSTRINGConstantA string literal enclosed in single or double quotesdecimal(precision: number, scale?: number)Functionprecisionis the total number of digits.scale(optional) is the number of digits in fractional part with a default of 0. For example, use these type definitions: decimal(11,5), decimal(15)char(length: number)FunctionFixed length character data, with a specified length between 1 and 255, such as char(10)varchar(length: number)FunctionVariable length character data, with a specified length between 1 and 65535, such as varchar(10)MiscellaneousNameTypeCommentsBOOLEANConstantValues aretrueandfalseBINARYConstantValue is in binaryComplexNameTypeCommentsarray(itemType: Type)FunctionAn array of some other typemap(keyType: Type, valueType: Type)FunctionA map of some primitive key type to any value typestruct(collumns: Column[])FunctionNested structure containing individually named and typed collumns