Package: etlops
EtlopsEtlops (ETL Operations) is a set of components that make developing ETL workloads easier.Submodulesclients: Wrappers around common relational (MySQL) and MPP databases (Snowflake), as well as cloud storages (S3 and GCP Storage).databseops: Components to work programmatically with relational and MPP databases like transactions and sequence of queries.streams: Pending.Examplesfrometlops.clients.gcpstorageimportGCPStorageClientfrometlops.clients.cloudstorageimportCloudFileimportosfromdatetimeimportdatefrompprintimportpprintg=GCPStorageClient()​# get serialized metadata of all buckets in the project (list of dicts)pprint(g.list_buckets())​# get serialized metadata of all objects in bucket (list of dicts)pprint(g.list_bucket_objects("bucket_name"))# Download a file as string:cloud_file=CloudFile("object_key_in_bucket","bucket_name",{gcp_download_mode:"string"})downloaded_file=g.download(cloud_file)print(downloaded_file.get_file())# Download a file as named_file:cloud_file=CloudFile("object_key_in_bucket","bucket_name",{gcp_download_mode:"named_file",gcp_downloaded_file_path="path/in/local/filesystem"})downloaded_file=g.download(cloud_file)withopen("path/in/local/filesystem",'r')asfile:print(file.read())# 'r' or 'rb' depending on file content# Download a file as file-like:cloud_file=CloudFile("object_key_in_bucket","bucket_name",{gcp_download_mode:"file-like",gcp_downloaded_file_path="path/in/local/filesystem"})downloaded_file=g.download(cloud_file)downloaded_file.get_file()# returns instance of BufferedWritter (https://docs.python.org/3/library/io.html#io.BufferedWriter)# Upload a file as string:cloud_file=CloudFile("object_key_in_bucket","bucket_name",{gcp_upload_mode:"string"},"file content")g.upload(cloud_file)# Upload a file as named_file:cloud_file=CloudFile("object_key_in_bucket","bucket_name",{gcp_upload_mode:"named_file"},"file content")g.upload(cloud_file)cloud_file=CloudFile("object_key_in_bucket","bucket_name",dict(gcp_upload_mode="named_file",gcp_uploading_file_path="path/to/file/to/be/uploaded"))g.upload(cloud_file)# Upload a file as file-like:cloud_file=CloudFile("object_key_in_bucket","bucket_name",{gcp_upload_mode:"file-like"},"file content")withopen("path/to/file/to/be/uploaded",'rb')asfile_handle:cloud_file=CloudFile("object_key_in_bucket","bucket_name",dict(gcp_upload_mode="file-like"))cloud_file.set_file(file_handle)g.upload(cloud_file)API Referenceclientsclassetlops.clients.cloudstorage.CloudFile(bucket_file_path: str, bucket: str, cloud_specific_params: dict = None, file_to_upload=None)Utility class used to configure upload and download operations on files across different cloud storage systems.Constructor parameters:bucket_file_path:str(required): Key of the object in the bucket it is located.bucket:str(required): ID of the bucket where the object was uploaded.cloud_specific_params:dict(optional): Parameters to specify the behaviour of the download or upload operation.file_to_upload (optional): Binary of the file to be uploaded. To be used only for upload operations. Its setting can be delayed prior to passing an instance of this class to the client's upload method by using the set_file method or also specifying a file path parameter so the client fetches the file from the filesystem.Attributes_bucket_file_path:str= Key of the object in the bucket.
_bucket:str= ID of the bucket to upload or download the object from._cloud_specific_params:dict= Key-value store of parameters to regulate the download / upload operation._file = Optional. Actual file to be uploaded if the mode of upload entails uploading a file-like or a string.Methodsget_cloud_specific_param(parameter:str): Returns the parameter value for the specified key (parameter argument).has_cloud_specific_parameter(parameter:str) ->bool: Returns True if the parameter exists for this CloudFile, False otherwise.get_bucket_file_path() ->str: Returns the _bucket_file_path attribute.get_bucket() ->str: Returns the _bucket attribute.set_file(file) ->None: Sets the file parameter to the _file attribute.get_file(): Returns the _file attribute.classetlops.clients.gcpstorage.GCPStorageClient()Wrapper of the GCP Storage client from the google-cloud-storage python SDK. See documentation here:https://googleapis.dev/python/storage/latest/index.htmlConstructor parameters:
NoneAttributes_client: Instance ofgoogle.cloud.storage.client.ClientMethodsdownload(cloud_file:etlops.clients.cloudstorage.CloudFile) -> CloudFile: Returns a CloudFile containing a downloaded file according to the parameters specified in cloud_file.You can instantiate the CloudFile class with the followingcloud_specific_paramsto configure the download operation:gcp_download_mode: One ofstring,named_fileorfile-like.gcp_bucket_filepath: Key of the object in the bucket to download the file from.gcp_downloaded_file_path: Only to be used whengcp_download_modeisnamed-fileorfile-like. Absolute or relative path on the local filesystem to download the file to. In the case offile-like, the content of the downloaded file will be written to the file the path points to.When downloading a file asfile-like, you have 2 options: - To setgcp_downloaded_file_pathas acloud_specific_parameter, which will write the downloaded file on a file on that path. - To not setgcp_downloaded_file_pathas acloud_specific_parameter, which will make available an openedBufferedWritterinstance in the _file attribute of the CloudFile instance.If you downloaded asstringorfile-likewithout specifying agcp_downloaded_file_path. You can access the downloaded file by calling the methodget_fileof CloudFile.upload(cloud_file:etlops.clients.cloudstorage.CloudFile) -> None: Uploads a given CloudFile given according to the parameters specificed in cloud_file.You can instantiate the CloudFile class with the followingcloud_specific_paramsto configure the download operation:gcp_upload_mode: One ofstring,named_fileorfile-like.gcp_bucket_filepath: Key of the object in the bucket to download the file from.gcp_uploading_file_path: Only to be used whengcp_upload_modeisnamed-fileorfile-like. Absolute or relative path on the local filesystem to the file intended to be uploadedWhen uploading a file asfile-like, you have 2 options: - To setgcp_uploading_file_pathas acloud_specific_parameter, which will make the file that paths points to be read and uploaded. - To not setgcp_uploading_file_pathas acloud_specific_parameterand setting explicitly the file to be uploaded calling theset_fileon the CloudFile instance.list_buckets() ->list: Returns a list ofgoogle.cloud.storage.bucket.Bucketinstances for the authenticated service account provided by the environment variableGOOGLE_APPLICATION_CREDENTIALS.get_bucket(bucket_id:str) ->google.cloud.storage.bucket.Bucket: Returns an instance ofgoogle.cloud.storage.bucket.Bucketmatching the providedbucket_id.list_bucket_objects(bucket_id:str) -> list: Returns a list ofgoogle.cloud.storage.blob.Blobinstances stored in the providedbucket_id.build_object(bucket_file_path:str, bucket_name:str) ->google.cloud.storage.Blob: Returns an instance ofgoogle.cloud.storage.Blobmatching the givenbucket_nameandbucket_file_path.classetlops.clients.awss3.AWSS3Client()Wrapper of the AWS S3 client from the boto3 python SDK. See documentation here:https://boto3.amazonaws.com/v1/documentation/api/latest/index.htmlConstructor parameters:
NoneAttributes_client: Instance ofboto3.resources.factory.s3.ServiceResourceMethodsdownload(cloud_file:etlops.clients.cloudstorage.CloudFile) -> CloudFile: Returns a CloudFile containing a downloaded file according to the parameters specified in cloud_file.You can instantiate the CloudFile class with the followingcloud_specific_paramsto configure the download operation:gcp_download_mode: Eitherfile-likeornamed_file.gcp_bucket_filepath: Key of the object in the bucket to download the file from.gcp_downloaded_file_path: Absolute or relative path on the local filesystem to download the file to. In the case offile-like, the content of the downloaded file will be written to the file the path points to.When downloading a file asfile-like, you have 2 options: - To setgcp_downloaded_file_pathas acloud_specific_parameter, which will write the downloaded file on a file on that path. - To not setgcp_downloaded_file_pathas acloud_specific_parameter, which will make available an openedBufferedWritterinstance in the _file attribute of the CloudFile instance.If you downloaded asfile-likewithout specifying agcp_downloaded_file_path. You can access the downloaded file by calling the methodget_fileof CloudFile.upload(cloud_file:etlops.clients.cloudstorage.CloudFile) -> None: Uploads a given CloudFile given according to the parameters specificed in cloud_file.You can instantiate the CloudFile class with the followingcloud_specific_paramsto configure the download operation:gcp_upload_mode: Eithernamed_fileorfile-like.gcp_bucket_filepath: Key of the object in the bucket to download the file from.gcp_uploading_file_path: Absolute or relative path on the local filesystem to the file intended to be uploaded.When uploading a file asfile-like, you have 2 options: - To setgcp_uploading_file_pathas acloud_specific_parameter, which will make the file that paths points to be read and uploaded. - To not setgcp_uploading_file_pathas acloud_specific_parameterand setting explicitly the file to be uploaded calling theset_fileon the CloudFile instance.list_buckets() ->list: Returns a list ofdictcontaining basic metadata of the buckets present in the authenticated account. To switch accounts, checkhttps://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html. Remember that you can set an alternative _client attribute after instantiating this class.
Contained metadata in thedictinstances:bucket_id: ID of the bucket.created_at: Timestamp (UTC) when the bucket was created.get_bucket(bucket_id:str) ->boto3.resources.factory.s3.Bucket: Returns an instance ofboto3.resources.factory.s3.Bucketmatching the providedbucket_id.list_bucket_objects(bucket_id:str) -> list: Returns a list ofdictcontaining basic metadata of the objects stored in the providedbucket_id.
Contained metadata in thedictinstances:object_key: Key of the object in the bucket it is located.bucket_id: ID of the bucket where the object was uploaded.created_at: Timestamp (UTC) when the object was uploaded.build_object(bucket_file_path:str, bucket_name:str) ->boto3.resources.factory.s3.Object: Returns an instance ofboto3.resources.factory.s3.Objectmatching the givenbucket_nameandbucket_file_path.DevelopmentTo develop on theetlopslibrary, the following is required:Black formattingAdding unit / integration tests where appropriateCode StyleThe code style isblack. All code should be run through the latest version ofblackto ensure code formatting consistency and minimal git diffs during changes. It is highly recommended to set up your editor to haveblackrun every time you save a file.TestingSetting Up Your Test EnvironmentAll unit tests, by default, can be run by simply cloning the project. However, in order to run integration tests, credential values must be supplied. In order to do this, first copy the.env.examplefile, rename it to.env, and populate all of the values as listed in the example. These values are automatically injected into the test environment so that the integration tests can utilize them.Executing the TestsEtlops usespytestto run and test our code. To run the entire test suite, including integration tests, simply run$pytestWe mark integration tests with a pytest marker,integration. To runonlythe integration tests, pytest can be invoked as:$pytest-mintegrationInversely, to run all testsbutintegration tests, pytest can be invoked as:$pytest-m"not integration".Marking TestsWhile we have a separatetests/integrationdirectory,pytestneeds a way of knowing these are the integration tests. While each test could individually be marked with the@pytest.mark.integrationdecorator, the simpler method is to specify the globalpytestmarkvariable. At the top of any file containing integration tests, the line:pytestmark=pytest.mark.integrationshould be present.