Datalake File Ingestion: From FTP to AWS S3
By: Furqan Butt
Tags: Python, AWS, Data Lake, Big Data, Ftp

More insight into our code:

Lets now discuss each section of our FTPIngestion class in some detail.

init method

This is the init method also called as the constructor of our class. When we create an object of our FTPIngestion class all the attributes get set at object creation time. Values for each attribute are being fetched from the config.py class where we have defined their values. We import the config.py file as cfg and set each attribute with its corresponding value.

init method

Recommendation: instead of using the config.py class we can use AWS SSM where we define our key value pairs for each of our attribute just like we did in our config.py file. Using SSM is more robust and secure.

create_ssh_connection method

This method creates a secure ssh connection with our FTP server using the given credentials. On successful connection it sets the self.ssh_ok attribute to True and returns. Otherwise self.ssh_ok is set to False and returned.

one thing to notice in this method is the self.ssh_client.set_missing_host_key_policy( paramiko.AutoAddPolicy()). We configured it to set the missing host keys with a auto generated policy. If you’re going to run this in AWS Glue as python shell job, instead of adding the auto add policy, use the host policies. For this, do the following in the code

#self.ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

# in production, use load_system_host_keys

self.ssh_client.load_system_host_keys()

create_sftp_connection method

This method calls the create_ssh_connection method creates the ssh connection and then proceeds to open the SFTP connection. On successful connection it sets the self.sftp_ok to True and on failure sets it to False and returns the value.

create_sftp_connection method

move_files_to_processed method

This method is called in the initiate_ingestion method after the file is successfully uploaded to s3. This method take the file name as argument. Sets the source and destination paths on the FTP server as src and dest, and then executes the following command which will move the file from the src path to the dest path.

try:

_, _, _ = self.ssh_client.exec_command("mv " + src+" " + dest) except Exception as error:

print("error moving files to processed directory, error: ", error)

Our move_files_to_processed method is as follows:

move_files_to_processed method

create_s3_partition method

This method creates the s3 partition structure. The partition consists of root directory name followed by year, month,day,hour. In this way we can have a time based partitioned data.

create_s3_partition method

s3_upload_file_multipart method

This method uploads the files to the specified s3 bucket. It uses the TransferConfig class to handle the multipart upload. We specify the configurations for TransferConfig as follows:

config = TransferConfig(multipart_threshold=cfg.MULTIPART_THRESHOLD,

multipart_chunksize=cfg.MULTIPART_CHUNKSIZE,

max_concurrency=cfg.MAX_CONCURRENCY,

use_threads=cfg.USER_THREADS

)

the multipart_threshold parameter determines the minimum size of file that should be after which the file will be uploaded via multipart. AWS recommends all files greater than 100MB be uploaded via multipart, so I have set this parameter to 100MB. This can be changed in the config.py file.

parameter determines the minimum size of file that should be after which the file will be uploaded via multipart. AWS recommends all files greater than 100MB be uploaded via multipart, so I have set this parameter to 100MB. This can be changed in the config.py file. the multipart_chunksize parameter determines the size of each part during multipart upload. I have set it to 20MB.

parameter determines the size of each part during multipart upload. I have set it to 20MB. the max_concurrency parameter determines the maximum number of concurrent S3 API transfer operations that should be used to upload each part. Default is 10.

parameter determines the maximum number of concurrent S3 API transfer operations that should be used to upload each part. Default is 10. lastly the use_threads parameter determines whether transfer operations use threads to implement concurrency. Thread use can be disabled by setting the use_threads attribute to False. If thread use is disabled, transfer concurrency does not occur. Accordingly, the value of the max_concurrency attribute is ignored.

We pass this config object to our s3.upload_fileobj as parameter

self.s3.upload_fileobj(source_file, self.s3_bucket_name,

s3_target_file, Config=config)

The TransferConfig class automatically checks whether to upload the file as a single part or via multipart. It automatically handles the failed uploads and takes care of retries in case of failures.

s3_upload_file_multipart method

initiate_ingestion method

This method initiates the calls to establish ssh and sftp connections. Changes FTP directory path to specified path where the files reside. Gets list of all the files in the FTP specified path and starts upload to s3. Files Successfully uploaded to S3 gets moved into the processed directory on FTP server. Once all files are uploaded closes all the connections.

close_connections method

This mehtod is used to close the ssh and sftp connections.

Using the code as AWS Glue Job

As mentioned earlier, the same implementation can be used in a managed environment such as AWS Glue. If user has files arriving on the FTP server at a certain time period (hour, day, month etc) he can schedule the job to run at a specific time to ingest all the files from the FTP server to s3.

We have to modify a few things in our code to be able to run this implementation as an AWS Glue Python shell job

In our imports section of python_glue_injestion_job.py file uncomment the following import. We are using easy install to install the paramiko module in the Glue Environment since it does not come with default Python installation.

install_path = os.environ['GLUE_INSTALLATION'] easy_install.main( ["--install-dir", install_path, "https://files.pythonhosted.org/packages/ac/15/4351003352e11300b9f44 a13576bff52dcdc6e4a911129c07447bda0a358/paramiko-2.7.1.tar.gz"] ) reload(site)

in create_ssh_connection method comment following

#self.ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())

and uncomment the follwoing

self.ssh_client.load_system_host_keys()

what we are doing is instead of using an auto generated policy we are using the host keys of the environment we are in.

After doing above steps we can run this code in as a Python shell job in AWS Glue.

Summary

In this article, we looked at how can perform the data ingestion step of a Datalake pipeline. We learned how we can use the paramiko library to create a secure ssh SFTP connection with a FTP server and upload files from the server to our AWS S3 buckets. We also briefly looked at how we can use the given implementation in a managed environment such as AWS Glue.

Link to my Github repository: