Datalake File Ingestion: From FTP to AWS S3 | by Furqan Butt | Towards Data Science

Hello everyone. When developing Datalake pipe lines, data ingestion is an important step in the entire process. We need a reliable, secure and fault tolerant method to bring our files from source(client FTP server) to our target( AWS S3). Considering how important this step is, I developed an implementation in python that can be used to ingest files from an FTP server to AWS s3. I have designed the code in such manner that it can be used in a managed(AWS Glue) or un-managed(local machine) environment to transfer files from an FTP server to AWS s3.
In this article I will explain how we can transfer files present on an FTP server such as FileZilla to Amazon s3 using the Paramiko Library in Python.
The file ingestion code can be found on my GitHub repository. The file ingestion code does the following things:
Creates a secure ssh SFTP connection with the File server using Paramiko.
Automatically handles single part and multipart upload depending upon the file size (multipart on 100MB or more).
Automatically handles retires in case of failed parts uploads during multipart upload.
Partitions the data in s3 based on current year,month,day,hour.
Ensures which file has been uploaded
Can be used as a python job in AWS Glue to transfer files from FTP to S3 using AWS Glue.
I will do a step by step walk through of each section of the code for better understanding. So lets get started.
Before we get started we need to do a little installation setup on our local machine and a directory setup on our FTP server.
I’m assuming you guys have python already installed on you machines. Visit this link to install python on your machine if you haven’t already.Before getting towards the code we need to make sure we have the following modules installed:
First we’ll install Paramiko.
pip install paramiko
Since we’ll be uploading files to s3 we will be needing the boto3 client.
pip install boto3
We need to create a few directories on our FTP server that we’ll be using in the implementation.
Note: this directory setup is a one time thing and we won’t have to do it every single time.
Connect to your FTP sever. I’m using FileZilla in my case. Goto site manager. Select the SFTP option in the protocol drop down menu. Enter your host, ip, username and password and press connect.
Once you are connected create two directories on your FTP server:
A parent directory where you will upload your files.
a processed directory. This directory will be used to keep track of which files have been uploaded to s3.
After doing the above two steps we now we have all the necessary modules installed and directory structure setup. We can now get started with the actual implementation.
Let’s first briefly go through the two files that are in the git repository.
Our entire FTPIngestion class implementation is as follows. I’ll explain each section of the code one by one.
Note: I called it a python glue job because we can run the same code in a AWS Glue python shell environment and achieve the same FTP file transfer functionality using AWS Glue.
This file is being to define all our configurations such as host-name, IP, port, username, password, s3 bucket name, ftp directory paths etc. It is a good programming practice to keep all the configurations in a separate file. This makes our code more manageable.
We use this file by importing it in our python_glue_ingestion_job.py file. Our config.py file looks as follows:
After having a brief overview of both of our files let’s dive into each section of the code.
Our code kicks off from the main method. We first create an object ftp_obj of Our FTPIngestion class. On object creation the __init(self)__ method is called which sets up all the configuration values.
The ftp_obj then calls the initiate_ingestion(self) method which will first create an ssh connection with our FTP server using the create_ssh_connection(self) method. On successful connection, an SFTP connection is then created using the create_sftp_connection(self) method which we will use to access the FTP server.
Once SFTP connection is established we do the following
change current directory to parent directory
self.sftp_client.chdir(self.ftp_directory_path)
get all the files that are to be uploaded to S3
files_to_upload = self.sftp_client.listdir()
create s3 partition structure using the create_s3_partition(self) method. I’m using the current date time to setup the partition structure as follows:
/<directory_name>/year = <year>/ month = <month>/ day = <day>/ hour = <hour>/file
after this we start uploading the files one by one. Each file in the list is passed to the s3_upload_file_multipart() method. If file size is greater than 100Mb files will be uploaded via multipart, otherwise as a standard file.
for ftp_file in files_to_upload:                        sftp_file_obj = self.sftp_client.file(ftp_file, mode='r')    if self.s3_upload_file_multipart(sftp_file_obj,                                     s3_partition+ftp_file):        print('file uploaded to s3')        files_to_move.append(ftp_file)
On each successful upload the file gets moved to the processed directory using the move_files_to_processed() method that we created on the FTP server. This is done in order to keep track of all the files that have been successfully uploaded to s3.
if files_to_move:                        for uploaded_file in files_to_move:        self.move_files_to_processed(uploaded_file)else:    print("nothing to upload")
after all the files have been successfully upload we close all the connections using the close_connections(self) method.
Lets now discuss each section of our FTPIngestion class in some detail.
This is the init method also called as the constructor of our class. When we create an object of our FTPIngestion class all the attributes get set at object creation time. Values for each attribute are being fetched from the config.py class where we have defined their values. We import the config.py file as cfg and set each attribute with its corresponding value.
Recommendation: instead of using the config.py class we can use AWS SSM where we define our key value pairs for each of our attribute just like we did in our config.py file. Using SSM is more robust and secure.
This method creates a secure ssh connection with our FTP server using the given credentials. On successful connection it sets the self.ssh_ok attribute to True and returns. Otherwise self.ssh_ok is set to False and returned.
one thing to notice in this method is the self.ssh_client.set_missing_host_key_policy( paramiko.AutoAddPolicy()). We configured it to set the missing host keys with a auto generated policy. If you’re going to run this in AWS Glue as python shell job, instead of adding the auto add policy, use the host policies. For this, do the following in the code
#self.ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())# in production, use load_system_host_keys             self.ssh_client.load_system_host_keys()
This method calls the create_ssh_connection method creates the ssh connection and then proceeds to open the SFTP connection. On successful connection it sets the self.sftp_ok to True and on failure sets it to False and returns the value.
This method is called in the initiate_ingestion method after the file is successfully uploaded to s3. This method take the file name as argument. Sets the source and destination paths on the FTP server as src and dest, and then executes the following command which will move the file from the src path to the dest path.
try:    _, _, _ = self.ssh_client.exec_command("mv " + src+" " + dest)        except Exception as error:    print("error moving files to processed directory, error: ", error)
Our move_files_to_processed method is as follows:
This method creates the s3 partition structure. The partition consists of root directory name followed by year, month,day,hour. In this way we can have a time based partitioned data.
This method uploads the files to the specified s3 bucket. It uses the TransferConfig class to handle the multipart upload. We specify the configurations for TransferConfig as follows:
config = TransferConfig(multipart_threshold=cfg.MULTIPART_THRESHOLD,                        multipart_chunksize=cfg.MULTIPART_CHUNKSIZE,                        max_concurrency=cfg.MAX_CONCURRENCY,                        use_threads=cfg.USER_THREADS                        )
the multipart_threshold parameter determines the minimum size of file that should be after which the file will be uploaded via multipart. AWS recommends all files greater than 100MB be uploaded via multipart, so I have set this parameter to 100MB. This can be changed in the config.py file.
the multipart_chunksize parameter determines the size of each part during multipart upload. I have set it to 20MB.
the max_concurrency parameter determines the maximum number of concurrent S3 API transfer operations that should be used to upload each part. Default is 10.
lastly the use_threads parameter determines whether transfer operations use threads to implement concurrency. Thread use can be disabled by setting the use_threads attribute to False. If thread use is disabled, transfer concurrency does not occur. Accordingly, the value of the max_concurrency attribute is ignored.
We pass this config object to our s3.upload_fileobj as parameter
self.s3.upload_fileobj(source_file, self.s3_bucket_name,                       s3_target_file, Config=config)
The TransferConfig class automatically checks whether to upload the file as a single part or via multipart. It automatically handles the failed uploads and takes care of retries in case of failures.
This method initiates the calls to establish ssh and sftp connections. Changes FTP directory path to specified path where the files reside. Gets list of all the files in the FTP specified path and starts upload to s3. Files Successfully uploaded to S3 gets moved into the processed directory on FTP server. Once all files are uploaded closes all the connections.
This mehtod is used to close the ssh and sftp connections.
As mentioned earlier, the same implementation can be used in a managed environment such as AWS Glue. If user has files arriving on the FTP server at a certain time period (hour, day, month etc) he can schedule the job to run at a specific time to ingest all the files from the FTP server to s3.
We have to modify a few things in our code to be able to run this implementation as an AWS Glue Python shell job
In our imports section of python_glue_injestion_job.py file uncomment the following import. We are using easy install to install the paramiko module in the Glue Environment since it does not come with default Python installation.
install_path = os.environ['GLUE_INSTALLATION']easy_install.main( ["--install-dir", install_path, "https://files.pythonhosted.org/packages/ac/15/4351003352e11300b9f44 a13576bff52dcdc6e4a911129c07447bda0a358/paramiko-2.7.1.tar.gz"] )reload(site)
in create_ssh_connection method comment following
#self.ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
and uncomment the follwoing
self.ssh_client.load_system_host_keys()
what we are doing is instead of using an auto generated policy we are using the host keys of the environment we are in.
After doing above steps we can run this code in as a Python shell job in AWS Glue.
In this article, we looked at how can perform the data ingestion step of a Datalake pipeline. We learned how we can use the paramiko library to create a secure ssh SFTP connection with a FTP server and upload files from the server to our AWS S3 buckets. We also briefly looked at how we can use the given implementation in a managed environment such as AWS Glue.
Link to my Github repository: