Syncing Data from Python to AWS RDS | by Onur Okyol | Towards Data Science

Amazon Web Services is the cloud platform provided by the Amazon company. It is one of the most frequently emphasized / sought-after platforms in job postings such as data scientist and data engineer. In today’s world, cloud platforms are becoming more and more important as big data grows day by day. In addition to flexibility, speed and practicality, cloud platforms like AWS are built on the philosophy of spending as much as you use with their scalable cost. Many startups, world giants, government agencies use AWS.
AWS RDS is the service for creating a traditional database service on the AWS platform. It can be installed very quickly and stands up immediately. You pay as much as you use within the scope of the need. You can resize according to the increasing data size and your need for speed.
If you install the RDS system on AWS, you don’t have to deal with hardware. You do not spend much time for database setup, and operations such as backup are done automatically. Often you just need to play with a few parameters and powers.
AWS RDS supports AWS Aurora, PostgreSQL, MySQL, MariaDB, ORACLE, Microsoft SQL database infrastructures.
Let’s Start Creating Database on AWS RDS!
1- First we open an AWS account.
We need to enter the AWS homepage to register with AWS. After entering this page, we proceed by saying create an AWS account. In all steps, we must enter our information and especially the e-mail address correctly. Then we need to enter our payment information. Some services of AWS are free to use if you do not exceed the limits. If you have entered payment information, make sure you do not exceed these limits. AWS charges a $ 1 fee to verify your payment account, then returns it.
For detailed information, you can follow the steps to create an AWS account.
2- Let’s enter to RDS and create the Database.
After creating an AWS account, we enter RDS.
We click on the Databases option on the left side of the console. We take the first step to create DB by saying Create database on the page that opens.
We need to select the database infrastructure on the page that opens. In this example, we will proceed through MySQL.
For our type of operation, we’re choosing the Free tier for now.
We need to determine the database name, user name, and password to be created from the Settings section. I created a db called the experiment for the example and created a user named admincv.
Amazon provides some standard features for the database to be created. If you are doing this study for trial purposes, it would be better not to turn on Storage autoscaling feature in order not to charge. If you use it within the limits set by Amazon, there will be no charge.
After a little wait, the system will make the database available. It will then appear as “Available” in the Status field.
Now the database has occurred.
3- Let’s enter the database we created.
Here, the endpoint & port information in the Connectivity & Security area is important and it is useful to take note. This information will then be required so that we can access DB from Python.
Now Let’s Switch to Python and Connect to the Database!
1- We download our libraries.
import pandas as pdimport pymysqlfrom sqlalchemy import create_engine
2- We enter our connection information.
We enter the above endpoint link in the host area. We need to select 3306 as a port. The username is “cvadmin” that we have previously created. The password is that you create for this user to connect. Our DB name was a “deneme”. After assigned this to the database variable, we completed all the operations.
host=’deneme.cykgvlpxbgsw.us-east-2.rds.amazonaws.com’port=int(3306)user=”cvadmin”passw=”yourpassw”database=”deneme”
3- We create an engine by collecting the connection information.
Fields such as user, passw in the following code feed on the variables that we created in the previous step.
mydb = create_engine(‘mysql+pymysql://’ + user + ‘:’ + passw + ‘@’ + host + ‘:’ + str(port) + ‘/’ + database , echo=False)
4- We print the data we have on the database.
We create a table with the name = “hisse” command and we can now print the data we have. stockshistoricdata, a dataframe I have. It gives daily prices of shares in Bist100. You can also use another DF.
stockshistoricdata.to_sql(name=”hisse”, con=mydb, if_exists = ‘replace’, index=False)
Let’s check if the data is printed on the table via DBeaver!
Our data is now in the “hisse” table.
Conclusion
Why would you want to press a data in Python to a database on AWS in a data science project? You’re right to ask that question. If you are working with big data, if the data is on a daily or weekly new data, and your python code works for minutes or hours, these situations can be a waste of time.
To avoid this situation, you create a script in Python, this new data prints this table every day. It does not touch old data. You can connect to the “hisse” table and work on the accumulated data at any time. You can save time and CPU by printing one-day stock data instead of the effort required when creating all data from scratch. In addition, more than one person can work on the same table.
I hope my writing has been useful. Have a pleasant reading!
Resources;
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Tutorials.WebServerDB.CreateDBIns
https://aws.amazon.com/tr/premiumsupport/knowledge-center/create-and-activate-aws-account/?nc1=h_ls
My Linkedin Profile;