Apache Sqoop. RDBMS to HDFS and back | by Prathamesh Nimkar | Towards Data Science

Across the globe, the most popular databases are SQL based. Considering this, it is paramount for any data lake to be able to pull data from an RDBMS database. Thus, Apache Sqoop was born.
Apache Sqoop supports bi-directional movement of data between any RDBMS and HDFS, Hive or HBase, etc. But, structured data only. It works in a batch-loading format (similar to extraction in ETL applications), supporting extraction of large volume of data, to the tune of multiple terabytes that RDBMS’ are simply, unable to handle.
The above GIF is a workflow of the Sqoop process to load data into HDFS from RDBMS. The converse also holds true. Here’s how it works:
Initiator: The client submits a job to the Sqoop server to load data from source to target (i.e. RDBMS to HDFS, in this case). The connection pool, schema and metadata validation is done at this stage.
Partitioner: The data is to be extracted now. However, for example, a table of size 1 TB cannot be handled as one chunk of data. So, we have a partitioner, wherein, the data is broken down into digestible chunks/blocks of data to be extracted in parallel. Please note, no data is stored here.
Extractor: In this part of the framework, the data is actually moved from the source in specific chunks/blocks. Important to note that not all of the data is loaded into memory for extraction at once. We can think of it as micro-batch processing. Again, no data is stored here.
Loader: The extracted data is now pushed to the target through the Loader phase of framework. Now the data is actually stored.
Cleaner: This is just a clean-up activity to free-up the resources used.
Sqoop 2.x was launched using Map Reduce as its primary processing framework. Sqoop with Spark is a possibility, which can be easily configured through Sqoop’s plug-n-play or modular framework discussed above. That’s the beauty/essence of this generic workflow.
If you haven’t completed the installations, now’s the right time to do so!
medium.com
medium.com
# Quick check of HDFS folders on instance-1sudo su -hdfs dfs -ls /user/root/projects
Let’s use Sqoop to automatically create a new HDFS folder called structuredFlightDataset and import data into it.
sqoop import triggers a request to Sqoop server to initiate a process.connect parameter accepts database connection pool information using the jdbc protocol.driver parameter (optional) accepts the default drivers that actually make the connection between the source, target and the sqoop server. Please note, this parameter is provided as a default for popular OLTP databases such mysql, oracle, db2, postgresql and sql server.username/password parameters to be entered here for authorization.table to import from parameter.target directory parameter where the data will be pushed to.Using standard OOTB mysql delimiters to handle the dataset column types.
sqoop import triggers a request to Sqoop server to initiate a process.
connect parameter accepts database connection pool information using the jdbc protocol.
driver parameter (optional) accepts the default drivers that actually make the connection between the source, target and the sqoop server. Please note, this parameter is provided as a default for popular OLTP databases such mysql, oracle, db2, postgresql and sql server.
username/password parameters to be entered here for authorization.
table to import from parameter.
target directory parameter where the data will be pushed to.
Using standard OOTB mysql delimiters to handle the dataset column types.
Please note, the above import command diagram and labels apply to the export command too with minor, obvious differences.
# Sqoop import from MySQL database on instance-2 to HDFSsqoop import --connect jdbc:mysql://instance-2/flight_data --driver com.mysql.jdbc.Driver --username root -P --table flightrepdata --target-dir /user/root/projects/structuredFlightDataset --mysql-delimiters# Prompt to enter passwordEnter password:
This would throw a nice little error as mysql’s java connector is missing on instance-1. Cloudera says that Sqoop doesn’t ship with third-party JDBC drivers and must be installed separately. Furthermore, it needs a headless open-JDK and some access privileges on the MySQL database front.
# Download the rpm for j-mysql-connector (your url may vary)wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.19-1.el7.noarch.rpm# Prerequisite: java-openjdk-headless rpm installed via yumyum install java-1.8.0-openjdk-headless# Install the rpm nowrpm -ivh mysql-connector-java-8.0.19-1.el7.noarch.rpm# copy to the required folderscd /usr/share/javacp mysql-connector-java.jar /var/lib/sqoop/# Without adding to oozie, it would throw a java exception# Oozie is an orchestration toolsudo -u hdfs hadoop fs -copyFromLocal mysql-connector-java.jar /user/oozie/share/lib/lib_20200429072044/sqoop/
Now, we need to grant access from remote servers/hosts on MySQL:
# Login to instance-2 and make the changes in mysql config filessh instance-2vim /etc/my.cnf# Change/add the binding address, save & close the filebind-address = 0.0.0.0# Restart mysql servicesservice mysqld restart
Login to MySQL, create users with specific IPs and provide the appropriate grants. Please note, MySQL no longer permits creation of users directly from GRANT commands. The username and passwords can be the same as long as the hostnames are different.Quick tip: Hostnames can be found here Cloudera Manager->Host->All Hosts
CREATE USER ‘username’@’instance1_hostname’ IDENTIFIED BY ‘new_password’;CREATE USER ‘username’@’instance2_hostname’ IDENTIFIED BY ‘new_password’;CREATE USER ‘username’@’instance3_hostname’ IDENTIFIED BY ‘new_password’;CREATE USER ‘username’@’instance4_hostname’ IDENTIFIED BY ‘new_password’;# Now let's grant the required privileges:GRANT ALL PRIVILEGES ON *.* TO ‘username’@’instance1_hostname’;GRANT ALL PRIVILEGES ON *.* TO ‘username’@’instance2_hostname’;GRANT ALL PRIVILEGES ON *.* TO ‘username’@‘instance3_hostname’;GRANT ALL PRIVILEGES ON *.* TO ‘username’@‘instance4_hostname’;# Flush the privileges to "activate" them:FLUSH PRIVILEGES;# Quick test:SELECT * from information_schema.user_privileges where grantee like “‘username’@’instance1%’”;     # Repeat for all other hosts
Once that is done, you can re-run the Sqoop import command. First, let’s analyze the logs in depth:
Let’s check the data imported into HDFS:
hdfs dfs -ls /user/root/projects/structuredFlightDataset
This should give you 5 files. You can view them from HDFS’ Name Node Web UI or Hue’s Web UI — http://instance-1:8889/ . You may need to download the files to view. Please note _SUCCESS file contains nothing, its just a flag value. The part-m-00000/1/2/3 files appear as a csv of the actual data.
In order to control how many parallel processes aka mappers/splits are created and executed, we can tweak the import command as such:
# Sqoop import using 1 mapper onlysqoop import --connect jdbc:mysql://instance-2/flight_data --driver com.mysql.jdbc.Driver --username root -P --table flightrepdata --target-dir /user/root/projects/structuredFlightDataset2 --mysql-delimiters -m 1
The number of mappers parameter is just a suggestion to YARN. YARN may choose to completely ignore this suggestion.
# Pretty much the same logs using a single mapper except:INFO mapreduce.ImportJobBase: Transferred 57.1199 MB in 38.2969 seconds (1.4915 MB/sec)
It gives 2 output files, namely, part-m-00000 & _SUCCESS flag. You’ll notice it is a bit slower, but this is just 600k records. Imagine the significant difference in performance if it were 100 million records.
Since mappers are threads, what if you provide twice as many mappers as host machines? Would that enable further performance improvements?
sqoop import --connect jdbc:mysql://instance-2/flight_data --driver com.mysql.jdbc.Driver --username root -P --table flightrepdata --target-dir /user/root/projects/structuredFlightDataset3 --mysql-delimiters -m 8# Notable differences in the log generated:INFO db.IntegerSplitter: Split size: 75918; Num splits: 8 from: 1 to: 607346INFO mapreduce.JobSubmitter: number of splits:8INFO mapreduce.Job:  map 0% reduce 0%INFO mapreduce.Job:  map 13% reduce 0%INFO mapreduce.Job:  map 25% reduce 0%INFO mapreduce.Job:  map 50% reduce 0%INFO mapreduce.Job:  map 75% reduce 0%INFO mapreduce.Job:  map 88% reduce 0%INFO mapreduce.Job:  map 100% reduce 0%INFO mapreduce.ImportJobBase: Transferred 57.1199 MB in 38.247 seconds (1.4934 MB/sec)
As you’ll note, there are no differences in the time taken but it results in 8 output files (part-m-00000–8) & 1 _SUCCESS flag.
Best to keep the number of mappers the same as number of data nodes, where the processing can/will be run.
An interesting question that arises here is, can we use Sqoop to load data into HDFS using SQL queries? Why yes, off course we can. In fact, we can have complex queries joining multiple tables as we see fit.
Bonus Tip: This query is submitted to the database layer and so, downstream Map Reduce or Spark processing is saved if you can filter the required data at this layer itself. But, this isn’t a best practice. The idea of a data lake is to get all the raw data in first.
sqoop import --connect jdbc:mysql://instance-2/flight_data --driver com.mysql.jdbc.Driver --username root -P --target-dir /user/root/projects/structuredFlightDataset3 --mysql-delimiters --query "SELECT UID, OP_UNIQUE_CARRIER FROM flightrepdata_exp WHERE 1=1 AND FL_DATE = '2020-01-01'" -m 2
Loading from OLTP databases in daily batches requires the ability to perform incremental load, i.e. loading only the delta from the previous execution. There are essentially two ways this can be achieved:
LastModified
# Let's create a last modified date columnALTER TABLE `flightrepdata` ADD `LAST_MODIFIED_DATE` TIMESTAMP NOT NULL DEFAULT NOW();# Quick import into HDFSsqoop import --connect jdbc:mysql://instance-2/flight_data --driver com.mysql.jdbc.Driver --username root -P --table flightrepdata --target-dir /user/root/projects/structuredFlightDataset4 --mysql-delimiters# Quick HDFS Checkhdfs dfs -ls /user/root/projects/structuredFlightDataset4# Quick insert nowINSERT `flightrepdata` SELECT NULL,FL_DATE,OP_UNIQUE_CARRIER,ORIGIN_AIRPORT_ID,ORIGIN_AIRPORT_SEQ_ID,ORIGIN_CITY_MARKET_ID,ORIGIN_CITY_NAME,DEST_AIRPORT_ID,DEST_AIRPORT_SEQ_ID,DEST_CITY_MARKET_ID,DEST_CITY_NAME,DEP_TIME,ARR_TIME,NOW()FROM `flightrepdata`WHERE 1=1AND uid IN (1, 2, 3, 4, 5);COMMIT;# Quick data test for the 5 new recordsSELECT uid, OP_UNIQUE_CARRIER, LAST_MODIFIED_DATE FROM flightrepdata ORDER BY 1 DESC LIMIT 5;+--------+-------------------+---------------------+| uid    | OP_UNIQUE_CARRIER | LAST_MODIFIED_DATE  |+--------+-------------------+---------------------+| 607352 | EV                | 2020-04-27 16:49:36 || 607351 | EV                | 2020-04-27 16:49:36 || 607350 | EV                | 2020-04-27 16:49:36 || 607349 | EV                | 2020-04-27 16:49:36 || 607348 | EV                | 2020-04-27 16:49:36 |+--------+-------------------+---------------------+
Now, for incremental import in “lastmodified” mode, we have to provide the following parameters: — incremental — To instantiate incremental extraction process and it’s mode — check-column — The date column from which the date should be checked — last-value — The value post which the data should be picked — append — Appending the new data
sqoop import --connect jdbc:mysql://instance-2/flight_data --driver com.mysql.jdbc.Driver --username root -P --table flightrepdata --target-dir /user/root/projects/structuredFlightDataset4 --mysql-delimiters --incremental lastmodified --check-column LAST_MODIFIED_DATE --last-value "2020-04-27 16:49:36" --append# Additional logsINFO mapreduce.ImportJobBase: Transferred 611 bytes in 26.6098 seconds (22.9615 bytes/sec)INFO mapreduce.ImportJobBase: Retrieved 5 records.INFO util.AppendUtils: Appending to directory structuredFlightDataset4INFO util.AppendUtils: Using found partition 4# Quick HDFS Checkhdfs dfs -ls /user/root/projects/structuredFlightDataset4
Append
# Let's run the same INSERT query as above and testSELECT uid, OP_UNIQUE_CARRIER, LAST_MODIFIED_DATE FROM flightrepdata ORDER BY 1 DESC LIMIT 5;+--------+-------------------+---------------------+| uid    | OP_UNIQUE_CARRIER | LAST_MODIFIED_DATE  |+--------+-------------------+---------------------+| 607359 | EV                | 2020-04-27 17:05:19 || 607358 | EV                | 2020-04-27 17:05:19 || 607357 | EV                | 2020-04-27 17:05:19 || 607356 | EV                | 2020-04-27 17:05:19 || 607355 | EV                | 2020-04-27 17:05:19 |+--------+-------------------+---------------------+
Now, for incremental import in “append” mode, we have to provide the following parameters: — incremental — To instantiate incremental extraction process and it’s mode — check-column — The column for which the value should be checked — last-value — The value post which the data should be picked
# Incremental Importsqoop import --connect jdbc:mysql://instance-2/flight_data --driver com.mysql.jdbc.Driver --username root -P --table flightrepdata --target-dir /user/root/projects/structuredFlightDataset4 --mysql-delimiters --incremental append --check-column uid --last-value 607352# Additional LogsINFO tool.ImportTool: Lower bound value: 607352INFO tool.ImportTool: Upper bound value: 607359INFO mapreduce.ImportJobBase: Transferred 611 bytes in 25.496 seconds (23.9646 bytes/sec)INFO mapreduce.ImportJobBase: Retrieved 5 records.INFO util.AppendUtils: Appending to directory structuredFlightDataset4INFO util.AppendUtils: Using found partition 9
Sqoop doesn’t create a table automatically, hence we’ve to create one with its underlying structure i.e. columns and data types.
# Login to instance-2 MySQL and appropriate database instanceuse flight_data# Create the tableCREATE TABLE flightrepdata_exp(UID INT PRIMARY KEY, FL_DATE DATE,OP_UNIQUE_CARRIER VARCHAR(10),ORIGIN_AIRPORT_ID INT,ORIGIN_AIRPORT_SEQ_ID INT,ORIGIN_CITY_MARKET_ID INT,ORIGIN_CITY_NAME VARCHAR(300),DEST_AIRPORT_ID INT,DEST_AIRPORT_SEQ_ID INT,DEST_CITY_MARKET_ID INT,DEST_CITY_NAME VARCHAR(300),DEP_TIME INT,ARR_TIME INT);# Sqoop export to MySQL database on instance-2 from HDFSsqoop export --connect jdbc:mysql://instance-2/flight_data --username root -P --export-dir /user/root/projects/structuredFlightDataset/ --table flightrepdata_exp --mysql-delimiters# Pretty much the same logs really
Please note the export command can also be run with multiple mappers. Just as for import commands, it is best to limit number of mappers = number of hosts.
Mapper and Reducer tasks of MapReduce jobs are run by YARN’s Resource Manager (RM) in 2 separate containers distributed across several nodes. But, if your dataset is small or your job contains small mapper tasks or your job contains just 1 reducer task — we can set the Uber mode to TRUE. This forces the RM to run the mapper and reducer task sequentially in one single container or JVM, thus reducing overhead of launching new containers and of networking across multiple nodes for a small job. The job completes faster.
Should you face any issues in Sqoop, the logs can be found here:http://<instance-1 IP>:8088/proxy/<YARN’s job application id>/or better yet for all jobs go here:http://<instance-1 IP>:8088/cluster
For more commands, perhaps you’d like to visit Apache Sqoop’s user guide
Quick cleanup:
# Be a good Samaritan and clean up your workspacehdfs dfs -rm -R /user/root/projects/structuredFlightDataset2hdfs dfs -rm -R /user/root/projects/structuredFlightDataset3hdfs dfs -rm -R /user/root/projects/structuredFlightDataset4drop table flightrepdata_exp;ALTER TABLE `flightrepdata` DROP `LAST_MODIFIED_DATE`;DELETE FROM `flightrepdata` WHERE uid >= 607347; COMMIT;
References:
[1] Sqoop User Guide, Apache Sqoop, ASF
[2] Managing the Sqoop Client, Apache Sqoop, Cloudera Documentation
[3] C. Chaudhari, What is Uber Mode? (2018), Cloudera Community
[4] A. Elmahrek, J. Cecho (2014), Sqoop2 New Generation of Big Data — Apache Sqoop Workflow, Apache Software Foundation
Please note, I didn’t download all of the columns to create the dataset. You can though.
The structured cleansed dataset can be found here. Explanation with some data? Here you go.The Homepage of the original data set — here. The raw data — here. More details on the fields — here.