Database Migration from Oracle to AWS Redshift using AWS DMS | by Shafiqa Iqbal | Towards Data Science

AWS DMS takes a minimalist approach and creates only those objects required to efficiently migrate the data. In other words, AWS DMS creates tables, primary keys, and in some cases unique indexes, but doesn’t create any other objects that are not required to efficiently migrate the data from the source. For example, it doesn’t create secondary indexes, nonprimary key constraints, or data defaults. So, to generate a complete target schema, use AWS Schema Conversion Tool (SCT). After setting up AWS SCT:
Step 1: Start a new project and connect the source (oracle) and target (Redshift)
Step 2: Right click on your schema which you want to migrate and choose convert schema
Step 3: Select Assessment Report View from View to check the report. Go through hints to fix any schema conversion issues.
Step 4: To fix issues, modify objects on your source database, modify the script generated by SCT manually and then apply them to the target database.
Step 5: After you’re satisfied with your schema conversion, browse through your source and target data types and validate if your datatypes and objects are converted properly.
During the full load phase, the tables are loaded out of order. Thus, we have some foreign key constraint violations if we keep constraints enabled on the target. Also, during full load, Views should be disabled. Because in my scenario all the tables that were being used in views were not loaded. So drop your constraints and views.
In order to replicate your data using DMS with CDC, you need to first prepare your Oracle database:
Setup logminer in Oracle database and configure the destination for Redo log files.
Enable ArchiveLog Mode.
Set up the supplemental logging that DMS needs to capture changes from the Oracle source database.
Make sure the oracle login that you are using in your source endpoint has the necessary permissions.
We’ll go through each point and prepare our source database for data replication.
As sysdba, install the logminer package (if not installed by default installed) from the following path:
@ORACLE_HOME/rdbms/admin/dbmslm.sql
you can simply check if it’s already available using:
desc dbms_logmnr
To build the logminer dictionary, open the database and put the database in quiesce state. Then the dictionary can be created using the following statement:
ALTER DATABASE OPEN;ALTER DATABASE QUIESCE RESTRICTED;EXEC DBMS_LOGSTDBY.BUILD;
After building the LogMiner dictionary, find the latest archive log. This file should be the starting point for recovery. The following SQL query can be used to find the latest archived log:
select   name,   to_char(completion_time,'yyyy/mm/dd hh24:mi') completion_timefrom   v$archived_logorder by   completion_time;
And the output will be similar to
NAME                                   COMPLETION_TIME   ------------------------------------   ----------------   /oracle/appsdb/arch/appsdb_1_248.dbf   2003/09/30 14:55   /oracle/appsdb/arch/appsdb_1_564.dbf   2003/10/03 09:54   /oracle/appsdb/arch/appsdb_1_568.dbf   2003/10/03 12:08
Run the command following to set logs to ARCHIVELOG modeALTER database ARCHIVELOG;
While enabling archive mode in the database, if you get below error, then you need to do the following resolution.
SQL> alter database archivelog;alter database archivelog*ERROR at line 1:ORA-01126: database must be mounted in this instance and not open in any instance
Follow this link to resolve your error.
dbaclass.com
Enable supplemental logging for the database by running the command following.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
This option specifies that when a row is updated, all columns of that row (except for LOBs, LONGS, and ADTs) are placed in the redo log file.
To use an Oracle database as a source in AWS DMS, grant the privileges following to the Oracle user specified in the Oracle endpoint connection settings.
SELECT ANY TRANSACTIONSELECT on V_$ARCHIVED_LOGSELECT on V_$LOGSELECT on V_$LOGFILESELECT on V_$DATABASESELECT on V_$THREADSELECT on V_$PARAMETERSELECT on V_$NLS_PARAMETERSSELECT on V_$TIMEZONE_NAMESSELECT on V_$TRANSACTIONSELECT on ALL_INDEXESSELECT on ALL_OBJECTSSELECT on DBA_OBJECTS – Required if the Oracle version is earlier than 11.2.0.3.
To access logminer redo logs, grant the following privileges to oracle user.
CREATE SESSIONEXECUTE on DBMS_LOGMNRSELECT on V_$LOGMNR_LOGSSELECT on V_$LOGMNR_CONTENTSGRANT LOGMINING – Required only if the Oracle version is 12c or later.
Step 1: First, create your replication instance. A replication instance runs the DMS task.
Step 2: Create your source and target endpoints. Make sure that you select the Refresh schemas option after a successful connection test and Run test before you finish creating each endpoint.
Step 3: Create your task.
For Migration type, choose Migrate existing data and replicate ongoing changes.
Because we are using AWS SCT to pre-create the schema, select Do nothing or Truncate for Target table preparation mode.
Enable logs to debug your task later. If you want to enable detailed debugging, follow the below link. DMS doesn’t give your clear errors and cloudwatch logs while troubleshooting, so enabling detailed debugging might give you some additional insights on the problem.
aws.amazon.com
Select Stop After Applying Cached Changes for the option Stop task after the full load completes. We want the task to stop temporarily after the full load completes and the cached changes are applied. Cached changes are changes that have occurred and accumulated while the full table load process was run. This is the step just before CDC is applied.
Under Table’s Mappings section, add a new selection rule and select your desired schema that you want to migrate. Add New Transformation Rule and transform the schema name to lower case for target PostgreSQL.
Transformation Rule
Do the same with table and column names by adding two more transformation rules.
Go to your Redshift cluster and open the attached IAM Role.
Make sure you have attached the following policies with your cluster — AmazonDMSRedshiftS3Role, AmazonS3FullAccess, AmazonRedshiftFullAccess, AdministratorAccess.
Now go to trusted relationships tab, edit trust relationship and “redshift.amazonaws.com” in service under the principal. This will allow you redshift to run copy commands from S3 once DMS has dumped files on S3.
Start your task and full load will begin.
While your task is running, you can check under table’s statistics to monitor its progress
Once the task is stopped after full load, you can enable the Foreign key constraints and create the views. After that resume your task and check for any CDC.
You can enable detailed logs in your DMS tasks as mentioned above.
If your task is failing, execute select * from stl_load_errors in your Redshift DB for additional error logging. This will help you if any target side errors are occurring.
In this tutorial, we migrated our database from Oracle to AWS Redshift using Schema Conversion Tool and AWS Database Migration Service.
For upcoming stories, you should follow my profile Shafiqa Iqbal.
That’s it, guys! Have fun, keep learning & always coding!