How to Migrate Your Oracle Database to PostgreSQL | by Shafiqa Iqbal | Towards Data Science

Using this walkthrough, you can learn how to migrate Oracle database to RDS PostgreSQL using AWS Data Migration Service and AWS Schema Conversion Tool (AWS SCT). In this post we will cover the following:
Migrate Schema using AWS SCT
Drop foreign keys and secondary indexes on the target database, and disable triggers.
Perform full load and change data capture (CDC) using AWS DMS
AWS DMS takes a minimalist approach and creates only those objects required to efficiently migrate the data. In other words, AWS DMS creates tables, primary keys, and in some cases unique indexes, but doesn’t create any other objects that are not required to efficiently migrate the data from the source. For example, it doesn’t create secondary indexes, nonprimary key constraints, or data defaults. So, to generate a complete target schema, use AWS Schema Conversion Tool (SCT). After setting up AWS SCT:
Step 1: Start a new project and connect the source (oracle) and target (RDS PostgrSQL)
Step 2: Right click on your schema which you want to migrate and choose convert schema
Step 3: Select Assessment Report View from View to check the report. Go through hints to fix any schema conversion issues.
Step 4: To fix issues, modify objects on your source database, modify the script generated by SCT manually and then apply them to the target database.
Step 5: After you’re satisfied with your schema conversion, browse through your source and target data types and validate if your datatypes and objects are converted properly.
During the full load phase, the tables are loaded out of order. Thus, we have some foreign key constraint violations if we keep constraints enabled on the target. Also, during the full load, secondary indexes should be disabled because they can slow down the table replication. They do so because the indexes need to be maintained as the records are loaded.
Execute the following script to generate SQL for disabling foreign key constraints.
--Use the below SQL to generate the scripts to drop foreign keysselect 'alter table '||con.conrelid::regclass||' drop constraint '||con.conname||';' as dropfrom pg_constraint conjoin pg_namespace ns on con.connamespace = ns.oidwhere ns.nspname = 'Schema_Name'and con.contype = 'f';
Save the output and execute it on your target PostgreSQL database.
Execute the following script to generate SQL for disabling foreign key constraints.
--Use the below SQL to generate the scripts to disable triggersselect 'alter table '||trigger_schema||'.'||event_object_table||' disable trigger '||trigger_name||';' as disablefrom information_schema.triggerswhere trigger_schema = 'hr';
Save the output and execute it on your target PostgreSQL database.
In order to replicate your data using DMS with CDC, you need to first prepare your Oracle database:
Setup logminer in Oracle database and configure the destination for Redo log files.
Enable ArchiveLog Mode.
Set up the supplemental logging that DMS needs to capture changes from the Oracle source database.
Make sure the oracle login that you are using in your source endpoint has the necessary permissions.
We’ll go through each point and prepare our source database for data replication.
As sysdba, install the logminer package (if not installed by default installed) from the following path:
@ORACLE_HOME/rdbms/admin/dbmslm.sql
you can simply check if it’s already available using:
desc dbms_logmnr
To build the logminer dictionary, open the database and put the database in quiesce state. Then the dictionary can be created using the following statement:
ALTER DATABASE OPEN;ALTER DATABASE QUIESCE RESTRICTED;EXEC DBMS_LOGSTDBY.BUILD;
After building the LogMiner dictionary, find the latest archive log. This file should be the starting point for recovery. The following SQL query can be used to find the latest archived log:
select   name,   to_char(completion_time,'yyyy/mm/dd hh24:mi') completion_timefrom   v$archived_logorder by   completion_time;
And the output will be similar to
   NAME                                   COMPLETION_TIME   ------------------------------------   ----------------   /oracle/appsdb/arch/appsdb_1_248.dbf   2003/09/30 14:55   /oracle/appsdb/arch/appsdb_1_564.dbf   2003/10/03 09:54   /oracle/appsdb/arch/appsdb_1_568.dbf   2003/10/03 12:08
Run the command following to set logs to ARCHIVELOG modeALTER database ARCHIVELOG;
While enabling archive mode in the database, if you get below error, then you need to do the following resolution.
SQL> alter database archivelog;alter database archivelog*ERROR at line 1:ORA-01126: database must be mounted in this instance and not open in any instance
follow this link to resolve your error.
dbaclass.com
Enable supplemental logging for the database by running the command following.
ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
This option specifies that when a row is updated, all columns of that row (except for LOBs, LONGS, and ADTs) are placed in the redo log file.
To use an Oracle database as a source in AWS DMS, grant the privileges following to the Oracle user specified in the Oracle endpoint connection settings.
SELECT ANY TRANSACTIONSELECT on V_$ARCHIVED_LOGSELECT on V_$LOGSELECT on V_$LOGFILESELECT on V_$DATABASESELECT on V_$THREADSELECT on V_$PARAMETERSELECT on V_$NLS_PARAMETERSSELECT on V_$TIMEZONE_NAMESSELECT on V_$TRANSACTIONSELECT on ALL_INDEXESSELECT on ALL_OBJECTSSELECT on DBA_OBJECTS – Required if the Oracle version is earlier than 11.2.0.3.
To access logminer redo logs, grant the following privileges to oracle user.
CREATE SESSIONEXECUTE on DBMS_LOGMNRSELECT on V_$LOGMNR_LOGSSELECT on V_$LOGMNR_CONTENTSGRANT LOGMINING – Required only if the Oracle version is 12c or later.
Step 1: First, create your replication instance. A replication instance runs the DMS task.
Step 2: Create your source and target endpoints. Make sure that you select the Refresh schemas option after a successful connection test and Run test before you finish creating each endpoint.
Step 3: Create your task.
For Migration type, choose Migrate existing data and replicate ongoing changes.
Because we are using AWS SCT to pre-create the schema, select Do nothing or Truncate for Target table preparation mode.
Select Stop After Applying Cached Changes for the option Stop task after the full load completes. We want the task to stop temporarily after the full load completes and the cached changes are applied. Cached changes are changes that have occurred and accumulated while the full table load process was run. This is the step just before CDC is applied.
Under Table’s Mappings section, add a new selection rule and select your desired schema that you want to migrate. Add New Transformation Rule and transform the schema name to lower case for target PostgreSQL.
Do the same with table and column names by adding two more transformation rules.
Start your task and full load will begin.
While your task is running, you can check under table’s statistics to monitor its progress
After your full load is finishes and cache changes have been applied, you need to reapply the triggers and foreign key constraints that you disabled before full load.
Execute the following script to generate SQL for enabling foreign key constraints
--Use the below SQL to generate the script to create foreign keysselect 'alter table '||con.conrelid::regclass||' add constraint '||con.conname||' '||pg_get_constraintdef(con.oid)||';' as createfrom pg_constraint conjoin pg_namespace ns on con.connamespace = ns.oidwhere ns.nspname = 'schema_name'and con.contype = 'f';
Execute the following script to generate SQL for enabling triggers
--Use the below SQL to generate the scripts enable triggersselect 'alter table '||trigger_schema||'.'||event_object_table||' enable trigger '||trigger_name||';' as enablefrom information_schema.triggerswhere trigger_schema = 'schema_name';
Run the generated SQL on target PostgreSQL.
Now that we have the foreign keys and secondary indexes back, we can enable the DMS task. Simply go to the DMS console and choose Tasks. Select the task in the list and choose Start/Resume. Select the Start option and choose Start task.
Once it’s started, you can see the status of your task as “ongoing replication”. You can test the replication by updating some records in source oracle and checking under table’s statistics tab and running queries in your target PostgreSQL Database.
In this tutorial, we migrated Oracle database to PostgreSQL database using AWS Data migration service with full load and CDC. We migrated the schema using Schema Conversion Tool (SCT) and replicated data using DMS.
For upcoming stories, you should follow my profile Shafiqa Iqbal.
That’s it, guys! Have fun, keep learning & always coding!