Announcing SchemaMapper a C# data integration class library
By: Hadi Fadlallah
Tags: Sql Server, Csharp, Programming, Database, Data Integration

SchemaMapper logo

SchemaMapper is a C# data integration class library that facilitates data import process from external sources having different schema definitions.

It can:

Import tabular data from different data sources (.xls, .xlsx, .csv, .txt, .mdb, .accdb, .htm, .json, .xml, .ppt, .pptx, .doc, .docx) into a SQL table with a user defined table schema after mapping columns between source and destination.

(.xls, .xlsx, .csv, .txt, .mdb, .accdb, .htm, .json, .xml, .ppt, .pptx, .doc, .docx) into a SQL table with a user defined table schema after mapping columns between source and destination. Replace creating many integration services packages by writing few lines of codes

by writing few lines of codes Allow users to add new computed and fixed valued columns.

Used technologies

SchemaMapper is developed with .NET framework 4.5 and utilizes from many technologies to read data from different source such as:

Microsoft Office Interop libraries to import tables from Word and Powerpoint

- Json.Net library to import JSON

- HtmlAgilityPack to import tables from HTML

- Microsoft Access database engine to import data from Excel worksheets and Access databases.

Step-by-step guide

In this example, we will import multiple files that contains credentials of SQL Server instances saved by many users into one table in a SQL database.

Source files

(1) Flat file

text file

File extension: .txt

.txt Columns: server, user, pass

server, user, pass FilePath: D:\SchemaMapperTest\Password_Test.txt

(2) Excel file

excel worksheet

File extension: .xlsx

.xlsx Columns: SQL Instance, username, password, AddedBy

SQL Instance, username, password, AddedBy FilePath: D:\SchemaMapperTest\Password_Test.xlsx

(3) Access database

access table

File extension: .accdb

.accdb Table name: Passwords

Passwords Columns: ID, Server Name, Login, Password, AddedDate

ID, Server Name, Login, Password, AddedDate FilePath: D:\SchemaMapperTest\Password_Test.accdb

Expected SQL destination table

Schema: dbo

dbo Table name: Passwords

Passwords Columns: [User_Name] nvarchar(255), [Password] nvarchar(255), [Server_Name] nvarchar(255) , [AddedDate] DateTime (contains current date) , [UserAndPassword] nvarchar(255) (concatenate user and password using a vertical bar | )

Initializing SchemaMapper class

To initialize the SchemaMapper class you should follow these steps:

1. Add SchemaMapperDLL as reference and import the SchemaMapping and Converters namespaces:

using SchemaMapperDLL.Classes.Converters;

using SchemaMapperDLL.Classes.SchemaMapping;

2. Create a `SchemaMapper class and pass the Destination schema and table names as parameters:

SchemaMapper smPasswords = new SchemaMapper(“dbo”,”Passwords”);

3. Define destination Columns within the `SchemaMapper` class:

//Define Server_Name , User_Name, Password columns

SchemaMapper_Column smServerCol = new SchemaMapper_Column(“Server_Name”, SchemaMapper_Column.ColumnDataType.Text);

SchemaMapper_Column smUserCol = new SchemaMapper_Column(“User_Name”, SchemaMapper_Column.ColumnDataType.Text);

SchemaMapper_Column smPassCol = new SchemaMapper_Column(“Password”, SchemaMapper_Column.ColumnDataType.Text); //Define AddedDate column and fill it with a fixed value = Date.Now

SchemaMapper_Column smAddedDate = new SchemaMapper_Column(“AddedDate”, SchemaMapper_Column.ColumnDataType.Date,DateTime.Now.ToString(“yyyy-MM-dd HH:mm:ss”));



//Define UserAndPassword column with and expression = [User_Name] + ‘|’ + [Password]

SchemaMapper_Column smUserPasswordCol = new SchemaMapper_Column(“UserAndPassword”,SchemaMapper_Column.ColumnDataType.Text,true,”[User_Name] + ‘|’ + [Password]”); //Add columns to SchemaMapper

smPasswords.Columns.Add(smServerCol);

smPasswords.Columns.Add(smUserCol);

smPasswords.Columns.Add(smPassCol);

smPasswords.Columns.Add(smAddedDate);

smPasswords.Columns.Add(smUserPasswordCol);

4. Now, you should add all possible inputs for each destination columns



//Add all possible input Columns Names for each Column

smServerCol.MappedColumns.AddRange(new[] {“server”,”SQL Instance”,”Server Name”});

smUserCol.MappedColumns.AddRange(new[] { “username”, “user”, “Login”});

smPassCol.MappedColumns.AddRange(new[] { “Password”,”pass”, “password” });

5. All unwanted columns should be added to the IgnoredColumns list

//Sys_SheetName and Sys_ExtraFields are an auto generated columns while reading Excel file

smPasswords.IgnoredColumns.AddRange(new[] { “ID”, “AddedBy”, “AddedDate”, “Sys_Sheetname”, “Sys_ExtraFields”});

Converting files into DataTables with unified schema

1. Now we should convert files into `DataTable` objects



//Declare DataTables

DataTable dtExcel;

DataTable dtText;

DataTable dtAccess; //Excel worksheet

using (SchemaMapperDLL.Classes.Converters.MsExcelImport smExcel = new SchemaMapperDLL.Classes.Converters.MsExcelImport(@”D:\SchemaMapperTest\Password_Test.xlsx”))

{

//Read Excel

smExcel.BuildConnectionString();

var lst = smExcel.GetSheets();

//Read only from the first worksheet and consider the first row as header

dtExcel = smExcel.GetTableByName(lst.First(), true, 0);

} //Flat file

using (SchemaMapperDLL.Classes.Converters.FlatFileImportTools smFlat = new SchemaMapperDLL.Classes.Converters.FlatFileImportTools(@”D:\SchemaMapperTest\Password_Test.txt”,true,0))

{

//Read flat file structure

smFlat.BuildDataTableStructure();

//Import data from flat file

dtText = smFlat.FillDataTable();

} //Access database using (SchemaMapperDLL.Classes.Converters.MsAccessImport smAccess = new SchemaMapperDLL.Classes.Converters.MsAccessImport(@”D:\SchemaMapperTest\Password_Test.accdb”))

{

//Build connection string and retrieve Access metadata

smAccess.BuildConnectionString();

smAccess.getSchemaTable();

//Read data from Passwords table

dtAccess = smAccess.GetTableByName(“Passwords”);

}

2. After reading data from files, we need to change the tables structure to match the destination table structure:

smPasswords.ChangeTableStructure(ref dtExcel);

smPasswords.ChangeTableStructure(ref dtText);

smPasswords.ChangeTableStructure(ref dtAccess );

Importing to SQL

1. To create the Destination Table we used the following command:

string connectionstring = @”Data Source=.\SQLINSTANCE;Initial Catalog=tempdb;integrated security=SSPI;”;

smPasswords.CreateDestinationTable(con);

2. At the end, we have to insert data into SQL. There are two methods to achieve that:

(1) Insert using Bulk insert method

smPasswords.InsertToSQLUsingSQLBulk(dtExcel, connectionstring);

smPasswords.InsertToSQLUsingSQLBulk(dtText, connectionstring);

smPasswords.InsertToSQLUsingSQLBulk(dtAccess , connectionstring);

(2) Insert using stored procedure with table variable parameter

smPasswords.InsertToSQLUsingStoredProcedure(dtExcel, connectionstring);

smPasswords.InsertToSQLUsingStoredProcedure(dtText, connectionstring);

smPasswords.InsertToSQLUsingStoredProcedure(dtAccess , connectionstring);

Result