Rep Connect
Tags: Reputation, Integration, Xml, Data Driven, Json

“In God we trust. All others must bring data” — Edwards Demin

Introduction

In this day and age that we are living in where specialized cloud platforms are sprouting and helping customers to run their business (like Salesforce for CRM, Workday for HR, Atlassian for work management etc.) it is imperative for any cloud platform to provide a robust and a scalable way to bring data into their system and also provide an efficient way to take data out. No system can afford to live in a silo. Reputation cloud platform is no exception to this.

As we spread our footprint deeper into the Customer Experience (CX), Reputation Management (RX) space; it has become imperative to have a robust framework to ingest data into our platform from various sources. Keeping this in mind, we ventured to build our next-gen Integrator framework.

Enter Rep Connect: “One-stop solution for a robust, scalable and self-service data ingestions”.

Evolution of Rep Connect

Reputation platform is driven by customer data — be it feedback, reviews, survey results, business profiles, metrics etc. The platform has a robust way of collecting a lot of unstructured data like reviews from various sources like Facebook, Google, Instagram etc. But customers want to bring their own data into the platform from their own sources and we have to provide a mechanism for the same. Before building out this feature — we need to answer 3 questions viz -

What data should be ingested?

When should it be ingested?

How to format the incoming data?

Answering the above questions and building a solution around it is the genesis of Rep Connect within the Reputation platform. Now let us answer the questions deeper.

What data?

The various modules within the Reputation platform require different types of data. For-instance the Business Listings module requires profiles data. Review Request module requires customer data from CRM systems to send survey or review requests. The Survey module requires survey results data and so on and so forth.

When?

Data can be ingested at varying frequencies. Business profile data can be ingested once a week since it mostly involves information that doesn’t change frequently like address, phone number, work hours etc. Review request uses customers data that changes daily. Survey data could be ingested daily. So to answer the question — we need to provide a flexible recurring schedule for customers to bring in data.

How?

Customers bring in data as CSV files which they host on an FTP site or drop it in our FTP servers. Some customers want us to directly pull their data from their CRMs or some provide their own web service endpoints which we have to connect and get data.

As customers bring in data from their systems in various formats and structures, one mandatory functionality we have to provide is a way to transform and map the data into our data models. Transformations can be a simple mapping for example from field X from customer file to field Y in our source table. Customers also require the ability to write their own scripts to massage the data or apply simple transformations like converting the case of a field or concatenating two fields, replacing values etc. before importing the data.

Now that we have a clear answer for What/When/How — we can now lay the foundation of requirements for Rep Connect —

Ability to integrate with various end-points like SFTP, REST, SOAP end-points or other systems like Salesforce, Yardi, Sitelink, CDK etc and download data. Ability to process data in 3 different formats viz. CSV, JSON & XML. Ability to schedule data imports at periodic intervals. Ability to transform and map the data from source to the relevant format inside Reputation.

Apart from the above to build a robust system we also need the following capabilities

A notification system which customer can subscribe to and get notified every time data is ingested into the platform. Ability to send digests per day or week or send notifications only when something fails or any other user-defined conditions. Services like Business listings, Surveys, Review Requests provide their own API to import data into their schemas. Rep Connect can understand the APIs provided by the services and directly call them. This would cause hard coupling between Rep Connect and the target services. The alternate approach we took was to put an intermediary service which takes input from the Rep Connect and does the job of figuring out which service to call and make the relevant call. This would adhere to design principle of Separation of Concerns and make Rep Connect independent of the business services and more importantly makes Rep Connect stateless.

Decomposing to services

Microservices are self-contained software components that are independently deployable focused on specific business capabilities running in an automated environment. When we started to decompose the above requirements into microservices, we arrived at something like this

Figure (a) — Rep Connect Architecture

We have come up with 3 connector services — each dealing with one type of connection — for FTP, REST & SOAP service. Each connector takes only one responsibility — the FTP connector, when passed in connection/credentials & directory details connects to the FTP server and downloads the file. REST connector when passed in API end-point, credentials connects to the REST endpoint and downloads the data. Same with the SOAP connector. All the three services after downloading converts the data into a common format which is consumed by Rep Connect (we decided CSV as the common format).

Rep Connect Service takes the data downloaded by the connector services and transforms it, does field mappings and makes it ready to be ingested by the platform. The responsibilities of the Rep Connect service are

a. Map source data fields to target fields

b. Transform the data using scripts or custom functions

c. Filter out data based on any criteria

d. Schedule the jobs that has to be run repeatedly on a user-defined schedule

e. Configure notification subscriptions

Apart from the connector services and Rep Connector service three more services play an important role. They are

Scheduler

The Reputation platform has a scheduler service that supports scheduling of jobs, provides retry mechanism in case of failures and a good monitoring on top. So instead of reinventing the wheel, Rep Connect leverages the common scheduler available in the platform.

Notification

Notification is also one of the core services in the Reputation platform that provides ability to set up message templates, manage subscriptions, manage preferences (digest vs non-digest) and support multiple delivery channels (like email, SMS, in-app etc.). Rep Connect would leverage the Notification service.

Importer

Importer is an intermediary service sitting between Rep Connect and other Business services (like Business listings service, Review request service, Survey service etc.). It takes the output from Rep Connect and calls the respective Business service API that does the actual work of importing the data.

Rep Connect Sequence

Figure (b) — Rep Connect Sequence

Conclusion

In this article, I presented the genesis and evolution of the Rep Connect module in Reputation platform. Today Rep Connect is getting used heavily to ingest data into the platform from various sources by our customers and it is making our (Engineering and Support teams) life easier as the entire setup, configuration and monitoring are owned by the customer in a truly self-service model.