# Presentation: 708078

## GemFire Enterprise Data Fabric

*GemStone Systems Inc.*

**Michael I Lazar, Federal Technology Director**
- _[Michael.lazar@Gemstone.com](mailto:Michael.lazar@Gemstone.com)__   301 325 8405_
**Steve Rixse, Director Public Sector**
- _[Steve.Rixse@Gemstone.com](mailto:Steve.Rixse@Gemstone.com)__  301 664 8494_

## GemStone at a Glance

**GemStone at a Glance**

- Founded 1982, with over 200 installed customers, in the Global 2000. Includes Bear Sterns, JP Morgan, Merrill Lynch, Northrop Grumman, NCI
- Unique and patented expertise in object management, high performance caching and data distribution technologies.
- Profitable, well funded and experienced management team.
- Office Locations - Beaverton OR,   Santa Clara CA, New York City, Washington DC;  60% in R&D, 20 % in Sales and Marketing
- Dedicated Federal and Financial teams
- 24x7x365 global support.

**Notes:**
1.  Our heritage in object management, IP, etc.
2.  Dedication to finance vertical
3.  Our strong fiscal position compared to competitors
4.  Operation capabilities to provide high QoS

## What do we do ?

- Distributed Operational Data Infrastructure
- Not just a distributed cache
- Key semantics of a database – ACID
- Key semantics of a Message bus
-  Enable data sharing and event notifications
- At memory speeds
- Enable apps to continuously analyze and react to very fast moving data

_**A Data Fabric for the Enterprise**_

**Notes:**
Setting the context
  -- customers usually think of us as a distributed caching vendor
  -- we think we are the next generation operational data infrastructure
    ... best of database semantics and messaging
  -- D caching is common --- use JBoss cache
  --  Focus areas:
  	main memory distributed data management – scale/perf
	reliable data routing and messaging
	continuous data analysis – active database
 -- Recent traction from forrestor  -- information fabric

## What do we do - GemFire Value proposition

- Enable notification of data changes to users and other applications
- Create framework for high performance data access
- Scale applications to meet unpredictable demands for information
- Boost performance across applications without increasing other hardware/software requirements
- Reduces network load and can work over low bandwidth networks
- Enhance the performance, scalability, and network characteristics of other software.
- Single product with multiple uses, easy to implement and little to no management requirements
- Standards compliant interfaces

**Notes:**
Alert Notification as data changes in real time memory driven
Application to memory to application vs. Application to disk to application
Scaling today means more boxes and more licenses without creating an elastic fabric – queries and collaboration can drive today’s systems in the ground when some event arises
NW load is an issue for your customers and deployed assets
One product, multiple uses, multiple benefits!
Cost savings

## GemFire Enterprise

**GemFire Enterprise  (GFE Java and GFE C++)**

**A scalable and high-performance software data infrastructure that**
-   Caches and distributes data in multiple formats (Java, C++, XML) across distributed applications
-   Connects to backend systems like databases, messaging systems, etc.
-   Provides high data availability mechanisms
-   Supports distributed transactions and standards-based querying
-  Highly instrumented

**GemFire Real-Time Events**

**In memory complex event processing (CEP) solution that facilitates agile, event-driven architectures by analyzing streams of data in conjunction with other static/historical data sources**
-  JDBC and SQL based, with Continuous Queries
-  Client side materialized view
-  Messaging paradigm with connected and disconnect clients
-  Highly instrumented

## EDF - Where Does it Fit?

**Mainframes**

**Historical Data**

**Customer  Data**

**Enterprise Data Fabric**

**Distributed Caching **
**High Availability  **

**Custom Application **
**Data Sources**

**Event Processing**
**Data distribution  **

**Data Modeling **

**Data Access **

**Application Access**

## What is GemFire EDF - Key Features?

**What is GemFire EDF - Key Features?**

**Virtualize data and events **
**from many sources**
**Manage and partition**
** ****data in **_**distributed **_
_**main-memory **_**for **
**blazing performance**

**Provide data access for**
**applications in multiple**
**formats to support**
**heterogeneity**

**High-speed data **
**distribution **
**(TCP/IP, Multicast) **
**across**
**multiple nodes**
**- Move data on demand**

**Replicate data to mirror**
**nodes synchronously for**
**high availability**

**Overflow or persist to **
**disk to scale **
**to large data volumes**
**and for data recovery**

**Standards-based **
**Querying**
**and Continuous **
**Querying on structured **
**data and streaming data**

**Distributed transactions**
**for reliable operations**

**Event**
**Streams**

**ENTERPRISE DATA FABRIC**

## SOA – Composite Applications

**GEMFIRE ENTERPRISE DATA FABRIC**

**Java, C/C++/C#, XML:DB, SOAP Interfaces**

**Transactions,  Logging, Authentication, Discovery Session Management,    Entitlements**

**ESB**

**Business Process**

**Composite Applications:  e.g,, Web Portals**

**File systems**

**Data Warehouses**

**Databases**

**Custom Application **
**Data Sources**

**Process**
**State**

**SOA**** – Composite Applications**

** ****Authentication**
** ****Session/Process**
**  ****State Management**
** ****Reference data access**
** ****XML Data Transformation**

##  SOA - Composite Processes

** ****SOA**** - Composite Processes**

**Web Service 1 **

**Composite**
**Application Flow**

**User/Process**

**Web Service 2 **

**Web Service 3**

**Web Service N **

** ****Authentication**
** ****Session State Management**
** ****Reference data access**
** ****XML Data Transformation**

**Web Service 1 **

**Web Service 2 **

**Web Service 3**

**Web Service N **

##  SOA - Authentication

** ****SOA**** - Authentication**

**Web Service 1 **

**Composite**
**Application Flow**

**User/Process**

**Web Service 2 **

**Web Service 3**

**Web Service N **

**Web Service 1 **

**Web Service 2 **

**Web Service 3**

**Web Service N **

**Identify and Authenticate  - Get Authorizations (IA&A)**
**	****Travel over network**
**	****Calls Policy Server/LDAP repository**
**	****Calls RDBMS**
**Repeat for each step in application flow**
**Often more time than actual service step**
**GemFire eliminates redundant lookups**

**This is also used for **
**Authentication Management for Enterprise Portals**

##  SOA - Session/Process State Management

** ****SOA ****- Session/Process State Management**

**Web Service 1 **

**Composite**
**Application Flow**

**User/Process**

**Web Service 2 **

**Web Service 3**

**Web Service N **

**Web Service 1 **

**Web Service 2 **

**Web Service 3**

**Web Service N **

**In SOA process state is either**
**	****Returned as XML between steps  **
**	****	****Often stored by workflow manager in RDBMS**
**	****Stored in RDBMS by state aware services**
**GemFire removes latency introduced by network and RDBMS**
**	**

**This is also used for **
**Session State Management for Enterprise Portals**

##  SOA – Reference Data 

** ****SOA**** – Reference Data **

**Service calls back-end data source to **
**look up same information over and over**
**GemFire removes need to make redundant calls to back end **

##  SOA – XML data transformation

** ****SOA**** – XML data transformation**

**Service returns same information over and over **
**(for a period of time) If it is called with the same request**
**GemFire removes need to retrieve data from back end, and**
**transform it into XML.**

##  SOA – XML data transformation

** ****SOA**** – XML data transformation**

**Service returns same information over and over (for a period of time) **
**If it is called with the same request**
**GemFire removes need to retrieve data from back end, and**
**transform it into XML.**

**GemFire also provides**
** ****Optimized XML data representation**
** ****Less Memory**
** ****Less Java GC/overhead**
** ****Optimized XML data access**

## GemFire Design Patterns

**GemFire Design Patterns**

- 	Application Survivability and Network Usage Reduction

- App Server

- Web Services
- Applications

- App Server or SOA applications

- Wide Area Network

- Wireless
- App
- Server

**     ****GemFire **
**Cache**

## GF Real Time Events Functional Overview

- GemFire Real-Time Events

**GF Real Time Events Functional Overview**

## Summary Highlights

- Solve numerous challenges in SOA and distributed systems environments
- Enhance Scalability, Performance, Reliability of systems
- Potentially reduce hardware and software costs
- Easy to insert into new and existing systems
- Highly Available, Highly instrumented, Highly configurable
- NSA Acquisition Security approved

## EDF - Key Technical Concepts

*GemStone Systems Inc.*

- This presentation contains information that is CONFIDENTIAL to GemStone Systems Inc.
- Do not distribute or copy this presentation without express written permission.

## GemFire Enterprise

**GemFire Enterprise**

**A scalable and high-performance data infrastructure that**
-   Caches and distributes data in multiple formats, across distributed
-       applications
-   Connects to backend systems like databases, messaging systems, etc.
-   Provides high data availability mechanisms
-   Supports distributed transactions and standards-based querying
-   Offers interfaces for C/C++/C# applications as well

- Target Environments
** **Java/J2EE based platforms
-  XML/Web Services

## GemFire Enterprise -- C++

**GemFire Enterprise -- C++**

**An extremely high throughput and performance data infrastructure that**
-   Native C++ bindings to avoid inefficiencies
-   Highly parallel distribution with optimized transport
-   Avoids unnecessary user-kernel crossings with no contention or
-       context-switching
-   Connects to backend systems like databases, messaging systems, etc.
-   Provides high data availability mechanisms
-   Offers interfaces for Java applications as well (future)

**Target Environments**
-  C++ -based application environments
-  Performance and throughput are primary concerns

## GemFire Real-Time Events

**GemFire Real-Time Events**

**GemFire Real-Time Events is a complex event processing (CEP) solution that enables an enterprise to facilitate agile, event-driven architectures by analyzing streams of data in conjunction with other static/historical data sources**
-   Define relevant business events through simple SQL queries
-   Discern patterns through continuous querying and event
-       correlation with other data sources
-   Distribute appropriate information to selective client applications

**Target Environments**
-   Fast-moving data streams
-   Real-time pattern detection and analytics

**GemFire Real-Time Events**

## Enterprise Data Fabric - What is it ? 

- Middle tier Distributed Data Infrastructure
- Operational Data
- Combines Distributed Caching (main-memory), DB semantics, Reliable Messaging/ Intelligent Routing and Continuous Analytics

**Distributed Data Space**

- Data warehouses

**Applications**

- Rational databases

**Notes:**
Middle tier high performance Distributed data management 
	For operational data  (current working set, fast moving data and not historical )
	Combines Distributed caching (main-memory) + DB semantics + reliable messaging and intelligent routing, + continuous analytics (let us leave this for the moment, CQ will be explained later)

Think of it as a distributed data space where all operational data required NOW by applications is available. It is a virtualized data space – i.e. the location of the data typically can be transparent. Its most important characteristic is that it is fast – primarily based on main-memory for managing data. 
It provides distributed caching features – managing objects and distributing these to other processes, but also includes DB semantics such as being able to index and query the data set, engage in transactions and disk persistence + recovery. 
Pub-sub semantics and reliable message delivery guarantees like a messaging system and finally “continuous analytics”. As often, this data infrastructure is used for fast moving data, increasingly apps need a way to intelligently detect patterns of interest and then be told as the changing data effects their interest. 

## Where is the data in the Fabric? 

- All across the Network
- NETWORK IS THE DATABASE

**Cache**

**Cache**

**Cache**

**Notes:**
Data is spread across the network
	Data maybe local to your process, to your machine, copies in multiple machines, spread across machines, on disk or even simply accessed lazily from some enterprise repository
	One way to think about this : Network is the Database
	Or it is an abstraction that combines data in many nodes or from many repositories makes it visible as it was just one logical database

## Enterprise Data Fabric -  Physical View 

- All across the Network
- NETWORK IS THE DATABASE

**Cache**

**Cache**

**Cache**

- Rational
- databases

**Clustered Application**

## 5 Key Features

- 1. Data Storage
- Optimized concurrent main-memory structures
- Topology
- Embedded in application process
- Client caches and Cache servers
- Partitioned across many servers
- n-way in-memory replication for HA
- On Disk
- Overflow to disk
- Recoverable disk regions – Asynchronous or Synchronous

## 5 Key Features

- 2. Access Model
- Object Caching API (get()/put()).
- Querying through Xpath, OQL and SQL
- XML /Web Services (HTTP/SOAP)
- APIs in Java, C++, and C#
- JTA Transactions
- Management APIs – JMX

## 5 Key Features

- 3. Data Distribution Model
- Consistency Model
- Distribution without ACKs
- With ACKs
- With global locking
- Reliable Pub-Sub semantics
- Novel declarative “role” based model to ensure message delivery
- Dealing with Slow subscribers, net/node failures, etc
- Multiple transports
- TCP, Reliable UDP MultiCast

## 5 Key Features

- 4. Integration with External Data Sources
- A Framework for read-through, write-through, and write-behind
- Custom Connectors link to DBMS, Middleware Messaging, EII solutions
- Components - Loaders, Writers and Listeners

## 5 Key Features

- Dealing with high event rate
- RFID streaming data, etc
- Many clients can subscribe to portions of the Data Fabric
- Express interest using complex query
- CQ engine continuously analyses data events
- Continuously calculates how the query view has been impacted
- Ships “delta” and merges into client result set.
- Notifications delivered with guaranteed latency threshold

- 5. Continuous Analytics