# Presentation: 123933

## DataShare

**Kate Fitta **
- NYC Mayor’s Office of the Criminal Justice Coordinator
**Jonathan Natarajan  **
- NYC Department of Technology and Telecommunications (DoITT)
**Doug Mamroth **
- iWay Software

**New York City’s **
**Enterprise **
**Integration **
**Platform**

## DataShare Overview – Progress to Date

- DataShare Architecture
- iWay Software the Role of Adapter Technology
- DataShare - Future Phases

## Kate Fitta

**Kate Fitta**
- NYC Mayor’s Office of
- the Criminal Justice Coordinator (CJC)

**DataShare Overview – Progress to Date**

## DataShare Overview

- The Mayor’s Office of the Criminal Justice Coordinator (CJC) and the Department of Technology and Telecommunications (DoITT) partnered in early 2004 to deliver DataShare 2.0 – the replacement for a mainframe system utilized by criminal justice agencies (DataShare 1.0) and NYC’s new integration platform.
- DataShare was designed for citywide integration, but deployed initially for 17 agencies (city, state, not-for-profits).
- The Public Safety Portal, which is the user-facing dimension of the City’s integration solution for criminal justice agencies, includes both administrative and business user functions.
- The initial phase concluded in July 2006 with the deployment of the Public Safety Portal and a set of GJXDM exchanges.
- This undertaking was a Mayoral initiative, but agencies did not have a mandate to participate

## Phase 1 – Facts & Figures

- 12 months for requirements gathering, solution analysis, systems integration procurement and solution selection
- 18 months to complete implementation – 2 major deployments
- 17 city, state and nonprofit agencies integrated via hub-and-spoke architecture
- 35 FTEs comprised the dedicated project team – City staff and consultants combined
- 60+ agency-side resources brought to bear during 1st phase
- $12M spent on professional services, HW, SW
- 100% of post prod support provided by City of New York upon Phase 1 completion

## Phase 1 – Scope and Deliverables

**Data Exchanges/Services**

*Agencies connected to new DataShare infrastructure and providing or receiving at least one data exchange via the hub or portal*

*Criminal justice data that is shared between and among participating agencies via the hub and portal*

*The user-facing dimension of the City’s integrated justice solution hosted by DoITT on CityShare*

**Agency Participation**

**Public Safety Portal Functionality and Links to Applications**

**Criminal Justice Integration Platform (Hub)**

**Phase 1 Providers**
- NYPD
- NYC DOC
- Criminal Justice Agency (CJA)
- OCA

**Phase 1 Receivers**
- NYPD
- Department of Probation
- Department of Correction
- District Attorneys (5)
- Special Narcotics Prosecutor
- CJA
- Law Department
- Dept. of Juvenile Justice (DJJ)
- OCA
- Legal Aid Society
- Assigned Counsel Plan
- HRA
- DoITT (on behalf of CJIS)

*The behind-the-scenes, criminal justice segment of the Citywide integration environment hosted by DoITT*

**Core Hub Features & Functionality**
- Push functionality
- Pull functionality
- Query application
- Notification application

**Phase 1 Business Deliverables**
- NYPD arrest data push (legacy & GJXDM)
- NYPD arrest notification (GJXDM)
- CJA ROR interview push and pull (GJXDM)
- OCA supreme and criminal court actions push (legacy & GJXDM)
- DOC inmate information (legacy & GJXDM)
- DOC release notification (GJXDM)
- DOC jail status query (GJXDM)
- CJA NYSID – SSN match (legacy & GJXDM)

**Core Portal Features & Functionality**
- Functionality for agencies to subscribe to notifications
- Functionality for agencies to perform queries
- Functionality for agencies to pull data
- System administration for agencies
- Reduced sign-on (RSO) for DNA HITS application
- e911 application link
- webCRIMS application link from OCA
- e-Justice portal link from DCJS

**Phase 1 – Scope and Deliverables**

## Decisions about DataShare architecture needed to consider the autonomy of agencies

DataShare had to be architected and designed in such a way that it could be scaled for citywide integration in future phases

DataShare was intended to serve as a pass-through system, with minimal data stored centrally

The deployment of DataShare 2.0 and decommissioning of DataShare 1.0 should have minimal impact on agencies and result in no loss of functionality for agencies.

The City decided to adopt GJXDM as its transport standard for criminal justice services

- DataShare had to be architected and designed in such a way that it could be scaled for citywide integration in future phases
- DataShare was intended to serve as a pass-through system, with minimal data stored centrally
- The deployment of DataShare 2.0 and decommissioning of DataShare 1.0 should have minimal impact on agencies and result in no loss of functionality for agencies.
- The City decided to adopt GJXDM as its transport standard for criminal justice services

**Guiding Principles & Strategic Decisions**

## GJXDM Strategy & Approach

- City decided to adopt GJXDM for all future criminal justice exchanges
- City decided to implement both legacy and GJXDM versions of existing exchanges via new platform to facilitate agency transition and minimize risk
- GJXDM SME consultants worked with provider and receiver agencies to develop schemas and leverage national model as appropriate.
- 30 representatives from 14 agencies attended GJXDM training in 2005
- City-State collaboration a priority for 2007
- GJXDM workgroup taking shape among stakeholder agencies to drive strategy going forward, especially collaboration with state and national initiatives;  DOP, NYPD, DOC to play key leadership roles.
- City will post IEPDs for Phase 1 exchanges this fall and continue to stay coordinated with national efforts

## Public Safety Portal Services

_**DOC Jail Status Query**_** – **This function enables users to enter a NYSID, book and case number or a combination of first name, last name and date-of-birth via the Public Safety Portal and retrieve the incarceration status and history for the individual(s) that match the criteria.  Jail status data includes information about the inmate's current incarceration status, incarceration history, warrant history and DOC housing history from 1999 - present.
_**CJA Interview Pull**_** –** This function enables users to enter a docket number or arrest number via the Public Safety Portal and retrieve the details of the corresponding interview the Criminal Justice Agency conducted with the defendant prior to their criminal court arraignment. Interview data includes information about the defendant's employment, residence, education and criminal history and is only available for interviews conducted from 2005 - present.
_**Subscriptions for NYPD Arrest Notification (Basic)**_** -**  This function enables business users to enter a NYSID, name information or location information via the Public Safety Portal and subscribe to be notified via email when an arrest event happens relative to the individual or location specified by the criteria.
_**Subscriptions for NYPD Arrest Notification (Advanced)**_** -** This function enables administrators to enter criteria corresponding to up to 150 arrest fields via the Public Safety Portal and subscribe to be notified via email when an arrest event happens relative to the specified criteria.
_**Subscriptions for DOC Release Notification**_** - **This function enables users to enter a NYSID or arrest number or book & case number via the Public Safety Portal and subscribe to be notified via email when the corresponding inmate is released from or admitted to City correction custody.
_**Links to Applications**_ - DNA HITS, eJustice, e911, WebCRIMS

**Public Safety Portal Services**

## New GJXDM Exchanges

_**CJA Interview Push**_** - **The CJA  Interview Push is a new, GJXDM data exchange that was first deployed through DataShare 2.0 in July 2006. In conducting interviews with criminal court defendants prior to arraignment, the Criminal Justice Agency collects information about the defendant's employment, residence, education and criminal history and submits it as a report to judges to help them make their bail decisions.  CJA sends this information to DataShare 2.0 post-arraignment as a single daily batch (Tuesday – Saturday at 2am EST) of all interviews.
_**NYPD Arrest Push**_** - **The NYPD Arrest Push is a new, GJXDM version of a legacy exchange that had available for years through DataShare 1.0.  It was first deployed through DataShare 2.0 in July 2006 and contains information about arrest incidents on a near real-time basis.
_**DOC Inmate Push**_** -**The DOC Inmate Tracking Push is the new, GJXDM version of a legacy exchange that had been available for years through DataShare 1.0. It was first deployed through DataShare 2.0 in July 2006 and contains data about DOC's entire inmate population: basic assessment information gathered at intake, information about inmate movement among DOC facilities, and a record of discharge. The DOC Inmate Tracking Push is a daily batch provided 7 days/week (2am - 6am EST).
_**OCA Court Actions Push**_** - **The OCA Criminal Court Action Push is the new, GJXDM version of a legacy exchange that had been available for years through DataShare 1.0. It was first deployed through DataShare 2.0 in July 2006 and contains criminal case information from OCA's CRIMS system including court dates, disposition, sentence information, judge name, and final information about case charges.  The OCA Criminal Court Action Push is a daily batch provided 7 days/week (12am - 2am EST).
_**CJA NYSID/SSN Push**_** - **The CJA NYSID/SSN Mapping Push is the new, GJXDM version of a legacy exchange that had been available for years through DataShare 1.0.  The exchange was first deployed through DataShare 2.0 in July 2006 and contains a mapping of defendants' NYSID, arrest number, and social security number. CJA sends the NYSID/SSN Mapping Push to DataShare 2.0 as a weekly batch (Wednesdays at 7am EST).

**New GJXDM Exchanges**

## DataShare

**Juvenile Case Management System **
**Law Department**
**Dept. of Probation**
**Dept. of Juvenile Justice (DJJ)**

**Richmond County DA**

**Richmond County DA**

**Public Safety Portal**

**Providers**

**Special Narcotics Prosecutor**

**Queens County DA**

**Kings County DA**

**Bronx County DA**

**Legal Aid Society**

**Manhattan DA**

**Criminal Justice Agency**

**Dept. of Corrections**

**NYPD**

**18B Web (Assigned Counsel)**

**Office Court Administrator **

**DataShare – Current Landscape**

**Receivers**

**Public Safety Portal**

**Richmond County DA**

**DataShare**

## 
Jon Natarajan

NYC Department of 
Technology & Telecommunications (DoITT)

**Jon Natarajan**
- NYC Department of
- Technology & Telecommunications (DoITT)

## Information Sharing Capabilities 

- The DataShare & Public Safety Portal solution facilitates the sharing of information across city agencies providing business process integration for the participating agencies.
- DataShare will provide the following services related to enterprise applications integration (EAI)
- Business process routing (via publish/subscribe)
- Data standardization
- Web-based portal that can offer truly distributed  business applications
- Business process auditing capabilities
- Field level filtering for messages
- Security
- 24/7 Monitoring

## Existing Systems Installed Base

- MVS
- VAX/VMS
- LINIX
- UNIX
- Microsoft
- FTP
- SMTP
- CICS Transactions
- Flat Files

- DB2
- ADABASE
- DATACOM
- VSAM
- ADMINS/VAX
- RDB
- CACHE
- MUMPS
- SQL SERVER
- ACCESS
- ORACLE
- EXCEL

## Standards-based (J2EE, LDAP, SQL) 

Enterprise Service Bus (WebSphere MQ and Broker)

GJXDM (iWay Transformation)

Public Safety Portal (Vingette & BEA WebLogic )

Involve the agencies that know the data

Involve the agencies that know the processes

Use existing City application implementation standards

- Enterprise Service Bus (WebSphere MQ and Broker)
- GJXDM (iWay Transformation)
- Public Safety Portal (Vingette & BEA WebLogic )
- Involve the agencies that know the data
- Involve the agencies that know the processes
- Use existing City application implementation standards

**Meeting The Needs – New Environment**

## Technical Architecture Requirements

- Security – Authentication, Authorization, Encryption, LDAP
- Performance – Handle 100’s of messages/second
- Scalability – Able to add new data exchanges easily
- Availability – 99.9% availability as ‘crime takes no holiday’
- Extensibility – Expose functionality via different methods like Web services
- Standards – J2EE, LDAP, GJXDM

## DataShare: The Solution

- The Solution consists of 6 key components

## Architecture & Components

**Enterprise Service Bus (DataShare 2.0 integration message broker)**
- Integration layer provides message brokerage/routing and integration business services for citywide agencies.
**Public Safety Portal**
- Portal provides front-end Web applications (such as notifications and queries), user administration, and reports to the agency user community.
**Notification Services**
- The DataShare 2.0 notification services provide the Public Safety Portal users and the criminal justice agency backend systems to subscribe to key transactions traversing the Enterprise Service Bus as notifications.

## Agency Adapter Platform
Agency iWay adapters and MQ used as the gateway to and from the criminal justice agencies and the enterprise service bus hosted at DoITT

Database
DataShare 2.0 database is used to store application and transactional metadata.

Novell eDirectory (DoITT’s enterprise directory server) 	
Enterprise directory server stores citywide agency users and groups. The eDirectory server will be used to provide services such as distributed provisioning, authentication and authorization.

- Agency iWay adapters and MQ used as the gateway to and from the criminal justice agencies and the enterprise service bus hosted at DoITT
**Database**
- DataShare 2.0 database is used to store application and transactional metadata.
**Novell eDirectory (DoITT’s enterprise directory server)**
- Enterprise directory server stores citywide agency users and groups. The eDirectory server will be used to provide services such as distributed provisioning, authentication and authorization.

**Architecture & Components**

## Agencies opted for one of two delivery methods:
Flat File
Database

Agencies that opted for flat file delivery then chose the method for inserting data in the file:
Populate data elements in the XML format, which preserves intact the format and contents of the XML
Populate data elements following a comma separated value (CSV) format or other appropriate convention

Agencies that opted for database delivery then chose the method for inserting data into the database:
Insert the entire XML as is in the database (e.g. as a character large object or CLOB), which preserves intact the format and contents of the XML
Convert the XML elements into columns in a relational or flat table and insert each XML record as a row in the table

- Flat File
- Database
- Agencies that opted for flat file delivery then chose the method for inserting data in the file:
- Populate data elements in the XML format, which preserves intact the format and contents of the XML
- Populate data elements following a comma separated value (CSV) format or other appropriate convention
- Agencies that opted for database delivery then chose the method for inserting data into the database:
- Insert the entire XML as is in the database (e.g. as a character large object or CLOB), which preserves intact the format and contents of the XML
- Convert the XML elements into columns in a relational or flat table and insert each XML record as a row in the table

**GJXDM Adaptor Approaches at Agencies**

## DataShare Leverages DOITT’s CityNet

## 
Doug Mamroth

iWay Software

**Doug Mamroth**
- iWay Software

## Enables connectivity to the integration server
from applications, 
application environments, and/or 
databases to an integration server platform

Provides the flexibility to integrate with 
the presentation
business logic
database

Exposes agencies systems for exchanging data in both directions

Provides an architecture that enables separate applications to work together 

Allows agency applications to be enhanced, added or removed without affecting others

- from applications,
- application environments, and/or
- databases to an integration server platform
- Provides the flexibility to integrate with
- the presentation
- business logic
- database
- Exposes agencies systems for exchanging data in both directions
- Provides an architecture that enables separate applications to work together
- Allows agency applications to be enhanced, added or removed without affecting others

**The Role of Global Justice XML**

## What iWay Initially Saw In DataShare

**Each Agency needed to provide the ability to:**
**Make requests (“Pull”) – Services oriented**
**This requires Request/Reply**
**Exchange data (“Push”) – Data centric**
**This requires Data send/receive**
**Take part in process – Process centric**
**This requires Event listener/emitter**

**  ****The combination of the above is the EAI**
**	****      ****It demanded a new form of B2B**
**	**** ****Protocol not commonly available - GJXML**

## Talk to the adapter using a variety of services (SOAP, FTP,        HTTP, . . . ) not the system
Transforms information from an existing format to GJXDM and publishes as a Web Service 
Creates WSDL in the process
Stores all services in a common directory
GJXDM built around an B2B EDI model
Transaction Type
Standard Elements
Provides “Wiggle Room”
Best Implemented Using Trading Partner Agreements

- Transforms information from an existing format to GJXDM and publishes as a Web Service
- Creates WSDL in the process
- Stores all services in a common directory
- GJXDM built around an B2B EDI model
- Transaction Type
- Standard Elements
- Provides “Wiggle Room”
- Best Implemented Using Trading Partner Agreements

**iWay’s Approach – Adaptive Framework**

## Trading Partners

- Every Agency Is Different
- Different Agencies Want Different Versions (GJXDM 3.0, 3.1, NIEM)
- Different Agencies Have All Of The Data (In Complete Records)
- Different Agencies Have Different Security Requirements
- Different Agencies Have Different Privacy Policies
- Different Agencies Have Different Backend Systems
- Different Errors Must Be Handled In Different Ways
- Thousands Of Messages Will Be Coming From Many Partners, Many Different Sources
- Think Of Every Agency You Exchange Information As A Trading Partner
- Always Consider Your Partners Capabilities

## Real-Time Processing To Handle The Differences

**Message Manager**:
- the history of a message

**Correlation Manager**:
- expectations for a message

- Names of transforms
- Process flows
- Pipelines
- Access keys (i.e. ISA06)

- Basic MOU
- Profile
- Formal names
- Error handler contacts

## Adapter Technology

**Doug’s material**

## Agency Adapter Platforms

- Enables connectivity from applications, application environments, and/or databases to an integration server platform
- Provide the flexibility to integrate with the presentation, business logic or database tier of an enterprise application
- Act as an extension to an enterprise application, providing a flexible solution for exposing external GJXDM messaging interfaces which can be leveraged for exchanging data in both directions
- Provide an architecture that enables separate applications to work together in a de-coupled fashion such that applications can be easily added or removed without affecting others
- Adapter is based on iWay Software

## Phase 2 Objectives

Related Integration Initiatives

Technical Enhancements

- Related Integration Initiatives
- Technical Enhancements

**DataShare – Future Phases**

## NYC Objectives for Integration in P2

- 1)  **Expand List of Participating DataShare Agencies**
- NYS Division of Criminal Justice Services (DCJS)
- NYS Office of Children & Family Services (OCFS)
- NYC Department of Investigation (DOI)
**2) Expand List of Services Available Via DataShare & Public Safety Portal**
- Integrated Juvenile Justice (IJJ) exchanges
- eArraignment exchanges
- Assorted other exchanges and queries
- 3) **Enhance Existing Shared Applications:**
- DNA HITS
- 18B Web
- e911
**4)  Develop New Shared Application - eArraignment**

**NYC Objectives for Integration in P2**

## DataShare & Related Integration Initiatives

| Application/
Initiative | Description | Agencies Involved | Future Integration via DataShare |
| --- | --- | --- | --- |
| eArraignment | New shared application to automate assembly of information between arrest and arraignment of defendants | NYPD, DA’s, Courts, Criminal Justice Agency, NYS Division of Criminal Justice Services (DCJS) | Application and relevant exchanges via DataShare (arrest info, criminal history, complaints, defendant interviews) to be developed in 2007 |
| DNA HITS | Shared application that facilitates case management across multiple agencies when DNA profiles match City or State databases | NYPD, DA’s, Chief Medical Examiner, NYS Division of Criminal Justice Services (DCJS) | Services provided via DataShare (jail status query, arrest and release notifications, criminal history, PSI, defendant interviews) will be integrated with the DNA HITS application |
| Integrated Juvenile Justice | Targeted initiative aimed at enhancing integration among agencies that deal with juvenile delinquents and offenders | Dept. of Juvenile Justice, Law Department, Dept. of Probation, Family Court, NYS Division of Criminal Justice Services (DCJS), NYPD
 | A set of new exchanges and services will be developed and deployed via DataShare to support this initiative in 2007. |
| e911 | Shared application that allows DA’s and Law Department to request recordings of 911 calls from NYPD | NYPD, DA’s, Law Department | XML requests will be submitted by some agencies via DataShare in 2007. |

## NYC Integration - Future Landscape

## NYC Integration – P2 Scope and Deliverables

**NYC Integration – P2 Scope and Deliverables**

**Phase 2 (targeted)**

**Phase 1**

## Technology Enhancements

- The Integration Service Offering solution has been built on flexible infrastructure, software & hardware, with expandability in mind.  As technologies and user needs evolve, the Enterprise Service Bus and its supporting J2EE infrastructure will enable new levels of integration through the use of solutions such as:
- Web Services (centralized web services registry)
- Distributed / federated applications
- Business activity monitoring (BAM)
- Workflow
- Business process choreography (BPC)
- As various City agencies see the true value in this solution their needs and requirements will evolve. This constant evolution process will inform / guide DoITT in its decisions to provide the above mentioned solutions or functionality.

## Q&A