# Presentation: 268865

## HPC Informatics (Titan)

- Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000.

*HPC Informatics (Titan)*

## HPC Informatics (Titan)

- The Titan project is a significant expansion of the open source Visualization ToolKit (VTK) to support the ingestion, processing, and display of informatics data. The Titan framework provides...
-  A Parallel Pipeline Architecture
-  Scalable
-  Flexible
-  A Large Set of Components
-  Database Drivers
-  Support for Tables/Trees/Graphs
-  N-way arrays (sparse and dense)
-  Linear Algebra
-  Tensor Methods
-  Graph Algorithms
-  Analytic Views
-  Statistical Charts and Plots

*HPC Informatics (Titan)*

- Client/Server over Corporate LAN

## Expanding the Architecture to Support Intelligence Analysis

***Query***
***- Select * from network_packet_table where date > 7/1/2008 and date < 7/8/2008***
***Data Transformations***
***- vtkTableToTree***
***- vtkTableToGraph***
***- vtkTableToSparseArray***
***- vtkTableToDenseArray****
***Algorithms***
***- Statistics***
***- Linear Algebra***
***- Tensor Methods***
***- Graph Algorithms***

- Parallel
- HPC
- Database

- Parallel
- HPC
- Database

- Data
- Transformations

*HPC Computation Platform*

- HPC
- Database

- Algorithms

- Layout and
- Rendering

**.**
**.**
**.**

- Presentation
- And
- Interaction

- Standard
- 100-T
- Network

- Query
- Database
- and
- Load Balance

***Layout and Rendering***
***- Tree Layout***
***- Tree Map***
***- Graph Layout***
***- Hierarchical***
***- Geodesic***
***Presentation and Interaction***
***- Client/Server***
***- Geometry and Image Delivery***
***- Cross Platform Qt User Interface***
***- Linked Selection***

## The Titan Toolkit

| Data Structures
 Table
 Tree
 DAG
 Directed Graph
 Undirected Graph
 Sparse n-way Array
 Dense n-way Array

Database Drivers 
 MySQL
 Postgres
 Oracle
 SQLite
 ODBC / Netezza

Readers
 Dimacs
 DOT
 GXL
 Chaco
 XML
 Tulip
 DelimitedText
 FixedWidth
 ISI, RIS

Multidimensional Analysis
 TPP / PARAFAC

MTGL Algorithms
 Community Finder 
 ST Search
 CSG Search
 Temporal Search
 | BGL Graph Algorithms
 Breadth First Search
 Connected Components
 Biconnected Components
 Brandes Centrality

PBGL Integration

MATLAB Integration

Statistics Algorithms
 Descriptive
 Order
 Correlative
 Contingency

Linear-Time Graph Layouts
 GSpace
 Hierarchical
 Clustered
 Three tree-based variants

Multiple View Types
 Render (3D)
 Graph
 Hierarchical Graph
 Tree
 Treemap
 Georeferenced

Multiple Platforms / Languages
 Windows, Linux, OSX, HPC
 Write components in C++
 Use with C++, Python, TCL, Java
 Use as OverView “plugins”

 |
| --- | --- |
|  |  |

## The NGC Prototype...

## Sunburst Data Ingestion and Processing Model

- Processing

- 1. Gather Packet Data
- (raw data, not suitable for analysis).

- 4. Analyst uses NGC Prototype to query data from Postgres database, conduct analysis and create views. Analyst can select among multiple views, analytical techniques to identify and explore patterns in data.

- Postgres
- Database

- Unstructured text (payloads):
- Emails, pdfs,
- images, word docs

- Structured network traffic
- (packets, timestamps,
- ports, protocols, etc)

- Inxight

- Sunburst

- SAS Scripts

- 2. Sunburst reassembles raw packets into legible traffic; dumps reassembled traffic into a relational (Postgres) database, which contains both structured and unstructured data.

- TCP Dump Data

- 3. Analyst parses unstructured text into linguistic elements (nouns, verbs, entities) using one of two tools. Processed data is stored in the database.

- Third Party
- Software

- LEGEND

- Data

- Data Store

**Sunburst Data Ingestion and Processing Model**

- Other Sandia
- Software

## NetView 0.4.1

***Flexing the Framework***
*** ******Databases***
*** ******Component Pipeline Architecture***
*** ******Statistics***
***- Tensor and Algebraic Methods***
***- Graph Algorithms***

***Ongoing challenges***
***- SQL ‘Magic’ Interface***
***- HCI/ Usability***
***- Cognitive Load***
***- Robustness Issues***
***- Closed Course / Professional Driver***

## Four Use Cases

**Which machines are talking to each other?**
**Which network transfers crossed political boundaries?**
**What ‘payloads’ were contained in the network transfers?**
**Analyst needs suspicious (out of the ‘norm’) behaviors “flagged” for further exploration**

## Use Case One: Which Machines are Talking to Each Other?

*Use Case One: ***Which Machines are Talking to Each Other?**

- Database

- vtkNetworkHierarchy

- vtkSQLQuery

- vtkTableView

- vtkTableToGraph

- vtkGraphView

- vtkHierarchicalGraphView

- Linked Selection

- vtkMTGLSTSearch

## Use Case Two: Which network transfers crossed political boundaries?

- Database

- vtkNetworkHierarchy

- vtkSQLQuery

- vtkGeoView

- vtkTableToGraph

- vtkGraphView

- vtkHierarchicalGraphView

- Linked Selection

- vtkMTGLSTSearch

## Use Case Three: What ‘payloads’ were contained in the network transfers?

- Database

- vtkPARAFAC + More

- vtkSQLQuery

- vtkGeoView

- vtkPARAFAC + More

- vtkTermView*

- vtkConceptTermView*

- Linked Selection

- Domain Mappers

**BTW: Mac Screenshot*** ***Awesomeness is free ******

- vtkTableToSparseArray

- * Just vtkGraphViews

## Use Case Four: Analyst needs suspicious (out of the ‘norm’) behaviors “flagged” for further exploration

*Use Case Four:** ***Analyst needs suspicious (out of the ‘norm’) behaviors ****“flagged” for further exploration**

- Database

- vtkSQLQuery

- vtkTableView

- vtkTableToGraph

- vtkGraphView

- vtkContingencyStats

## Demo!