# Presentation: 513343

## The Southeastern Regional Visualization and Analytics Center

**The Southeastern Regional Visualization and Analytics Center**
**NVAC Consortium Meeting**
**May 24, 2006**

## Southeastern RVAC Research Team

**UNC Charlotte**
- William Ribarsky
- Jianping Fan
- Robert Kosara
- Anita Raja
- William Tolone
- Jing Yang

**Georgia Tech**
- John Stasko
- Jim Foley
- Ashok Goel
- Beki Grinter

*Plus postdocs, several Ph.D. students, M.S. students, and undergrads*

## Southeastern RVAC Research Model

** ****Analytical Reasoning Structure**

**Raja, Fan, Yang**

**Goel, Raja**

**Foraging/Analysis Loop**

**Sensemaking Loop**

**Visualization Framework**
**Ribarsky, Grinter, Tolone**

**Visual Representations/Interaction**
**Stasko, Kosara,** **Yang**

**Notes:**
Overview of the Southeastern RVAC research and education structure and how the parts interrelate. Interactive visual representations are the connecting tissue and will serve every step of the foraging/analysis and sensemaking loops. Evidence gathered and Knowledge to be visualized will be selected and mapped according to the products from these loops.

## Automated Image Content Classification 

- Approach Overview

- Example: Images with “sea world” concept. Relevant salient objects such as “sand field” are detected.

## Exploratory Analysis: Large Scale Image Browser

- Users can select a sample image and select images similar to it
- Selected images can be visualized in different ways
- global MDS
- selected only
- Sequential order
- MDS of selected images

*-Yang, Fan, et al.*

**Notes:**
This is only a subset of exploratory interactions that can be performed. In clockwise order from top left, overall display of image clusters with “selection by example” using a sample image; selected images alone; reclustering of selected images; rainfall display with respect to a sample image (at bottom)—most relevant images fall first and least relevant last. MDS refers to multidimensional scaling.
Can scale up to many more images. Can add annotations (time, space, user-made annotations, such as in Flickr) and relate to image concepts. Can analyze for new concepts. Can even use the Value & Relation display in conjunction with the conceptual display. Eventually this will be part of a general multimedia exploratory analysis capability (e.g., collect and associate images, video, text, and sound from the Web and other sources without regard to media type).

- MDS image view   Search by sample image

- VaR content view   Search by content

**Exploratory Analysis: Large Scale Image Browser**

## Video Analysis: Story Boundary Detection 

**Automated Analysis of Broadcast Video **

## Video Analysis: Object Segmentation  

**Automated Analysis of Broadcast Video **

## Video Analysis: Object Tracking  

**Automated Analysis of Broadcast Video **

## Video Analysis: Multi-Modal Analysis  

- Visual Signals

- Auditory Signals

- Other Signals

**Multi-Modal Data Fusion**

**Automated Analysis of Broadcast Video **

## Full Multimedia Exploration and Discovery 

- Time

**Full Multimedia Exploration and Discovery **

- Story boundary detection

- Feature tracking & understanding

- Auditory & text understanding

- _Multimedia analysis process_

**Notes:**
Depiction of initial multimedia analysis tool. It shows keyframes for most important concepts reported over time. As time flows, most important topics flow from the center of the screen to either side. The user can click on any frame to see the video segment. Analysis process is entirely automated and language independent. Text analysis comes from closed caption review. We will be expanding to (eventually) hundreds of channels and hundreds of concepts. A visualization interface is being developed for this much larger scale exploration tool.

## Parallel Sets

## Parallel Sets

- Customer survey data
- Liquid detergents and tabs more popular in cities
- Families tend to buy (cheaper) powder
- People with higher income live in cities

**Notes:**
Handling of textual data
Better support for many dimensions
Integration with
Ontologies
Blackboard
Value and Relation display
Handling of large(r) data sets

## Application: Risk Management

- Bank of America Project
- This is a major activity because banks have a regulatory responsibility to find and report these activities.
- Non-compliance can be very damaging to reputation and very expensive (e.g., billion dollar fines for a large bank).
- The *exploration*, *discovery*, and *hypothesis-building* processes are similar to those used in *intelligence analysis*.

**Notes:**
Need to analyze overwhelming amounts of data; incomplete and may be unreliable.

## Application: Risk Management

**Application: Risk Management**

- Identify

- Prioritize

- Investigate

- Report

- All transaction activity

- Interactive
- Visualization

- Google

**Notes:**
Identify potential suspicious activity worthy of further investigation. Prioritize with risk ratings.

Make high dimensional, large scale visualization the overall user interface for the whole process.

## Wirewatch: Bank of America Anti-Money Laundering Project 

- Each keyword
- Row:
- A cluster of accounts
- Cell:
- The number of times a keyword occurs in the transactions of these accounts
- Distribution Graph:
- Histogram of occurrences of a keyword for each account in the cluster

**Wirewatch: Bank of America Anti-Money Laundering Project **

## Wirewatch: Bank of America Anti-Money Laundering Project 

- Distribution View:
- Each cell represents the distribution of keyword occurrences for that cluster of account.
- The brighter the color, the less evenly distributed the occurrences are within that cluster.

**Wirewatch: Bank of America Anti-Money Laundering Project **

## Wirewatch: Bank of America Anti-Money Laundering Project 

- Two cells of similar occurrences (343 and 306), but with very different distributions
- Cell on the top shows irregular distribution.  Notice a few accounts have higher occurrences than other.
- Cell below shows most accounts have 1 occurrence of the keyword.

**Wirewatch: Bank of America Anti-Money Laundering Project **

## Knowledge Capture and Organization

- Notional interactive ontology navigator

- The following ontologies have been built

- Geopolitical ontology
- Analysis tool concept ontology
- Beginnings of a general purpose ontology to connect tool concepts to user-selected concepts

- These ontologies will provide an interface between the analysis products and the reasoning system.
- Our exploratory visual analysis scheme will also work with lightweight knowledge structures or with none.

**Notes:**
The geopolitical ontology has all countries of the world, over 31,000 U.S. cities, and nearly 17,000 foreign cities. The scene on the right is from an interactive demo. The user places and moves the box around, getting the close-up view in the lower right panel and displaying representations within the box at the highest level of detail appropriate with the scale of the view. Eventually users will be able to do many other interactions. For example, selection of one or more node will show those nodes and the nodes most directly connected to them to be most important. Eventually rich, glyph-based visualizations will be displayed on the map view, and these interactions will be connected with other visualizations.
This structure and display has garnered much interest with the bankers. BoA wants to add risk information and such items as fraud and drug traffic to the properties for cities, for example.

## Integrative Multidimensional Visualization

- Example: a geospatial zooming coupled with a zooming into a glyph cluster

## Handling Multiple Source Collections of Evidence

- Organize reports and stories in an XML structure.
- Apply text analysis to bring out names and links to other names and locales.
- Organize in database for efficient search.
- Investigate various visualization methods.

**VAST Conference Dataset**
- 1200 reports and news stories in English;
- Several maps; Several photographs;
- Several types of numerical data in multiple formats.

**Handling Multiple Source Collections of Evidence**

*Graphs without Graphs*

**Notes:**
People, places, things from reports. Does report get a node? Graphs without the graphs. Different colors for different types. Another window with full report. Moving towards interactive graph layout. Work at a higher conceptual level? 

## User-System Collaboration (System Assist)

**Sensemaking**

**User-System Collaboration (System Assist)**

- Mixed Initiative

- Interactive Visual Representations:
- Knowledge; Reasoning Artifacts; Analyses

- Visual Interface

**User (Intelligence Analyst)**

**Notes:**
The autonomous controls in the mixed initiative system decide data, knowledge, and reasoning artifacts to show and which visualization to apply. All this could be overridden by the user.

System may take its own initiative, but user makes decisions at key points.

## Foraging/Analysis Loop

- Visualization Analysis
- Methods

- Execution Results
- from Environment

- Action Choices

- Data

- Hypotheses

- Models of
- Alternatives With
- Performance Profiles

- Models

- MDP-based
- Decision Control

- Images

- Video

- Trans-
- actions

- Reports

- Knowledge
- Structures

- Open
- Sources

- Multimedia or Other Analysis
- Methods

**Notes:**
Markov Decision Process 
Supports Sequential decision making process under uncertainty I.e optimization of not just the immediate utility, but the longer-term utility as well
- Leverages inherent uncertainty of environment

Naïve Bayes Classifier:
- Train classifier on previously encountered problems t Obtain performance of various solution modes for different queries
- Use performance profiles to predict expected performance of new problem instance and use that info to make decisions

Blackboard: 
- Multi-leveled database for the information discovered so far
Input data, partial solutions, various problem solving states
- Layered hierarchy 
allows for explicit modeling of concurrent goal driven and opportunistic data processing
Maintains clear evidential path for supporting and contradictory information

MDP = Markov decision process model. Given a set of hypotheses and the corresponding problem solving for the subsets of these hypotheses, the decision support component (DISC) will handle the following questions: which of the hypotheses should be verified to have a high certainty of solving the query; and, what steps should be taken to verify these chosen hypotheses, i.e., how many resources (time, processing power etc.) should be allocated to each step in the process given the deadline for solving the query. This involves determining which data sources should be accessed, whether data should be visualized, when it should be rendered and the parameters of the visualization.
When a query/hypothesis set arrives at the system, it will trigger the DISC for information foraging and analysis. The control flow for DISC is described in Fig. 3. The hypotheses are posted on a multi-level blackboard system[1] [2]. To support reasoning about time/quality/cost trade-offs, and thus a range of different resource/solution paths, the DISC contains problem solving models that are abstractions of the low-level execution model. These models enumerate multiple different ways to achieve the goals and describe them statistically in three dimensions, duration, quality, and uncertainty, via discrete probability distributions. The problem solving model associated with a hypothesis is then translated into a sequential decision making process[3] that accounts for the fact that, in resource-bounded, uncertain environments, decision choices made earlier in the process will affect the kinds of decision choices made later. [1] D. Corkill. Blackboard Systems. AI Expert, 6(9):40-47, Sept 1991
[2] V. Lesser, B. Horling, F. Klassner, A. Raja, T. Wagner, and X. Zhang. BIG: A resource-bounded information gathering agent. Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI-98), July 1998
[3] A. Raja, V. Lesser and T. Wagner. Toward Robust Agent Control in Open Environments. Proceedings of the Fourth International Conference on Autonomous Agents, pp. 84-91, ACM Press, July 2000

## Foraging/Analysis Structure

- Multiple analysis tools and data resources available.
- Must choose best data and analyses given time and resources available.
- Requires ontologies (key concepts in analyses and derived from data)

**Notes:**
Initial Foraging/Analysis structure. Using the forager interface, the user selects options w.r.t. data sources, time available, etc. The system suggests the best data sources and analysis tools. For example, there are multiple levels of abstraction for some analyses; the system will suggest which level to use. These suggestions can be overridden at any time. (Ultimately there will be a visual interface organizing and listing resources.) A general task structure organization will replace the limited organization depicted here. Performance profiles capture expected performance of analysis tools. Markov-Decision Processes are used to explicitly capture uncertainty. Eventually a multilevel blackboard approach, which can capture more complex reasoning processes, will be incorporated.

## Sensemaking Loop

**Sensemaking Loop**

- Foraging/
- Analysis Results

- External Intelligence

- Hypotheses

- Libraries of
- Generic and
- Case-Specific
- Models

- Information Storage
- and Retrieval

- Continual Generation,
- Evaluation and
- Updating of Multiple
- Hypotheses

## Sensemaking Structure

- Developing method to generate, track, and update multiple hypotheses based on SPADE
- Particular hypotheses will be elaborated into predictive models.
- Ranked hypotheses with expectations will be sent to the Foraging/Analysis system.
- Investigating scenarios and developing cases for VAST dataset and for pandemic.

**Notes:**
Spade takes a declarative form of news stories or reports as input, and gives as output a hypothesis that best explains the data. 

## Foraging/Analysis Loop

- Models of
- Alternatives With
- Performance Profiles

- Images

- Video

- Trans-
- actions

- Reports

- Knowledge
- Structures

- Open
- Sources

- MDP-based
- Decision Control

**Notes:**
Markov Decision Process 
Supports Sequential decision making process under uncertainty I.e optimization of not just the immediate utility, but the longer-term utility as well
- Leverages inherent uncertainty of environment

Naïve Bayes Classifier:
- Train classifier on previously encountered problems t Obtain performance of various solution modes for different queries
- Use performance profiles to predict expected performance of new problem instance and use that info to make decisions

Blackboard: 
- Multi-leveled database for the information discovered so far
Input data, partial solutions, various problem solving states
- Layered hierarchy 
allows for explicit modeling of concurrent goal driven and opportunistic data processing
Maintains clear evidential path for supporting and contradictory information

MDP = Markov decision process model. Given a set of hypotheses and the corresponding problem solving for the subsets of these hypotheses, the decision support component (DISC) will handle the following questions: which of the hypotheses should be verified to have a high certainty of solving the query; and, what steps should be taken to verify these chosen hypotheses, i.e., how many resources (time, processing power etc.) should be allocated to each step in the process given the deadline for solving the query. This involves determining which data sources should be accessed, whether data should be visualized, when it should be rendered and the parameters of the visualization.
When a query/hypothesis set arrives at the system, it will trigger the DISC for information foraging and analysis. The control flow for DISC is described in Fig. 3. The hypotheses are posted on a multi-level blackboard system[1] [2]. To support reasoning about time/quality/cost trade-offs, and thus a range of different resource/solution paths, the DISC contains problem solving models that are abstractions of the low-level execution model. These models enumerate multiple different ways to achieve the goals and describe them statistically in three dimensions, duration, quality, and uncertainty, via discrete probability distributions. The problem solving model associated with a hypothesis is then translated into a sequential decision making process[3] that accounts for the fact that, in resource-bounded, uncertain environments, decision choices made earlier in the process will affect the kinds of decision choices made later. [1] D. Corkill. Blackboard Systems. AI Expert, 6(9):40-47, Sept 1991
[2] V. Lesser, B. Horling, F. Klassner, A. Raja, T. Wagner, and X. Zhang. BIG: A resource-bounded information gathering agent. Proceedings of the Fifteenth National Conference on Artificial Intelligence (AAAI-98), July 1998
[3] A. Raja, V. Lesser and T. Wagner. Toward Robust Agent Control in Open Environments. Proceedings of the Fourth International Conference on Autonomous Agents, pp. 84-91, ACM Press, July 2000

## Education Plan

- Our education plan is centered around a visual analytics curriculum and novel training and dissemination to NVAC, RVAC, other universities, and companies.

- We will take the lead in developing  a core visual analytics course and the visual analytics concept taxonomy.
- We are developing the online VA Education Digital Library. The VAEDL is being populated with our course materials and then with materials from NVAC and the other RVACs.
- Presentation and organization of the VAEDL is being improved via user input. Instructional materials for creating good online content will be distributed to NVAC and the RVACs.
- We are developing a visual analytics curriculum where students can choose to pursue either a graduate certificate or an M.S. in visual analytics.

**Notes:**
At the consortium meeting, we will lead the building of the VA taxonomy. Concepts for this exercise are being collected from discussions among the RVACs and with NVAC, from the VAST keywords, and from an IN-SPIRE analysis of Illuminating the Path.
We will teach the core visual analytics course in the Fall. We will establish the visual analytics certificate in the Fall.

## VADL: Who Is the Audience?

- General domain:  Design and Implementation of Visual Analytics Systems
- Graduate students taking courses directly relevant to Visual Analytics systems
- Faculty preparing to teach graduate courses directly relevant to Visual Analytics systems
- Practitioners seeking review of graduate-level material directly relevant to Visual Analytics systems
- NOT
- Researchers
- Those professional users of Visual Analytics systems

_*vadl.cc.gatech.edu*_

## VADL: What is in It?

**Curricula, course syllabi, course material**
- Lectures
- PowerPoint
- Video
- Lecture Notes
- Homeworks, example projects, demo videos
- Small set of classic research and survey papers
- Modeled on Georgia Tech HCC EDL
- _[http://hcc.cc.gatech.edu/](http://hcc.cc.gatech.edu/)_
- Access via taxonomy or search
- Do-it-yourself submission tool
- Link on home page

_*vadl.cc.gatech.edu*_

## VADL: Interface and Content

**VADL: Interface and Content**

- Easily-created short lectures
- Experimentally shown to be effective

- VADL main interface

## Questions?

**Thank You**
_***www.viscenter.uncc.edu***_