Information Integration- An Overview of StreamSpinner Project -Yousuke WATANABE(JST/CREST)
Kitagawa Data Engineering Laboratory,
Center for Computational Sciences, University of Tsukuba
October 31, 2007

Background• Various types of information sources– Data formats, access methods, and query languages• Structured data (RDB), Semi-structured data (XML), Plain text
• Pull-based system, Push-based system• Information integration is quite important– Provides an uniform access method to users
Information Integration
Environment
wrapper
wrapper

wrapper
wrapper

wrapper
wrapperURL
keywords

contentsmailaddress

Mediator
Mediator
wrapper
wrapper

Path

File

Mailing-list
Mailing-list

SQL

File
File SystemSystemXML
Document

Relation

RDB
RDB

Web
Broadcast
Broadcast Serv.
Serv.

wrapper
wrapper

XQueryXPathWebpages

mailwrapper
wrapper

XMLDB

Sensing Devices in Real World
Sensor-network
Mote (Crossbow)

AirSense (Hitachi)

Location
Information
AirLocation (Hitachi) Ekahau (Ekahau)

RFID(OMRON)

CameraMicrophone

GPSStreamSpinner• Integration environment
for heterogeneous
information sources– SQL-like continuous
query language– Event-driven operator
evaluation– Distributed queryprocessing scheme– Java API
StreamSpinner

MASTER Sensor
SELECT *FROM Sensor[1sec]
WHERE Sensor.Temp >= 30

Application

Spinlet API
Query Analyzer
StreamArchiverConnector

Mediator
Wrapper

WrapperSensors

CameraProxyModuleRDBMSStreamSpinnerStreamSpinner

Query result

Continuous query

StreamSpinnerStreamSpinner

Example 1: simple filtering
Deliver sensor data when thetemperature sensor(ttxd11)
becomes greater than 25℃
MASTER Turbine
SELECT timestamp, fsr
FROM Turbine[1]
WHERE Turbine.ttxd11 > 25(ttxd11 expresses a valueof the temperature sensor)
Temperature
RPM…

Sensors monitor
behavior of agas turbine

StreamSpinnertimestamp
timestamp ttxd11
ttxd11 ttxd12
ttxd12 tnhrpm
tnhrpm fsr
fsr
9867664…
9867664… 23.3
23.3 23.9
23.9 1234
1234 87.2
87.2
9867664…
9867664… 23.4
23.49867664…
9867664… 23.6
23.6

24.1
24.1 1234
1234 87.3
87.3
24.1
24.1 1235
1235 87.5
87.59867664…

24.4

24.1

1244

Turbinestream87.8

Plants(Gas turbines)

Example 2: integration of stream
and database
Function to compute
similarity of two arrays

MASTER Clock_1minute
SELECT dist(S1.v, S2.v)
FROM( SELECT array(value) AS v FROM Turbine[60s]) AS S1,( SELECT array(value) AS v FROM TurbineDB) AS S2Aggregate function to convert
data into arraysimilarity
similarity
DBHistorical sensor datastored in DB

Report similarity between
the current pattern and thepattern in the database

Queryresult

StreamSpinnerStreamSpinner
Turbine
10:02
10:01
28℃
10:0026℃
30℃

Demonstration

Information delivery in exhibition hall
Announcemessages fromexhibitors

RDB

StreamSpinner

Wireless sensors
monitor objects inhall
Positions of guests
detected bylocation tags

Guest info.
Exhibitor info.
Hall info.
Announces

ExhibitionboothSensors

Provide contents associatedExhibition boothExhibition
to the current booth ofbooththe user

Exhibitionboothhall

Locations ofusers

demo

ExhibitionboothNotify some changes in hall
Resting places(e.g. Number of empty chairs)
demoDistributed Query Processing
Requirement collaborating
multiple StreamSpinners

QueryApplication program

MASTER Sensor1
SELECT *FROM Sensor1 [1s], Sensor2 [1s], Sensor3[1s]Sensor1.temp > 50 AND Sensor2.temp > 50
ItWHEREis difficult
for users to specify tasks
AND Sensor3.temp > 50

Execute a monitoring
program which detects fire,and gather sensor data tothe program

executed on individual node

Definition of output stream

CREATE STREAM Alarm (Bldg_id long, Temp long)

Application-specific task (written in java)
Sensor1

Receiving data, detecting fire, sending result

Stream σ
Spinner １

Management Tool forDistributed Environment

Deciding the optimal
location for each task
App 3

DB2 ×3
1 × 23σσσ

App×StreamSpinner ２ σ

Stream ×
Spinner ３ σ
DBDBSensor2

Sensor3

Sensor1 Sensor2 Sensor3

StreamSpinner ４

DBSensor4

Sustainable Query Processing••

Dependable systems are required to execute queries in long term
Collaboration with “Sustainable System” (developed by other group)– Sustainable system periodically creates a snapshot of running virtualmachine, and distributes the snapshot to other nodes– When the node fails, backup node recovers the virtual machine from thesnapshot

Processing nodeInformationSourceDataBackup nodeQueryQueryStreamSpinnerStreamSpinner

Virtual Machine

Virtual Machine

Sustainable System

Sustainable SystemOSOS

HardwareHardware
Network

Application

Example: monitoring remote buildings
SecurityCenterBackup nodetakes over query
processing

Execute a monitoring
program which detects fire,and gather sensor data tothe program

Sustainable System
Snapshot of
Virtual machine
Sustainable System

StreamSpinner

Monitoring
ProgramManagement Tool forDistributed Environment

Network

RequestRequestStreamSpinner

RequestStreamSpinnerStreamSpinner
SensorRoom3
SensorSensor

Room1

Room2

DemoConclusion• StreamSpinner– Integration environment for heterogeneousinformation sources• Future work
– Experimental evaluation about distributed
query processing– Development of real applications– Open source