# Presentation: 808727

##  D-SIFTER: A Distributed Information Filtering System

**Indiana University Purdue University Indianapolis****Indiana University Bloomington****Rajeev R. Raje**
**Mathew J. Palakal**
**Snehasis Mukhopadhyay**
**Javed Mostafa**
**http://sifter.indiana.edu**
**http://sifter.cs.iupui.edu**

## SIFTER: Motivation

***Information Overload***** --A Reality in Today’s World**
**A Need for locating highly ‘Relevant Information’**
**A Necessity for an existence of a single tool for accessing multiple data sources and formats**
**Continuous Updating of relevant information **
**Privacy**
**Collaborative Work environments**

## The Current Model

- Yahoo!

- Private Data Sources

- Internal Data
- Unexplored Data

- X

## The New Model

- Yahoo!

- Private Data Sources

- Internal Data
- Unexplored Data

**SIFTER**

## SIFTER

## D-SIFTER: Distributed Multi-agent Filtering System

- Objectives
- To achieve improved classification and accuracy
- To elegently handle inadequacies of thesauri
- To obtain a high degree of flexibility in dealing  with multiple domains

## Distributed Classifier

## Distributed Classifier

**Domain: ****Computer science**
**Number of documents: ****1064**
**Performance improvement with 3 agents: 44%**
**Experiments with 3/5/50-70 Agents**
**Different Economic Schemes**
- Minimum and Maximum Limits
- Various Window Set Values

## Distributed Filtering Performance

- Sample Result

## Dynamic Acquaintance Lists

- Classification Performance Saturates
- The response time continues to increase

## Dynamic Acquaintance Lists

- Interaction with a small (4) number of acquaintance agents, rather than exhaustive collaboration
- The agents in the list are automatically decided based on real-time learning
- Successful agents are rewarded, while others are penalized
- First-come or QoS

## Dynamic Acquaintance Lists

- Probabilistically selects the acquaintances so as to converge to the best ones
- Generates a model of all agents denoting their suitability
- Automatically reconfigures the acquaintances in the presence of agent failures and/or new agent entries

## Results -- First Arrival as the Best Classification

## SIFTER II -- Global View

- User

- WWW
- Java

- Results/Documents

- Feedback

**ACM **** **

**Med**
**Line**

**HGP-**
**DB**

**...**

- WWW, Java

- Administrator

- Wrapper-Agents

- Maintain
- Agents

- Classifer
- Agents

- Representer
- Agents

- User
- Agents

- Domain
- Agents

- Agent World

## Agent Types

- Wrapper agents
- Representer agents
- Classifier agents
- Domain agents
- User agents
- Administrator agents
- Each Agent has an Uniform Architecture
- Communicator
- Executor
- Controller

## RMI

**RMI**

- “buy” as many doc’s        as possible
- Evaluate source
- Supervise direct retr.

- Set of
- (Domain-Relevance,
- Thes.-Relev.-Vector,
- Est.Rel.Prob.-Vector,
- Action.-Prob.-Vector)

- User-Agent

- “sell” as many doc’s        as possible
- gain high relevance         feedback

- Thesaurus, Set of
- (Inv. Doc.Feq.Vector,
- Set of Centroids,
- Doc.Assign. Vector),
- DAM-Relevance-Vector

- Feedback
- Inv.Doc.Freq.Vector
- Doc.Assign.Vector
- Centroids
- Keywords as Retr.
- Thesaurus
- Documents

- Domain-Agent

- Provide Classification

- Classification Algorithm:
- Generation of Classes,
- Assigning Docs to Class.

- Inv.Doc.Freq.Vector
- DocVector
- Doc.Assign.Vector
- Centroids

- Documents
- Thesaurus
- Keywords as Retr.
- Feedback

- Provide  Representation

- VectorSpaceModel

- Representer

- Documents
- Thesaurus
- DocVector
- Inv.Doc.Freq.Vector

- Provides Documents
- Research on Keyw.
- Sell Documents

- Document Space
- Information Sources
- Automatic Retrieval
- Keyword Retrieval

- Keywords
- DocHandle
- Documents

- DAM-Agent

- Classifier-Agent

- SIFTER II

## Future Work: Economic Framework for Collaboration

- Bidding function based on
- quality of service & financial state
- probability of seeking remote collaboration
- user-centric vs. money-centric
- Four stage contracting (request/bid/select/execute) vs. more stages of negotiation
- Auction-based contracting
- Sub-contracting

## Slide 18

## Thank You

- {rraje, mpalakal, smukhopa*}@cs.iupui.edu*
*jm@indiana.edu*
*sifter@cs.iupui.edu*