# Presentation: 488588

## User intent inferencing and information retrieval

- Eugene Santos Jr.
- University of Connecticut
- Intelligent Distributed Information Systems Lab
- Eugene@engr.uconn.edu

## Outline

- Problem
- Motivation
- IPC User model
- Evaluation on CRANFIELD collection
- Evaluation on CNS collection

## Problems in processing information by users:
Massive data from different resources
Repetitive tasks
Lack of tools for linking known events and knowledge
Modeling users is central to creating a synergistic relationship between a user and tools used to search/ analyze information.

- Massive data from different resources
- Repetitive tasks
- Lack of tools for linking known events and knowledge
- Modeling users is central to creating a synergistic relationship between a user and tools used to search/ analyze information.

- Motivation

**Notes:**
As other speakers have talked about the problems that intelligent analysts have faced in their every days job which include massive data from different resources, our belief is that by modeling the analysts, we create a good relationship between an intelligent analyst and tools used to process information and thus help analyst improve the quality of their jobs.

## Problem

- Employing a cognitive user model for Information Retrieval (IR):  capture and use **knowledge** about a user to improve *retrieval performance* and *user satisfaction*.
- Why do we need user models for IR?
- IR system needs
- understand context of why this information is needed
- helps users formulate queries based on context
- retrieve relevant documents with good quality for each query
- But, traditional IR framework supports very little user involvement (Saracevic 96)
- Solutions from IR community:
- Query specification, query formulation
- Relevance feedback in interactive IR
- Stereotype, domain knowledge

**Notes:**
Now, let’s start with the problem. We study the problem of employing a cognitive user model for information retrieval in which knowledge about a user is used to improve retrieval performance and user satisfaction.
There are a few keyword needed a little more explanation here. First. knowledge about a user refer to information such as how a user searches, how a user interacts with IR systems over a period of time, what context motivates user engages him/herself in a search for needed information. Retrieval performance refer to the performance of the IR system measured by standard metrics in IR. User satisfaction refer to the user subjective experience such as they are happy or not and the user objective performance such as they are accomplishing a task in timely fashion.
Now, we would wonder why we need to use user model for IR? Does it mean we add an extra layer on top of already-complicated IR system?
A typical IR system would consist of 3 components: a user with his information needs, such as an assignment to write an essay about UCONN basketball problems for Daily campus, a mediator which could be a librian in era before online searching or could be a search engine such as google, and of course information resources which could be paper catalog prior to computer age, or could be databases stored on harddisks. A user came to use mediator to help him to solve his information needs, for example a freshmen who tries to be sport writer for Daily campus was assigned to write an essay about UCONN basketball programs in 2 days. What an IR system needs to do here is to help users to solve his information needs effectively by (i) understanding the context of why this information is needed (ii) formulating the right queries based on knowledge of the context (iii) retrieving relevant documents with good quality with regards to each query. In an ideal situation, for example in this case, the freshmen is not familiar with UCONN basketballs programs in the past but possess basic knowledge of basketball, it may be ideal to pint point to him that he should write about transition games which is the trademark of our school programs and that has helped us to reach two final fours in last 5 years. That’s the ideal case. Unfortunately, traditional IR framework supports very little user involvement, except user queries, as identified by Saracevic in 96. IR community has been since tried to address these problems by asking users to construct the model of their information needs (query specification), or design a system in which a user can formulate their queries (query formulation) or using their implicited information to iteratively change queries (relevance feedback) or using information about user demographic info, or domain knowledge to modify queries. 
These attempts are still going on but suffer from the main short-coming which are that the IR, as a discipline wasn’t paying much attention to how best to construct the model about users, and how best the model helps IR process in the long-term period. That’s why UM techniques have been exploited.

user search behaviors such as searching styles from general to specific, analytical vs shawllow exploration, how a user interacts with interfaces etc. Retrieval performance relates to # relevant documents retrieved and user satisfaction refers to subjective user experience and objective measure of user performance.
Why do we need to use user model for IR? Do we add an extra complexity on top of IR system. The ultimate goal of IR is to retrieve as much as possible relevant information quickly to satisfy a user’s information needs. The best definition of information needs is defined by Wilson 81, in which need is defined as a subjective experience that occur only in the mind of a person in need is not directly assessible to an observer.  For example, a student has a writing assignment about Weapons of Mass Destruction and Iraq. This request is transferred to a sequence of queries, for examples Weapons of Mass Destruction, Iraq, Saddam Hussein and so forth. If the user is new to this domain, he likely does not know how to formulate the right queries because of poor domain knowledge or lack of understanding about IR indexing system and so on.  In this case, they may not know that sarin is a kind of biological weapons, as well as scud missle is a kind of nuclear weapons which are all considered as WMD too. Therefore, the results of the search may not return the expected information. One intuitive idea is that if for magically, IR system has knowledge about what a user background and a user needs, it would serve a user better, which means it would retrieve more relevant documents to fulfill a user’s needs.

But modeling a user for IS is not a simple problem. This problem is well-known for the challenges it creates because often times we only know partial information about user knowledge, user’s information needs only exist on their brains and difficult to observe, modeling every user’s interactions, movement, thoughts involves of uncertainty factor. The user him/herself sometimes doesn’t know why he chose this way or that way. Besides, a user’s information needs oftentimes is expressed vaguely because of poor domain knowledge or misunderstanding. To make the matter worse, a user changes over time as a results of receiving new knowledge, new context or simply because we are changing as the time goes, what interests us now may not be attractive to us 1 or 2 years from now. These challenges are the main reasons for some exciting research in user modeling. 

## Problem

- Challenges of employing User Modeling (UM) techniques for IR
- Partial information
- Uncertainty
- Vagueness
- Dynamics

**Notes:**
But modeling a user for IS is not a simple problem. This problem is well-known for the challenges it creates for both user modeling and information retrieval communities. We have identified 4 primary challenges of this problem. The first difficulty is that we only know very little information about user and his information needs. Information needs have been defined as the user subjective experience that only occur in the mind of person in need and is inaccessible for observer. Second is the uncertainty factor while modeling user interactions because even the user him/herself sometimes doesn’t know why he chose this way or that way. Besides, a user’s information needs oftentimes is expressed vaguely because of poor domain knowledge or misunderstanding. To make the matter worse, a user changes over time as a results of receiving new knowledge, new context or simply because we are changing as the time goes, what interests us now may not be attractive to us 1 or 2 years from now. These challenges are the main reasons for some exciting research in user modeling. 

## Motivation

- Existing methodologies for building user model for IR include:
- System-centered approaches: use IR techniques. (*e.g Spink & Losee(1996), Efthimis(96), Lopez-Pujalte (03),Drucker et al(02))  *
- User-centered approaches: use Human Computer Interaction (HCI)/Artificial Intelligence (AI) techniques *(e.g:Belkin(93), Radlord(96), Balanovic(98), Billsus and Pazzani(99,00))*
- Hybrid approaches: combine IR with HCI/AI techniques. *	**(e.g Logan et al. (94), Decampos et. al 98) *
- Very little crossover between IR and AI/HCI to build user models for IR

**Notes:**
These challenges inspired us to explore this problem.
 The UM community have focused on three main approaches
 to build a user model: first group contains those techniques that use IR
 concepts for example relevance feedback and query expansion to build user models. The user model built by these techniques are short-lived which means the user model only has effects on each query instead of a search session. The system-centered approaches focus on system side and system performance.  
The second group use HCI techniques to build user models such as question analysis and interview before search after search or cognitive task analysis to build user models. These techniques focus more on user side instead of system side and user performance. One key problem arise with these approaches is that they are concerned with user behavior but little has to say about why a person might engage in one particular behavior. In order to find out why, we have to establish the relationships between the behaviors and the problems and the relationship between goals and subgoals which are unfortunately missing from this type of approaches.
The third group combine system-centered and user-centered for building user models. What makes the hybrid approaches important because this is the only way to satisfy two goals improving retrieval performance and user satisfaction at the same time. What makes the hybrid approaches hard include the difference, even incompatible, or contradictary in conceptual levels, in representation levels, in measurement, algorithms from different fields IR, HCI and AI. Take for example, in traditional IR view points, system designers, authors as well as searchers do share the same views, the same terminology. Searchers are assumed to know the right index terms to find relevant documents. Therefore, commonly use metrics for IR is assess how many documents have been retrieved actually been assessed as relevant by indexer by subject matter to a certain query, not by a user who actually is using the IR system. That assessment is done independently with regards to the context which motivates the user to ask the specific query.
Whereas from HCI view points, users may belong to several groups with different background with ill-defined information needs and requests. 
Therefore the common metrics used in HCI is to assess how many documents the user who is actively used the system found relevant to the topics and how efficiently the user performs his searching tasks.
The fact of the matter is, there is very little overlap between IR in one hand and AI/HCI in another hand to build user models for IR and thus wheels are reinvented in each field. And even though there are some attempt for hybrid approaches, there are still a number problems in that department, including comprehensive evaluations which assess both retrieval performance from system side and user satisfaction from user side. 
Therefore, for our purpose, we focus our efforts to develop a hybrid approach to build user model for IR.

## Motivation

- Important factors for building user models for IR:
- Relevance feedback.
- Adaptation.
- Intention.
- Author’s intention
- User’s intention

**Notes:**
When we explore this problem and the challenges that it poses to UM and IR communities, we have identified key factors for solving this problem. First, as we know, a user may start with vague, poorly expressed information needs as the poor domain knowledge. However, even though a user may be very new to a specific domain and thus can not formulate the “right” query using the “right” terms, if a user has 
a chance to look at some documents related to the topics, 
he/she can recognize and revise his/her needs accordingly. 
Therefore, RF is a way used to refine a user’s vague information needs iteratively. 
Secondly, one of the challenges is that user keeps changing over time. In this case, if an IR application can adapt to the user’s changes promptly, it may help a user’s solve his/her IS needs better.
Most importantly, we have identified a very important factor which is intention.  Retrieving a document is an purposeful and active activities because the user wants to solve his information needs. Writing a document is also an intentional activity because an author wants to convey a message through his writing. Therefore the content of each document reflects the author’s intention. Therefore, user intent factor has been played 
very important role in IR. We need to find a way to unified user intention (seeker’s intention) and author’s intention with each other.

## Approach 

- We try to improve retrieval performance and user satisfaction by
- Developing a cognitive user model to capture user intent dynamically through analyzing behavioral information of retrieved relevant documents *(Santos E., Nguyen H., Brown S. 01; Santos E., Nguyen H., Zhao Q., Pukinskis E. 03a; Santos E., Nguyen H., Zhao Q., Wang H. 03b) *
- Combining the captured user intent with the elements of an IR system *(Nguyen H., 2004)*

**Notes:**
So all that’s intuitions lead to the core of our research. With behavioral information of retrieval relevant documents, we refer to the commonality information (information overlapping) of RRdocuments. Here, we try to infer the user intent through the content of RRdocuments which represents authors intent. We does it through relevance feedback process. The user model is built dynamically and thus changes as a user changes which satisfy the adaptivity mentioned in the last few slides.

## IPC User Model  

- Captures ***user intent***.
- Consists of 3 components:
**User interests (I)**: *“*_***What***_* is the user interested in? What needs to be done or accomplished?” *
**User preferences (P)**: *“*_***How***_* is something done or accomplished?”*** **
**User context (C)**: *“*_***Why***_* is the user trying to accomplish something?” *

*(Santos E.,  Nguyen H. and  Brown S. M.  00,  01)*

**Notes:**
The ultimate goal of our model is to capture user intent and use that captured user intent to modify a user’s query. We participation user intent into three formative components which are Interests (I), Preferences (P) and Context ©. Read the definition of each. 

## User Modeling for Analyst Intent

***Analyst***

- Analyst Interests

**Short-Term**

**Long-Term**

- Analyst Preferences

## IPC User Model

- GOAL: Predict and explain user’s beliefs, goals, and intentions.
- A user model *M={I, P, C}*
*I*: an interest set
- Denotes the topics or subjects which the user is focusing on.
- Weighted set of interests derived from user queries, retrieved documents, and analyst feedback.
*P*: a preference network
- Denotes how the user would go about acquiring and viewing the desired information.
- Network that determines how queries are modified.
*C*: a context network
- Where the user is working on based on short and long-term interests.
- Network that models the user/analysts context; i.e., identifies relationships between concepts of interests and infers and explains the concepts with respect to the analyst.

## Create and update user interests

- Each element of interest set consists of ***interest concept**** (**a**) *and ***interest level**** **L(a)*
- For each interest concept *a, *update L(a) :
*L(a) = **** **** (L(a) + n/m)*
*	*where
*	**	***: learning factor
*	**	**n*: number of relevant document containing *a*
*	**	**m*: total number of relevant documents

## Create and update context network

- Context network is constructed “on-the-fly” by finding a common set of sub-graphs among the relevant documents.
- Each document is represented as a document graph (DG).
- A sub-graph *X* of DG *Y* is:
*X* is a DG
-  node a *X*  a *Y *

## Create and update user’s preferences

-  We use a Bayesian network to represent a preference network (P):
-  Pre-condition (*Pc*)
-  Goal (*G*)
-  Action (*A*)
-  Each *G* node associates with a set of *Pc* nodes, and each *A* node associate with one *G* node.

## User Modeling Functional Architecture In Information Retrieval System

## Generate Document Graph

**Input text file**

**Break into sentences**

**Take one sentence**

**Parse by link parser**

**Get Noun Phrases**

**Create taxonomy relations from Noun Phrases**

**Sentence!=NULL**

**End**

**Yes**

**No**

- Create nodes
- in Document Graph

## Example of Document Graph

- FBI 1) Report Date: 1 April, 2003.
- FBI: Abdul Ramazi is the owner of the Select Gourmet Foods shop in Springfield Mall. First Union National Bank lists Select Gourmet Foods as holding account number. Six checks totaling $35.000 have been deposited in this account in the past four months and are recorded as having been drawn on accounts at the Pyramid Bank of Cairo, Egypt and the Central Bank of Dubai, United Arab Emirates. Both of these banks have just been listed as possible conduits in money laundering schemes.

## Example of Query Graph

- Example of Query Graph

## Context Network

## Interest Set

- Interest set:
- terrorism,	level=0.96
- terrorist,	level=0.9
- money___laundering, level=0.93
- al___quaeda, level=0.85
- deposit,	level=0.82
- withdraw,	level=0.81
- explosion,	level=0.8
- terrorist___training, level=0.79
- forged___document, level=0.77
- bank_ __account, level =0.76

## Preference Network

## Modified Query Graph

- Original query graph

- Modified query graph

- banking_
- transaction

- transaction

- banking

- Related_to

- Isa

- Related_to

- Related_to

## Assess the Effectiveness of User Modeling for IR Using CRANFIELD collection

- Test bed: CRANFIELD collection.
- One of the oldest test collections in the IR community (Cleverdon, 1967).
- Widely used (Salton and Buckley, 1990; Loper-Pujalte et al., 2003)
- 1398 papers on aerodynamics and 225 queries with relevancy assessment.
- A relevant document with regards to a specific query is rated on a numerical scale from –1, and 4 to 1.
- Compare against Ide dec-hi approach with term weighting by term frequency and inverted document frequency (TFIDF) (Salton and Buckley, 1990), which considered to be the best traditional approach in IR community.

## UM vs. Ide dec-hi

- Ide dec-hi
- Each query is converted to a query vector, which contains weighted terms.
- The query vector is compared against each document vector.
- Runs each query twice.
- The first 15 returned documents from initial run will be used to as feedback in modifying the query vector for the feedback run.
- UM
- Runs through the query sequence, and UM updates itself based on the relevancy in the first 15 returned documents.
- Uses the UM learned as the seed UM and run the experiment again.

## Experiments with UM Approach

| Experiment 1 | Starts with an empty UM for every query. After the initial run for each query, the UM updates itself based on the relevancy in the first 15 returned documents. Then rerun the query.The UM is reset to empty before a new query has been issued. |
| --- | --- |
| Experiment 2 | Starts with an empty UM for the first query. The UM updates itself based on the relevancy in the first 15 returned documents.  |
| Experiment 3 | Same as Experiment 1, but uses the UM learned from Experiment 3 as the seed UM.
 |
| Experiment 4 | Same as Experiment 2, but uses the UM learned from Experiment 3 as the seed UM.
 |

## Average Precision at Three Point Fixed Recall in Experiment with Original Collection

- Average Precision at Three Point Fixed Recall: the average precision at recall points of 0.25, 0.5 and 0.75.

## What’s in the Top 15?

| Relevancy | TFIDF | Exp 1 | Exp 2 | Exp 3 | Exp 4 |
| --- | --- | --- | --- | --- | --- |
| Ranked 1 | 13 | 19 | 19 | 20 | 22 |
| Ranked 2 | 33 | 41 | 42 | 41 | 39 |
| Ranked 3 | 49 | 64 | 62 | 65 | 62 |
| Ranked 4 | 26 | 31 | 32 | 31 | 28 |
| Total | 121 | 155 | 155 | 157 | 151 |

## Evaluation Results

- Our approach produces better performance in the initial run compared to TFIDF approach in all of four experiments.
- Our approach retrieves more quality in the top 15 compared to TFIDF approach.
- Knowledge about a user intent helps improve retrieval performance.

## Evaluation with CNS collection

- Evaluation took place in NIST laboratory.
- Test bed: Center for Nonproliferation Studies (CNS) database (sept. 2003).
- Country profiles on WMD
- Arms control
- WMD Terrorism
- UM vs. Verity Query Language.
- Subjects use two different systems in parallel.
- There are 10 scripted queries on “Iran R&D supporting BW”.
- Only first 10 documents have been reviewed for relevancy.
- Three analysts took part in the experiments.

## Analyst Profiles

## Entry Questionnaires

## Slide 32

## Ten Queries

- Iran research biological warfare
- Iran research institute, university biological warfare
- Iran biological research and biological warfare
- Biological research facilities in Iran
- Intelligence assessment on Iran biological research
- Iran foreign connections in biological weapons program
- Cuba, Iran and Russia connections to WMD
- Iran’s biologists visits Cuba
- Russian biotechnology, missiles, aid to Iran
- China supply and Iran biological weapons program

## Query Assessments

## More Relevant Documents found with the User Modeling Assistant

## User Model Helps Find Documents Based on Individual Interests

## Evaluation Summary

- UM helps retrieve more unique documents that relevant to analysts’ interests.
- UM assisted IR has higher precision and recall than the Verity Query Language.
- UM helps system track the interests and focus of individual analyst.

## Conclusion

-   Initial research shows that by capturing user interests, preferences and context provides fine-grained model that effectively captures the user’s intent in information seeking tasks.
-  Empirical evaluation shows that the user model improves the retrieval performance by adapting to an individual’s styles and focuses in online searching.

## Reserve Slides

## Example:  Case Study #4Represent Incoming Documents/Messages as Document Graph

- FBI 1) Report Date: 1 April, 2003.
- FBI: Abdul Ramazi is the owner of the Select Gourmet Foods shop in Springfield Mall. Springfield, VA. (Phone number 703-659.2317). First Union National Bank lists Select Gourmet Foods as holding account number 1070173749003. Six checks totaling $35,000 have been deposited in this account in the past four months and are recorded as having been drawn on accounts at the Pyramid Bank of Cairo, Egypt and the Central Bank of Dubai, United Arab Emirates. Both of these banks have just been listed as possible conduits in money laundering schemes.

## Example:  Case Study #4User Interests and Context Network

- terrorism,	level=0.96,
- terrorist,	level=0.9,
- money_laundering, level=0.93,
- al_qaeda,	level=0.85,
- bomb,	level=0.85,
- bombing,	level=0.9,
- explosives,	level=0.9,
- explosives_training, level=0.85,
- forged_document, level=0.77,

## Example:  Case Study #4Query graph modification based on the user model

- Query: banking transaction

- Original query graph

- Modified query graph

-  No messages have been returned

- 3 messages have been returned.
- FBI report #1, #15 and #16.
- .

- The 3 returned messages have all been picked up by the analyst as feedback, and sent to the prior/tacit knowledge subsystem.

## Example:  Case Study #4Modified user context network based on user feedback

## Example:  Case Study #4Proactive Query Generation

## Average Precision at Three Point Fixed Recall in Experiment with Residual Collection

- Average Precision at Three Point Fixed Recall: the average precision at recall points of 0.25, 0.5 and 0.75.