Confucius and Its Intelligent Disciples:
Integrating Social with Search
Xiance Si

Edward Y. Chang

Google Research
Beijing 100084, China

Google Research
Beijing 100084, China

sxc@google.com

edchang@google.com

Zoltán Gyöngyi

Maosong Sun

Google Research
Mountain View CA, USA

Tsinghua University
Beijing 100084, China

zoltang@google.com

sms@tsinghua.edu.cn

ABSTRACT
Q&A sites continue to flourish as a large number of users rely onthem as useful substitutes for incomplete or missing search results.
In this paper, we present our experience with developing Confucius, a Google Q&A service launched in 21 countries and four languages by the end of 2009. Confucius employs six data mining
subroutines to harness synergy between web search and social networks. We present these subroutines’ design goals, algorithms, andtheir effects on service quality. We also describe techniques for andexperience with scaling the subroutines to mine massive data sets.

1.

INTRODUCTION

Despite the success of Internet search engines, Q&A sites continue to attract a large number of users. For instance, Yahoo! Answers serves an estimated 1 billion pages to 80 million unique visitors each month.1 Users visit Q&A sites either because the soughtafter information may not be available on the Web, or because theavailable information is not summarized in a conveniently accessible way. For example, when a very recently launched mobile
device fails to boot, it may happen that the fix to the problem is not
yet published on the Web. Even when relevant information is available, it may be buried in some long product specification in a waythat makes it difficult to locate. Furthermore, in many Asian, Middle Eastern, South American, and Eastern European countries withemerging Internet presence, the number of local Web pages available to search engines for indexing is relatively low (when compared to that in the US and Western Europe). In these countries,
1As
reportedbytheDoubleClick
AdPlanner(http://www.google.com/adplanner/site profile#siteDetails? identifier=answers.yahoo.com),ited on March 7, 2010.

tool
vis-

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Articles from this volume were presented at The
36th International Conference on Very Large Data Bases, September 13-17,
2010, Singapore.
Proceedings of the VLDB Endowment, Vol. 3, No. 2
Copyright 2010 VLDB Endowment 2150-8097/10/09... $ 10.00.

Q&A sites are often a prime source of online information. Indeed,
for instance, in China, 25% of Google’s top-search-results pages
contain at least one link to some Q&A site. Thus, Q&A sites are
not only useful when some information is not present on the Web,but also constitute a versatile source of indexable content for searchengines.
In this paper, we present the key technologies that we relied
on for designing and developing Confucius, a Google Q&A system that is meant to leverage the social network of Q&A users tosynergistically produce high-quality content that may benefit Web
search. The performance goal of Confucius is to provide highquality answers in a timely fashion. In building a system that meetssuch a goal, we encountered four particular technical challenges.
First, typical users are seldom willing to provide answers without
incentives. Second, in presence of incentives, abusers/spammers
are always quick to emerge. The abusers/spammers may lead to adegradation in service quality that can discourage knowledgeable
users from contributing. Third, when a question is asked, it should
be routed quickly to a domain expert to reduce answering delay.
Finally, should an answer to a question already be available, thesystem must surface it to avoid redundant work (and unnecessary
delay).
The first two of the challenges affect answer quality, while thelast two affect the timeliness of answers. Besides answer quality and timeliness, we discovered that the nature of the questions
has great impact on the quality of the collected Q&A corpus, andthus on the overall user experience. When we first launched Confucius in 2007, as a service on an existing social network, mostaskers sought subjective information, such as relationship advice.
The consequent interactions did not yield generally relevant, highquality content. As one of the stated goals of Confucius is to improve Web search quality by contributing useful content, we sinceenhanced the system so that it attracts more questions that have
objective, factual answers.
To tackle the aforementioned technical challenges, we developed
six system components—six disciples of Confucius—that help us
establish a healthy ecosystem spanning both social and search. These
components and their purposes are as follows:
Search integration: We integrate Confucius with the Google
search engine. When search terms indicate a wh-query (when,
where, why, etc.) or when the search engine cannot return sufficiently relevant results (e.g., the content overlap between query andpotential top result pages is low), a Q&A session is recommended,encouraging the searcher to post a question on Confucius.
Question labeling: Once a question has been typed in, a set of
category labels are suggested. Labels are useful for organizing thequestion corpus, for providing a subscription mechanism, and forrouting questions to answerers. We employ a parallel implementation of Latent Dirichlet Allocation (PLDA) [12] to make labelsuggestions.
Question recommendation: Given a question, this subsystemfinds similar earlier questions and their already available answers.
Such prior information often reduces the time it takes for a user toobtain a satisfactory answer. PLDA is used once more to accomplish this recommendation task.
Answer quality assessment: We evaluate the quality of each answer based on several factors including its relevance to the question
and its originality. Being able to assess answer quality in a semiautomatic way is a key requirement in identifying top contributors
and in curbing spam.
User ranking: We rank users based on their contributions ina domain specific way. User ranking is used for identifying top
contributors within each domain so that we can provide appropriate
incentives, enhance the ranking of Q&A pairs in search, and routequestions to domain experts.
NLP-based answer generation: Though Natural Language Processing (NLP) research is still in relative infancy, it provides avenues leading to an improved Q&A experience. In particular, we
employ NLP techniques to automatically generate answers. Notethat a typical user of a Q&A service can tolerate a delay of several minutes (as opposed to the instantaneously expected search
results). This relaxed latency constraint allows us to run timeconsuming, complex NLP algorithms that may yield good answers.
Also note that NLP answers are optional, to the extent that the system may always elect to discard low-quality generated results andsimply wait for an actual human response. Accordingly, we canharvest the best NLP results without compromising overall quality.
NLP techniques also prove to be helpful in assessing the type ofquestions and the quality of user-provided answers.
In addition to being accurate, the subroutines underlying our system’s components must also be able to deal with massive amountsof data, in the order of terabytes. To address the scalability issues, our foundational algorithms (such as PLDA [12] and UserRank [17]) can be executed on thousands of machines in parallel.
Since 2007, Confucius has been deployed in 21 nations in four
languages. In China, for example, the usage of Confucius quadrupled in 2009. Confucius content has a marked presence in Google
search results now. Each of the subsystems played a clear role in
the quality and timeliness improvements that lead to such a wide
adoption.
The rest of this paper focuses on the six subroutines. First, we
review the other existing Q&A services and related research results(Section 2). Next, we introduce the Confucius system and its subroutines in detail (Section 3). We continue by presenting key usage
statistics (Section 4), revealing how the presented subroutines havehelped improve content quality and traffic. Section 5 discusses thelessons we learned and some future research/development directions. Finally, we provide concluding remarks in Section 6.

2.

RELATED WORK

More than ten large online Q&A services were launched since1998. Not all of these systems are alike—they differ mainly depending on how they approach three key issues: question submission, question routing, and incentives. The approaches to these

Name
Internet Oracle
Ask.com
WikiAnswers
Yahoo! Answers
Baidu Zhidao
Google Confucius
Aardvark [6]Powerset
Quora

Question submission
by emailby searchin community
in community
in community
by search & incommunity
by IM
by searchin community

Key elements
Question
Incentives
routing
to experts virtual
N/A
N/A
to public
virtualto public
virtualto public virtual & $& experts
to public virtual & $& experts
to friends
virtual
N/A
N/A
to friends
virtual

NLQ&A

Year

no
yes
no
no
no

1989
1996
2002
2005
2005

yes

2007

no
yes
no

2009
2005
2009

Table 1: Representative Q&A sites.
three issues can be regarded as independent variables, which affectdependent variables, such as answer presentation.
Question submission: A user can ask a question through two access points: within a search engine (like with Confucius’ search integration) or on a community site. A post-search question tends toseek facts or objective answers, whereas an asker within a community tends to expect more personalized answers tailored to his/herparticular situation.
Question routing: When the answers are provided by users, aquestion can be routed to domain experts, social contacts, or both.
The choice of the routing algorithm affects the nature of the gathered answers. When a question is routed to an expert, the answer
tends to be formal and objective. When an answer is provided by afriend, it tends to be more informal and personalized. For example,
assume that the question is “Which are the must-see sights in Singapore?” If the asker is known to the answerer, the answerer may
tailor the answer to focus exclusively on, say, natural landmarks,
historical sights, or contemporary attractions. In contrast, a domain
expert may choose to pursue a more balanced approach.
Incentives: Without incentives, few users would provide highquality answers. Incentives can be built around community reputation or monetary rewards and provided to askers or answerers.
Content quality is the primary consideration, but incentives may
also help induce bias on content type or category. Let us assume
that a site aims to collect facts to eventually build a Wikipedia-likeservice. Correspondingly, questions with objective answers may
be more encouraged on the site. In contrast, a site interested incollecting diverse opinions on restaurants, fashion, or travel, may
encourage questions and answers on these topics by differentiallyrewarding them.
Independently of the above three issues, some services provide
machine-generated-answers via search or NLP. NLQ&A is a branchof the general NLP and AI research with more than ten years of history [4]. One NLQ&A approach is to generate answers from a formatted knowledge base, such as Wikipedia or relational databases.
A typical system consists of three parts: question analysis, information retrieval, and answer selection. During the question analysis step, the system classifies a question by the expected type of
the answer, using regular patterns [16] or machine learning [11].
Then, the question is used as a query to search the document set
and possible answers of the expected type are extracted from thesearch results. Third, the system ranks the candidate answers byfrequency, sophisticated linguistic features [9] or information distance [20].
A different NLQ&A approach is based on finding similar questions within an existing Q&A database. Lai et al. [10] proposed amethod to mine questions and answers in Web FAQs. Jeon [7] modeled the similar question finding problem as a machine translationtask, and showed such modeling to be more effective than someother similarity measures. Xue et al. [19] extended previous workby combining the language model and IBM’s translation model.
Murdock et al. [13] used non-content features to help retrieve answers for procedural questions. Surdeanu et al. [18] also presenteda system that leverages NLP features to improve the accuracy ofquestion retrieval. Agichtein et al. [1] reported their experiencein assessing the quality of existing answers, which helps NLQ&A
systems rank matching answers. Since no human is involved in theanswering process, NLP is more suitable for answering questions
seeking factual answers rather than opinions.
Table 1 lists the main Q&A services and their positioning withrespect to the three key issues, together with the answer sources.
For instance, Confucius aims to collect factual information by providing search integration, question-label suggestion, expert identification and routing, and quality-based incentives.

Search Integration subroutine (Online)
Given: A SVM model obtained offline MSI , a threshold of PageRank value ǫ
Input: A Web query q
Output: Decision δ, show the triggered link if δ = true
1: if MSI classify q as a question2:
δ = true
3: else if max{P ageRank(r)|r ∈ F irstP ageResults} < ǫ4:
δ = true
5: else if the user clicked Next on the first page6:
δ = true
7: else8:
δ = f alse

Figure 2: SI subroutine.

3.

THE CONFUCIUS SYSTEM

by the system. The asker can optionally select preferred notification mechanisms such as email or instant messaging for when
an answer becomes available. Askers may also provide feedbackon the submitted answers, in the form of best-answer selection or
agree/disagree votes. However, in practice most answers will lacksuch feedback. To assess the quality of answers with a broader
coverage, Confucius implements the AQ subroutine (Figure 1(4)),
which learns the relationship between different features and thequality of the answer as indicated by user votes, and uses the learned
model to assign quality scores to answers without explicit feedback.
Quality scores are then aggregated by UR (Figure 1(5)), whichquantifies user contributions and ranks users in a topic-dependent
manner. Finally, the NL subroutine (Figure 1(6)) provides a machinegenerated answer when it is capable of. In the remainder of thissection, we describe the goals, functionalities, and algorithms ofeach key subroutine.

3.1 Search Integration (SI)
Figure 1: The Confucius Q&A system.
Figure 1 presents the question answering work flow, including
steps that involve Google search and the six main subroutines ofConfucius (represented by numbered bubbles). For convenience,
we introduce the following acronyms for the corresponding subroutines:
SI : Search Integration,
QL : Question Labeling,
QR : Question Recommendation,
AQ : Answer Quality Assessment,
UR : User Ranking, andNL : Natural Language Q&A.
Currently, there are two access points to Confucius: through theGoogle search engine and through the Confucius site. After a user
issues his/her search query, SI (Figure 1(1)) decides whether the
search engine should recommend the user to initiate a Q&A session. Once the user submits a question, the QL subroutine (Figure 1(2)) suggests a list of category labels to the user to select from.
The QR subroutine (Figure 1(3)) compares the submitted question
with existing ones and, in case of a match, recommends the corresponding answers to the user. If the user finds the matchingquestion and answers satisfactory, he/she can terminate the Q&A
session by canceling the new question asking process. Otherwise,
the question is routed to the users who have subscribed to someof the chosen category labels and/or to domain experts identified

There are two goals of SI : improving search experience andenhancing search quality. To improve search experience, SI provides the user an opportunity to ask a question when search results
are not satisfactory. The goal of enhancing search quality is more
subtle but equally important. A search user tends to seek factual information. Thus, should he/she ask a question, the desired answers
are probably objective, fact-based ones. If such fact-based answers
are provided, they may be indexed and used for providing bettersearch results to subsequent similar queries (thus potentially eliminating the need for a later repeated question submission by anotheruser). Currently, SI triggers a Q&A session under the following
two conditions:
1. When the query is in a question form, or
2. When the computed relevance of all the search results to thequery (whether in a question form or not) falls below somethreshold.
Confucius classifies search keywords into eleven categories, listed
in Table 2. We assembled the list of categories by sampling a set ofquestions and manually annotating them. The resulting categories
are not exhaustive, though they still cover the majority of the questions. SI uses a multi-class SVM classifier with word features. The
classifier is trained offline. When the terms of a search query can
be classified into one of these eleven categories with a high SVM
margin, SI triggers a Q&A session.
The search result evaluator uses two signals to identify queries
result sets that are not relevant enough. The first signal is based on
the maximum query-independent quality of any page in the result
set. The query-independent page quality can be approximated using algorithms such as the well-known PageRank algorithm [3]. IfCategory
REC
FAC
YNO
HOW
WHY
TSL
DEF
OPN
TRA
INT
MAT

PurposeRecommendation
Seeking facts
Yes/No decision
How-to question
Seeking reason
Translation
Definition
Seeking opinion
Transportation
Interactive
Math problem

Expected answer
Suggested item and the reason
Date, location, or nameYes/no and reason
The instructions
The explanation
Translated text in target language
The definition of given entity
Opinions of other users
Navigation instructions
Discussion thread
Solution with steps

Table 2: Categories of questions in NL.
Rank
1
2
3
4
5
6
7
8
9
10

From Community
GossipWeb
Software
Online Games
RelationshipEntertainment
Downloads
Mood
TravelMedicine

From Web Users
Medicine
SoftwareNews
HardwareEngineering
Web
Online Games
Technology
EconomyProducts

Table 3: Top labels of questions, from community users andfrom SI-directed users.
for a query q, no result page has a high PageRank score, one cannaturally expect user dissatisfaction. The second signal is the presence or absence of a user click on the Next link on the search result
page. Moving to the second result page or beyond implies that theuser was less than perfectly happy with the top results.
When at least one of the two triggering conditions is satisfied, thesearch results page will include an additional link to Confucius at
the bottom of the result list, with a suggestion like “Would you liketo ask a question about search terms on Confucius?” Upon clicking
on the link, the user is led to Confucius’ question submission page.
On that submission page, the user can refine the question and obtain possible existing answers by browsing previously-asked, similar questions on the right-hand side. If the user has not registeredon Confucius, he/she may choose to ask the question anonymously,
and leave an email address for receiving notifications when answers
become available.
SI may appear to be a straightforward integration, but it yields avariety of significant benefits. First, it provides a substantial flowof questions to Confucius. At a reasonable question triggering rate,
SI can bring tens of thousands of questions to Confucius per day.
Also, as we mentioned, search integration yields not only quantity, but quality as well. Table 3 shows the most frequent question labels collected on our Chinese site, from community users
and from SI-directed users, respectively. Before search integration
was launched, 90% of the submitted questions were of subjective
nature, soliciting opinions on gossips, relationships, fashion, andentertainment. With SI-directed questions, Confucius has begun
to cover more objective topics, such as medicine, engineering, andeconomy.

3.2 Question Labeling (QL)
The goal of QL is to help organize and route questions with automatically recommended labels. The QL subroutine takes a question
as the input and outputs an ordered list of labels that best describe

Question Labeling Subroutine (Offline Training)
Input: Questions Q = {q1 , . . . , qn }, in which qi ={wi,1 , . . . , wi,n }, Labels L = {l1 , . . . , lm } and their relationshipwith labels R ∈ Q × L
Output: LDA models M = {(θ, φ, k)}
1: for l ∈ L
2:dl = {w|w ∈ d, ∀(d, l) ∈ R}
3: Remove stop words and rare words from all dl , l ∈ L
4: M = {}5: for k ∈ {32, 64, 128, 512}6:
Train LDA model (θ, φ, k), with {dl } and k topics7:
M ← (θ, φ, k)
Question Labeling Subroutine (Online)
Given: LDA Models M = {(θ, φ, k)}
Input: Question q = {w1 , . . . , w|q| }Output: Suggested Labels Lq = {l1 , . . . , ln }1: Infer θq,k with M
2: Sk (θq , θdl ) = CosSim(θq,k , θdl ), ∀l ∈ L
3: S(q, l) = {Sk (θq,k , θdl ,k )} /* Mean similarity */4: Lq = {l||{l′ |S(q, l′ ) > S(q, l)}| < N } /*Top N labels*/Figure 3: QL subroutine.
the question. Labels consist of a set of words or phrases that best
describe the topic or type of the question. Confucius allows at most
5 labels per question, but puts no limit on the size of the global
label vocabulary. Confucius organizes the most important categorylabels into a two-layer hierarchy, in order to provide a better browsing experience. QL is used by two other subroutines: UR and QR .
When ranking users, UR uses popular labels to compute the topicdependent rank scores. QR assigns questions to users via either
subscription or expert identification, during which labels generated
by QL are used for matching. The precision and recall of suggested
labels are two important metrics for measuring QL performance.
Precision measures the correctness of suggested labels, while recall
measures the completeness. In Confucius, the precision and recallof QL are measured by online user feedback, which we present in
Section 4.
Figure 3 shows the two parts of QL: offline training and online
suggestion. In the offline training part, we employ Latent Dirichlet Allocation (LDA) [2], a latent semantic model, to model the
relationship between words and topic labels. The training data is
the existing set of questions with user-submitted labels. First, wemerge all questions with the same label l into a meta-document dl ,and form a set of meta-documents {dl } (Figure 3, steps 1-2). Second, we remove all stop words and rare words to reduce the sizeof each meta-document (step 3). Third, we use {dl } as the corpus
to train LDA models (steps 5-6). The label corresponds to the document in LDA definition, while the words in the meta-documents
correspond to the words. PThe resulted LDA model decomposes
the probability p(w|l) to z p(w|z)p(z|l)—this is similar to thefactor model in recommendation algorithms, expressed in terms ofprobabilities. Instead of a single model, QL trains several LDA
models with different number of latent topics. Using multiple LDA
models with different k-s is known as bagging, which typically outperforms a single model and avoids the difficult task of setting an
optimal k, as discussed by Hofmann [5].In the current QL system,
the following numbers of topics are used: k = 32, 64, 128 and
256. We collect all LDA models into a set M (step 7) and save it to
disk. The training part works offline. To handle large training data,
we use a parallel implementation called PLDA [12] on thousandsof machines in order to maintain training time within the range ofa few hours.

The online suggestion part assigns labels to a question as theuser types it. The bottom half of Figure 3 depicts the suggestion
algorithm. First, we use each LDA model in M to infer the topic
distributions {θq,k } of the question q (step 1). Then, we computethe cosine similarity CosSim(θq,k , θdl ,k ) between θq,k and θdl ,k(step 2). Third, we use the mean similarity over different values ofk as the final similarity S(q, l) between a question and a label (step3). Finally, we sort all l ∈ L by S(q, l) in descending order, andtake the first N (say ten) labels as recommended ones (step 4).
Using PLDA for QL has two benefits: semantic matching andscalability. PLDA decomposes each question and answer into a
distribution over a set of latent topics. When encountering ambiguous words, PLDA can use the context to decide the correct semantics. For example, QL suggests only labels such as ‘mobile’ and‘iPhone’ to the question “How to crack an apple?”, although theword apple also means the fruit “apple.” In addition, PLDA is designed to scale gracefully to more input data by employing moremachines. The scalable implementation enables SI to handle fastgrowing training sets to deliver more accurate suggestions.

3.3 Question Recommendation (QR)
Question Recommendation Subroutine (Offline Training)
Input: Questions Q = {q1 , . . . , qn }, in which qi = {wi,1 , . . . , wi,n }. Answers
A = {a1 , . . . , an }, in which ai = {wi,1 , . . . , wi,n }. Labels L = {l1 , . . . , ln }.
The relation RQA = Q × A between Q and A, and the relation RLQ = L × Q
between labels and questions
Output: LDA model Mq2q = (θ, φ, k), inverted word index Iw ∈ W × Q, andinverted label index Il ∈ L × Q
1: Remove stop words from all q ∈ Q
2: Train LDA model (θ, φ, k), with {q} as the corpus and k topics
3: Iw = {} /* Build word index */4: for q ∈ Q
5:for w ∈ q
6:
Iw ← (w, q) /* Build index with words of the question*/
7:
for w ∈ a, ∃(q, a) ∈ RQA
8:Iw ← (w, q) /* and its answers */9: Il = {} /* Build label index */10: for q ∈ Q
11:for l, (l, q) ∈ RLQ
12:Il ← (l, q)
Question Recommendation Subroutine (Online)
Given: LDA Model (θ, φ, k) and inverted index Iw , Il , the current time tn ow,
the asking time tq of each question q as days, Each word’s inverted document
frequency IDF = {idfw }, Time-sensitive words Wt
Input: Question q = {w1 , . . . , w|q| }Output: A list of recommended questions Q′ = {q1′ , . . . , qn′ }1: Build candidate set Q′ = {q ′ |(w, q ′ ) ∈ Iw , ∀w ∈ q, or(l, q ′ ) ∈ Il }2: Infer topic distribution θq with LDA model (θ, φ, k)
3: Build TF*IDF word vector γq with IDF .
4: Find question location lq
5: for q ′ ∈ Q
6:Infer topic distribution θq′ with LDA model (θ, φ, k)
7:
Build TF*IDF word vector γq′ . with IDF
8:
Sim(q, q ′ ) = CosSim(θq , θq′ )α1 CosSim(γq , γq′ )α29:Find question location lq′
10:LocationBoost(q, q ′ ) = EuclideanDistance(lq , lq′ )−111:
if q ∩ Wt 6= ∅14:
F reshness(q ′ ) = (tnow − tq′ )−1
13:
else
12:
F reshness(q ′ ) = 1
15:
Quality(q ′ ) = max{quality(a), a ∈ q ′ }16:Rank(q ′ ) = Sim(q, q ′ )λ1 LocationBoost(q, q ′ )λ2
F reshness(q ′ )λ3 Quality(q ′ )λ4
17: sort all q ′ ∈ Q′ by Rank(q ′ ) in descending orderFigure 4: QR subroutine.
QR recommends similar, existing questions to a user when he/sheissues a question. The goal is to recommend relevant questions that
already have good answers so that the user can access the solicited
information faster. QR formulates the question recommendation

problem exactly in the same way as QL formulates the label suggestion. Given a question, QL assigns relevance scores to all existing questions and ranks them accordingly. The input to the QR
subroutine is a question, the output is a list of questions sorted bytheir relevance scores. Similarly to QL, QR also employs PLDA
and consists of an offline training part and an online recommendation part.
Figure 4 presents the training and recommendation algorithms.
The offline part builds an LDA model and an index for all questions (Figure 4 top). First, we remove all stop words from thequestions and train a PLDA model (steps 1-2). Then, we build
an inverted word index of all questions (steps 3-8). The purposeof inverted indexing is to shrink the candidate question set, sincecomputing similarity between all questions is a computationally intensive task. However, the word index may miss candidates, sincea typical question contains very few words, and the same meaning
may map to multiple words. To enrich the data for an answeredquestion, we also include the words in its answers when indexing
it (steps 7-8). QR also indexes questions using labels generated by
QL (steps 9-12).
After a user has typed in a question but before he/she hits the submit button, QR recommends related questions (Figure 4 bottom).
QR uses the words in the question and its labels to query the index.
The top N results are returned as the recommended questions (step1). QR then adds additional factors to rank these returned questions. The considered factors include similarity, freshness, quality,
and location boost (steps 8-12), detailed as follows:
Similarity. Similarity is the main factor in candidate question
ranking. Questions are short, introducing the word sparsity problem in similarity computation. We take a corpus-based approach inquestion similarity computation using PLDA. The similarity scoreis computed in step 8, where Sim(q, q ′ ) is the similarity score between questions q and q ′ , and θq is the topic distribution vector of q.
Parameters α1 , α2 are blending factors for topic-based and wordbased similarities. The similarity measure uses both words andtopics—the word vector is used to emphasize exact matching, while
the topic vector is used to capture non-overlapping synonyms.
Freshness. For some questions, up-to-date answers are preferable to old ones. For instance, for “Who is the winner of the latest
Olympic gold in men’s 100m sprint?.” QR emphasizes recent answers through freshness scores. QR uses a set of manually selected
trigger words Wt , such as today and recently, to identify questions
that expect a recent answer. If the input question contains a trigger
word, QR computes the freshness score of each candidate question
as the reciprocal of the days since the question was answered (step12). The newer the question q’s answer(s), the higher the freshness
score F reshness(q) (Freshness is set to one when an answer wasprovided today.) If the input question does not contain any trigger
words, QR sets all freshness scores of the answers to 1, so as not to
induce suffer unwarranted demotion in ranking (step 14).
Quality. Besides freshness, QR also considers the quality of
the recommended questions. If two perfectly matching questions
exist, one with a best answer, and the other with a spam answer,
we would like to display the best-answered one first. Quality(q)measures the quality of a question by the quality of its answers (step15), where the answer quality quality(a) is provided by user feedback and our answer quality predictor (Section 3.4). If an answer
is marked as a best answer, we set quality(a) = 1. If an answer
was reported as spam, we set quality(a) = 0. Otherwise, we use
the automatic quality predictor to predict the quality of all answers
to that question, then use the maximum answer quality as the question’s quality score. The quality of an unanswered question is set
to zero.

FactorRelevance

Parameter#rword#rlda

Originality#origt#origuTimeliness#prompt

Coverage

#cov
#nau

Spam

#npbau#naul#npbaul#entropy

Description
Computed as the cosine similarity between
TF*IDF weighted word vectors of the question
and the answer.
Computed as the cosine similarity between the
PLDA latent topic distributions of the question
and the answer.
Compared to earlier answers for the same question.
Compared to earlier answers from the sameuser.
The time from when the question was asked towhen the answer was provided.
The sum of the unique words’ IDFs.
Total number of answers provided by the answerer in the past.
Percentage of best-answers in all closed
threads in which the answerer has participated.
Number of answers posted by the answerer toquestions on the same topic.
Percentage of best answers posted by the answerer in closed threads on the same topic.
The entropy of the user’s topic distribution ofanswers.

Table 4: Quality factors and parameters in AQ .
Location Boost. Some questions are closely related to geographical locations, such as questions asking for the best restaurant
in a city. To emphasize location in question matching, we add location boost to the final rank score (steps 9-10). The location mentioned in the question is extracted using a standard named-entity extractor, such as one for country and city names. Then, we convert
the location description to geographical coordinates (steps 4 and
9). Location boost is computed as the reciprocal of the geographical distance between the current question and an earlier candidate
question. If no location-related entities are detected in the originalquestion, we consider it to be a location-independent question, andset all candidate questions’ location boost to 1.
In step 16, QR combines four factors into a final score Rank(q)using weights λ, which are chosen empirically. In the final step
(step 17), QR sorts all candidate questions in descending order by
their Rank(q), and returns the top ones to the user.

3.4 Answer Quality Assessment (AQ)
User feedback, such as best-answer votes, is a commonly utilized way of assessing answer quality in community Q&A sites.
However, the majority of answers do not have enough votes, andfresh answers are not likely to obtain any votes soon. We leverage an automatic answer-quality assessment routine AQ to produce
quality ratings. Since AQ can be trained and executed off-line, its
primary design goal is accuracy over speed.
We model the AQ task as a binary classification task of best andnon-best answers. Each instance is an answer, represented by a setof features. The classifier outputs a confidence score for an answer
being a best answer, which can be deemed as a quality score. The
training data is Q&A threads with user annotated best answers. We
consider five groups of quality factors as features: relevance, originality, timeliness, coverage, and spam. These quality factors arequantified by parameters enumerated in Table 4.
Relevance. An answer must be relevant to the question in a Q&A
thread. We use the same approach as in question recommendation
to compute the relevance between the question and the answer (thebottom of Figure 4, step 9). We also share the same offline models
with question recommendation.

Originality. Content copying is widespread and mostly unavoidable on Q&A sites. Some duplication—such as straightforward
plagiarization of other users’ answers or using the same text to answer multiple questions—is clearly unwanted and the authors’ reputation should be diminished accordingly. Parameter #origt indicates how an answer differs from other answers on the same Q&A
thread. Parameter #origu indicates how an answer differs from theother answers provided by the same user. To quantify content originality, we leverage BleuScore [15], which is a standard metric in
machine translation for measuring the overlap between n-grams of
two text fragments.
Timeliness. The time it takes to answer a question is relevant
to quality. However, a short response time may not always indicate high quality. Whereas some questions can be answered very
quickly, well thought-out answers to some others (think collegelevel science and mathematics, for instance) may take time. For thislatter kind of questions, users providing prompt answers are more
likely to be spammers than genii. Therefore, though the parameter #prompt is simple to compute, its quality interpretation may be
topic-dependent. Rather than relying on an explicit rule for applying the timeliness factor, we use supervised learning to weight thisfactor through the training process.
Coverage. The length of an answer can imply high quality if the
response time is not too short, the answer is highly relevant, and thecontent is original. Since we want to emphasize the useful part ofan answer, we measure content coverage by summing the inverse
document frequencies (IDF) of its unique words (#cov).
Spam. Spam signals are an important factor in determining answer quality. Some individuals may have strong economic incentives to attract post views by users. To achieve their goal, spammers
generate a large number of posts using multiple accounts. To curbcommon spam, such as commercial and porn spam, we train targeted porn and commercial classifiers for identifying corresponding posts. Identified questions/answers can be traced back to theperpetrating users. Then, once a spammer is identified, the rest of
his/her content can be surfaced in a transitive manner.
Besides the monetization of content visibility, potential reputation within a community may also motivate spammers. This is particularly true if a service offers financial rewards to its power users,
as we do in Confucius. To identify active spammers, an #entropy
score is computed for each user based on the topics that his/her answers cover. Let θ be the topic distribution of an answer that is inferred by PLDA. Then, the focus scoreP
is the average
Pk entropy of all
answers of that user, #entropy(u) = N(−
i=0
j=0 θi,j log(θi,j )),where N is the number of answers of the user and θi,j the probability that the ith interaction is on the j th topic. In other words,#entropy(u) measures the uncertainty in user u’s latent topic distribution. Thus, a higher #entropy(u) means that the user tends
to answer more types of questions, whereas a low #entropy(u)indicates that a user focuses on a small set of topics.
With our training data and features, we take a data-driven approach to finding an appropriate prediction algorithm. AQ uses
cross-validation to choose the most appropriate classification algorithm. The cross-validation accuracy of different classifiers ispresented in Section 4.

3.5 User Ranking (UR)The goal of the UR subroutine is to find the high quality users
on different topics. As depicted in Figure 5, UR consists of threemain steps. First, we convert the Q&As to topically weighted interactions between users (step 1). Then, we generate the user activity
graph G from the interactions (step 2). Finally, we run a weighted
and topic-sensitive HITS computation on G (step 3), and combine

Figure 5: The UR subroutine. Two different topics are represented by solid and dotted lines. Line widths convey the weightsof the interactions.
the hub and authority scores into a final UR score.
Each pair of question and answer is an interaction between theasker and the answer provider, denoted as (uf , ut , w), where uf
is the answer provider who followed in this interaction, ut is theasker who started the interaction, and w is the topical weight. To
get topical weighting, we use the LDA model from QL once again
to infer for each interaction the topic distribution of the question,
denoted as θ. Then the topical weight vector w can be computed
as wi = wq θi , where wq is the quality score computed by AQ in
Section 3.4.
We use the user activity graph to connect users by interactions.
An edge from uf to ut represents one or more interactions from uf
to ut . Each edge has an assigned weight vector w(uf ,ut ) , where(u ,ut )represent the weight for the ith topic. We compute thewi fedge weight from interactions as:(u ,ut )

wi f

wi′

X

=

∀(uf ,ut ,w′ )∈Yuf ,ut|Yuf ,ut |

,

(1)

′

X
u′ ∈in(u)(n)

Pwu ,u · au′PkPwu,u · hu′
Pku′ ,u∗
i=0 wiu∗ ∈out(u′ )′a(n+1)= (1 − ǫ)uXu′ ∈out(u)u∗ ∈in(u′ )

+ǫ

(2)

+ǫ

(3)

(n)

u∗ ,u′
i=0 wireach out to other users, and the authority score represents a user’s
ability to gain attention from the others. In Confucius, providing
more good answers is encouraged, so hub scores may be valued
higher. We take a linear combination of the hub score and the authority score to produce a single final UR score. The final topicsensitive score vector is φu = γhu + (1 − γ)au , where γ is the
mixture weight. There are two ways to choose γ. If one has sometraining data (that is, a list of manually ranked users) readily available, one can learn the most appropriate γ by regression. If one
does not have training data and it is expensive to obtain it, one canset γ empirically. For example, one can use γ = 1 to emphasize
answering behavior.

3.6 NLP-based Answer Generation (NL)

where Yuf ,ut represents the set of all interactions from uf to ut .
Accordingly, for each topic, we use the mean weight of the interactions as the edge weight. The mean is preferable over the sum, as
it reduces the effect of a large set of low-quality interactions.
We adopt Ng et al.’s randomized HITS [14], which adds a universal random jump probability to standard HITS [8]. To incorporate
topic distribution, we use vectors to represent the topic-specific hub
and authority scores and edge weights, namely, h, a and w(ui ,uj ) .
The number of dimensions is the number of topics k. The valuesin w(ui ,uj ) represents the topic-specific weight of the interactions
between user ui and uj . We use the following vector-based HITS
formula to get h and a :h(n+1)= (1 − ǫ)uFigure 6: The NL subroutine.

where (·) represents the Hadamard (or entry-wise) product of two
vectors. Weighted topic-specific HITS propagates topic-specific
hub and authority scores through the edges. The final h and areflect the user’s reputation scores for different topics. When there
is only one topic, weighted topic-specific HITS becomes weighted
HITS.
The result of HITS on the user activity graph captures two aspects of user reputation: the hub score represents a user’s ability to

NL generates an answer from existing documents to answer aquestion. Figure 6 shows the four key steps of the NL subroutine.
First, NL classifies a question as we discussed for the SI subroutine.
Based on the category, NL delivers the question to two sub-servers:
the one-box server and the community database server. Each subserver uses its own method to generate an answer a with some confidence score C(a) ∈ [0, 1]. Besides categorization, NL employs astandard named-entity recognizer to extract named entities from thequestion. Named entities may include locations, dates, company
names, person names, and product names. Entities are provided tothe sub-servers to perform further processing.
The first sub-server is the one-box sub-server. One-boxes arecommon features of web search engines, which aim to provide
direct answers instead of page snippets to simple and structured
queries, such as “San Francisco weather” or “movies in Beijing.”
Usually, the one-box server can answer queries such as time, date,currency exchange rates and other financial data, math calculations,route planning, machine translation, and weather. NL sends TSL,TRA and MAT questions (see Table 2) to the one-box sub-server.
If the one-box server finds an answer, we set the confidence score
to 1, otherwise 0.
The second sub-server is the community database sub-server,which indexes all questions with high quality answers in the Confucius database, and provides the answer by finding the matchingquestion. The basic routine for finding similar questions mimics the
QR subroutine (Section 3.3). However, while QR focuses on providing a list of related questions to the user, NL focuses on provid-

Average Waiting Time (Minutes)

New Answers Per Day

New Questions Per Day

From Anonymous Users
From Login Users(c) AWT

Click Through Rate(b) NA

Coverage

Pageview Per Day

1
/1
09
20 0
/1
09
20 09
/
09
20 08
/
09
20 7
/0
09
20 06
/
09
20 05
/
09
20 4
/0
09
20 03
/
09
20 02
/
09
20 1
/0
09
20 12
/
08
20 11
/
08
20 0
/1
08
20 09
/
08
20 08
/
08

20

1
/1
09
20 0
/1
09
20 09
/
09
20 08
/
09
20 7
/0
09
20 06
/
09
20 05
/
09
20 4
/0
09
20 03
/
09
20 02
/
09
20 1
/0
09
20 12
/
08
20 11
/
08
20 0
/1
08
20 09
/
08
20 08
/
08

20

1
/1
09
20 0
/1
09
20 09
/
09
20 08
/
09
20 7
/0
09
20 06
/
09
20 05
/
09
20 4
/0
09
20 03
/
09
20 02
/
09
20 1
/0
09
20 12
/
08
20 11
/
08
20 0
/1
08
20 09
/
08
20 08
/
08

20

(a) NQ

1

0

/1

09

20

9

/1

09

20

8

/0

09

20

7

/0

09

20

6

/0

09

20

5

/0

09

20

4

/0

09

20

3

/0

09

20

2

/0

09

20

1

/0

/0

09

09

20

20

1

0

/1

09

20

9

/1

09

20

8

/0

09

20

7

/0

09

20

6

(e) COV

/0

09

20

5

/0

09

20

4

/0

09

20

3

/0

09

20

2

/0

09

20

1

/0

/0

09

09

20

20

1
/1
09
20 0
/1
09
20 09
/
09
20 08
/
09
20 7
/0
09
20 06
/
09
20 05
/
09
20 4
/0
09
20 03
/
09
20 02
/
09
20 1
/0
09
20 12
/
08
20 11
/
08
20 0
/1
08
20 09
/
08
20 08
/
08

20

(d) NPV(f) CTR

Figure 7: Overall statistics for Confucius China. Following the initial deployment, SI was launched in August 2008, UR was launchedin September 2008, and QR was launched in February 2009.
ing accurate answers. Thus, we consider two more factors on top of
the QR subroutine: entity matching and category matching. Entity
matching is computed as the number of matched entities between
the input question and existing questions. Category matching is anindicator function, which outputs one if the category of the input
question is the same as that of the existing question, and zero otherwise. We send REC, YNO, FAC, HOW, WHY, INT, and OPN questions to the community database sub-server. The confidence scoreof a returned answer is computed as Rank(q, q ′ ) ∗ quality(a),where Rank(q, q ′ ) is the similarity rank between the question q
and the indexed question q ′ , and quality(a) is the quality of theanswer provided by the AQ subroutine.
After collecting all sub-servers’ responses, the main server sorts
all answers by their confidence scores. The answer with the highest
confidence score is returned to the user. NL cannot cover all questions, especially those seeking non-factual answers. If the confidence scores of all sub-server answers are lower than a pre-determinedthreshold, NL will not return any answers.

4.

PRODUCT STATISTICS AND EXPERIENCEThis section reports some key statistics and evaluation results of
the Confucius system. In particular, we present how the six subroutines have helped us harness the synergy between search andcommunity.

4.1 Overview
Confucius was launched in 21 countries so far. Among thesecountries, Russia was the first one, followed by China, Thailand,
and Egypt. We use the statistics collected for Confucius China toreport on our experience. Since some raw performance figures constitute confidential proprietary information, we opt to report percentages of improvement rather than absolute numbers on question/answer volume, page views, and click-through rates.
Figure 7 shows the overall statistics for Confucius China, alongwith the approximate launch time of the six subroutines. QL waslaunched at the very beginning, so it is not shown separately. We
discuss several individual metrics as follows:
NQ: Referring to the number of new questions per day, it represents a system growth indicator. Note that Confucius features

SI, which allows anonymously posted questions. Hence, we show
NQ for logged-in and anonymous users separately in Figure 7 (a).
Compared to logged-in users, NQ for anonymous users is muchhigher since SI was launched. This indicates that SI is the sourceof a large number of new questions.
NA: Referring to the number of new answers per day, it represents another growth indicator. As the number of questions increases, the number of answers grows too. Figure 7 (b) shows NA
for Confucius China.
AWT: AWT is the average waiting time, which is computed as
the number of minutes between the submission of the question andthe first answer. AWT measures the timeliness of answers, which isa central goal in Confucius. Figure 7 (c) shows the AWT for Confucius China. Along with the growth of the community, AWT drops
gradually. The steady decrease of AWT shows once again that thecommunity is capable of handling questions from SI. Typically, a
new question in Confucius China will receive its first answer within
one hour. The shortening AWT provides askers with positive feedback and encourages them to ask more questions.
NPV: NPV is the number of page views per day, a widely used
popularity metric for web sites. For a Q&A site, more questions
and answers are only part of the goal, while letting more users benefit from the Q&A database is also important. Figure 7 (d) shows
the NPV for Confucius China, indicating that Confucius is becoming more popular. Notice that NPV increased significantly aroundFebruary 2009, which corresponded with the launch of QR . Besides showing QR recommendations on the question submission
page, Confucius also put QR on every Q&A thread page. Allowing users who browse existing Q&As to review related questions
boosted the NPV of Confucius, and let users spend more time on
the site. NPV also relates to content quality: should users encounter
low quality content on Confucius over and over again, they would
not come back and NPV would stop increasing. This is certainly
not the case at this time.
COV: The coverage of Confucius content in web search result, iscomputed as the percent of queries that contain results from Confucius on the first page (10 results). High COV is another indicator ofcontent quality, since site reputation and overall quality are important factors in the independently generated search engine ranking.

1.00
RecallF1
Precision

0.80
0.60
0.40
0.20
0.00

20

20

20

20

09

09

09

09

1
/1

0
/1

9
/0

8
/0

Figure 7 (e) shows COV for Confucius China in 2009, indicatingthat content quality is increasing and more users get to see resultsfrom Confucius while searching the web.
CTR: The click-through rate of Confucius results in web searchis also a direct measure of the content quality in Confucius. Figure 7 (f) shows that the CTR for Confucius China was increasingover the last year.
Next, we show the performance of each subroutine with detailed
evaluation data.

20

20

20

20

20

20

20

20

/0

6

7

8

09

/0

4

09

/0

3

/0

/0

2

5

/0

1

/0

/0

09

09

09

09

09

09

SI leads unsatisfied users from Web search to community Q&A.
We use trigger rate and acceptance rate to measure SI ’s performance. Trigger rate is computed as the percentage of queries thattrigger the link to Confucius. Acceptance rate is computed as thepercentage of triggered links that user actually clicked. The higher
the acceptance rate, the more users expect SI links to be useful. On
the one hand, we want SI to cover as many queries as possible; on
the other hand, showing the Confucius link for completely irrelevant queries hurts the user experience, which we want to avoid.
Figure 8 shows the two rates of SI of Confucius in China.

Click-Through Rate

Figure 9: Precision, recall and F1 measure of QL .

4.2 Search Integration

Figure 10: CTR of recommended questions by QR.
vors questions that require objective answers (such as medical andbusiness related), whereas community users tend to favor qeustions
with subjective answers.

4.3 Question Labeling(a) Acceptance(b) Trigger Rate

Figure 8: Acceptance(a) and Trigger Rate(b) of SI .
Recall that SI uses two algorithms to decide about the visibilityof the Confucius link: question recognizer and search result relevance assessor. At first, we enabled the question recognizer only,
without the search result relevance assessor (before May 2009).
The coverage reached a stable level quickly after launch. The CTR
was very high in the first several months, then returned to a stablevalue. Users’ curiosity was the main reason for the higher CTR in
the beginning, as they were not familiar with the Confucius link,
and might have expected instant and correct answers by following
the link. After users became familiar with the functionality of the
Confucius link, they tended to click less often, only when it wasnecessary to fall back on community answers. In May 2009, welaunched the search result relevance assessor in SI to work alongwith the question recognizer. The first thing that changed was thecoverage, which quickly increased by four times. As the coverageincreased, the CTR dropped temporarily. However, after users had
adapted to the SI logic change, the CTR returned to its previous
level. So, the overall result of launching the second algorithm wasan increase in the number of questions initiated from web search,which did not compromise the other metrics.
Besides yielding more questions, SI also helps us bridge theneeds of web users and community members. Prior to SI , allquestions in Confucius were submitted by logged-in community
users. The topics of interest of community users, often biased towards subjective ones, are not necessarily the same as those of web
users. Table 3 (presented in Section 3.1) shows the label distributions of questions from community users and web users. SI fa-

We measure the effectiveness of QL by user feedback. For eachnew question, QL outputs the suggested labels as an ordered list.
The user can choose from the recommended labels, or add newlabels manually. The user can choose at most five labels for one
question. We keep the log of what labels are suggested and whatlabels the users used, so as to make automatic evaluation possible.
We use precision, recall, and F1 to measure the effectiveness of
QL . Precision measures the correctness of suggested labels. For
each question, precision is defined as (|LS | ∩ |LU |)/|LS |, where
LS is the set of suggested labels and LU is the set of user-chosenlabels. Recall measures the completeness of suggested labels andit is defined as (|LS | ∩ |LU |)/|LU |. F1 is the geometric mean of
precision and recall, computed as 2P R/(P + R). Precision, recall,and F1 are widely used metrics for recommender systems. Since at
most five labels are allowed, we compute precision@5, recall@5
and F1@5.
Figure 9 shows the precision, recall, and F1 of QL for Confucius China. First, the recall of QL was high and increasing, from
0.6 to near 1. Higher recall means that users tend to select fromamong the recommended labels instead of creating new ones. As
QL only recommends labels with high frequency, such behavior
helps to limit/reduce the size of the label vocabulary. Consideringthat the mean number of labels for a question is 2.32 (which suggests that users prefer fewer labels), the precision at around 0.4 isalready quite promising.

4.4 Question Recommendation
Figure 10 shows the CTR of recommended questions after thelaunch of QR . The higher the CTR, the more useful the recommended questions to the user. As we can see, the CTR increased
quickly after the launch, then stabilized after the first month. Whenstablized, the CTR is quite high (around 0.38). The rapid adoptionof QR means that the users found it helpful in identifying related
questions. QR brings two benefits to Confucius. First, if the user
finds a related question on the question submission page, he/sheTable 5: Side-by-side experimental results for UR on improving search results. All control groups are Google Search results. Impact
indicates the fraction of queries that differ in quality between the experimental and control groups. Mean score is the sum of ratingscores divided by the number of queries with agreement. Larger mean score is better.
Experimental Groups
UR without graphUR without weight
UR without topic
Full UR

Number of Queries at Different Ratings

-2
0
0
5
4

-1.5
1
6
6
8

-1
0
2
9
6

-0.5
18
30
16
17

Table 6: The accuracy of best/non-best answer classification,
averaged over 10-fold cross-validation.
Algorithm
Random Forest
AdaBoost
Logistic Regression
SVMs (RBF)
SVMs (polynomial)
SVMs (linear)
Perceptron
SVMs (sigmoid)
Naive Bayes

Accuracy
0.770
0.753
0.739
0.698
0.696
0.661
0.641
0.527
0.499

may get to the answer directly without submitting the question.
This reduces the redundancy in the Q&A database. Second, within
the community, active users are the source of new answers. Moreclicks on QR links keep the users engaged within the community
for longer periods of time, thus increasing the probability that they
contribute answers.

4.5 Answer Quality Assessment
We prepared the training data for AQ from user voted Q&A
threads. Specifically, we used the Q&A threads that contain askerselected best answers. Threads that contain less than 3 answers
were not used, since the less participation, the more risk of it being
spam. For each thread, we used its best answer as the positive example, and randomly sampled an answer not marked as the best as
the negative example. We collected 100, 000 pairs of positive andnegative samples for the experiment.
As described in Section 3.4, AQ uses a data-driven approach toselect the best algorithm for answer quality prediction. Thanks to
our massive distributed computing facility, we could test a number of algorithms in a short time, covering all possible parameterchoices. We included Naive Bayes, Logistic Regression, Decision
Tree, Perceptron, AdaBoost, and SVMs in the selection pool. For
SVMs, we used common kernels, including linear kernel, RBF kernel, polynomial kernel, and sigmoid kernel. Table 6 lists the 10fold cross-validation accuracy of best/non-best answer classification, showing each algorithm at its best across different parametervalues.
The best result came from the Random Forest, a Decision Treebased probabilistic classifier, which outperformed the others, yielding an accuracy of 0.77. We noticed that the non-linear methods,
such as Random Forest and AdaBoost, outperformed linear ones.
This result suggests that the target prediction function is not linear.

4.6 User Ranking
To test UR, we incorporated it into web search result ranking.
Lacking internal links, community Q&A sites, such as Confucius,
can barely benefit from PageRank or similar link-based ranking algorithms. We used UR as the query-independent quality indicator

0
237
178
205
230

0.5
31
44
21
31

1
0
9
21
16

1.5
0
6
4
6

2
0
0
5
5

ImpactMean Score

0.174
0.353
0.298
0.215

0.017
0.050
0.039
0.054

for web search, and performed a side-by-side experiment to examine the changes in the search results.
We sampled 300 queries from Confucius’ search log, and retrieved the corresponding search results from Google. We used site
search restrictions2 to limit the result to the Q&As from Confucius.
For each search result, we combined the relevance score and URscore into a reordering score λRel + (1 − λ)U serRank, where
the linear mixing factor λ = 0.9 was selected by a small scalecross-validation.
We used a side-by-side experiment to evaluate the effectivenessof UR. Our side-by-side experiments were a form of double-blind
testing. Two groups of search results were shown to human raters
in the same format and in random order. In the experimental group,
the search results were ordered by the combined score, while thecontrol group used the original ordering of Google search. The
raters were not able to identify which one is the experimental group.
They were given the task to assign each query a score: Score value
2 means that side A is much better than side B, −2 means thatside B is much better than A, and other scores captures levels in
between. We let multiple raters rate the same query until there wasan agreement, or at most three raters had participated but still had
no agreement.
UR scores are determined by three factors: the graph of users,
the quality of answers, and the topic of interactions. We also evaluated each factor’s contribution by temporarily remove them fromthe computation. Table 5 shows the results of the four side-by-side
experiments. Impact measures how many results are affected by
the reordering and it is computed as |Qnonzero |/|Qagreed |, where|Qnonzero | is the number of agreed queries with non-zero ratingsand |Qagreed | is the total number of agreed queries. The greater
the impact, the more queries are affected. Mean score measures
the improvements for the agreed queries and it is computed as(S+ + S− )/|Qagreed | , where S+ is the sum of positive ratingscores, and S− is the sum of negative rating scores. The higher themean score, the better the experimental group is, compared to thecontrol group. Queries with rating disagreement are ignored when
computing impact and mean score. All control groups are Google
Search results. “UR without graph” means computing UR without
the user activity graph; “UR without weight” means computing UR
without the quality weight on each edge; “UR without topic” meanscomputing topic-independent rank scores for each user.

4.7 NLQ&A
We evaluated the NL subroutine by a set of 2500 questions sampled from web queries. We used only the queries that would trigger
SI to ensure that they are questions. We use precision and coverageas the evaluation metrics. Precision measures the proportion of correctly answered questions among all questions that have been answered. Coverage measures the proportion of answered questions
2
Added site:wenda.tianya.cn to each Google query.
wenda.tianya.cn is the URL of Confucius China.

first answer precisionsubjectivity, which is not possible with the current set of features.
Indeed, to our best knowledge, constructing complex user models
that encapsulate such bias remains a grand open problem in natural
language processing.

1
0.95
0.9
0.85
0.8

Factoid Questions

0.75
0.7
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0.8

0.9

1

threshold for confidence score(a) Precision
0.6
coverage

0.5
0.4
0.3
0.2
0.1
0
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

threshold for confidence score(b) Coverage

Figure 11: Precision (a) and coverage (b) of NL.
among all questions. We asked three human raters to rate each answer provided by NL as good or bad. We treated an question ascorrectly answered only if all raters said its top-ranked answer was
good.
Precision and coverage are not weighted equally in practical applications. In Confucius, we favor the precision over the coverage,
since giving the wrong answer hurts the user experience more thankeeping silence. We used the threshold mentioned in Section 3.6 tocontrol the tradeoff, that if the confidence score of the top-ranked
answer was below the threshold, NL would refuse to answer thatquestion.
Figure 11 shows the precision and coverage of our preliminary
NL subroutine with respect to different threshold values. The precision increased as the threshold went higher (Figure 11 (a)), while
the coverage drops (Figure 11 (b)). When using 0.8 as the threshold, NL can answer 10% of the questions with the precision 0.9,
which is an acceptable performance for Confucius.

5.

DISCUSSIONS

Though significant progress has been made, several challenges
remain to be tackled to maintain high service quality.

Opinion Questions
Many users ask for recommendations on things to buy or for advice on life matters. Examples include “Who are more dependable, women or men?” and “Which camera brand should I go with,
Canon or Nikon?.” We call these types of questions opinion questions. Most opinion questions do not have an objective best answer.
At best, there may exist a handful of main-stream opinions, with alarge number of users supporting each. At worst, users may havesignificantly diverging subjective opinions. Each party may thenlist some supportive evidence, which can trigger a debate in thethread. If such opinion questions are ever closed, the best answer
is typically chosen based on the asker’s arbitrary perception of theoutcome of the argument.
Opinion questions present challenges to automatic answer quality assessment. The model training step presented in Section 3.4
relies on best answers as positive samples and non-best answers
as negative samples. However, for opinion questions, the distinction between a best and a non-best answers is subjective. To handle them adequately, we would need a way of capture the asker’s

We rely on relevance, coverage, and originality for AQ . However,for certain factoid questions, these factors may be insufficient. Factoid questions — about weather conditions, the solution to a particular math problem, or driving directions — typically have an
objective, short, and specific “correct” answer. For instance, theanswer to “Did it snow in Beijing yesterday?” is simply yes or no.
First, relevance does not work for these extremely short answers.
Both word-relevance and topic-relevance require some topically related terms to appear in the answer. Yes, no or the number 43 arenot likely to be found in the question, nor do they provide any evidence for topic inference. Second, higher coverage does not necessarily mean better answer quality for these questions. In contrast,
long answers for simple factoid questions may be spam. Third,
since the form of the answer to simple factoid questions is quite
rigid, most answers will be identified as copies, although the answerer is earnest.
There are possible ways to deal with factoid answer assessment.
First, we can leverage the question classifier in NL to decide a question’s type. Then, if the question type is factoid, we can check if theanswer is within the desired class and format. For instance, whenchecking the answers to “Did it snow in Beijing yesterday?”, we
can expect a positive or negative reply, like “yes” or “didn’t.” NL
also provides a way to assess answers to factoid questions. Though
not perfect, NL’s accuracy on simple factoid questions is promising, as reported in TREC QA [4]. Checking the answers with NL’s
output might help judge the quality of the answer.

Best Answer Spam
Early in the history of Confucius, spammers identified best answers
as a prime spam target. As we rely on best answer labels in training, our ranking is somewhat susceptible to this type of spamming.
In order to generate fake best answers, a spammer creates multiple
user accounts first. Then, he/she uses some of the accounts to ask
questions, and others to provide answers (most of which are typically concealed ads). If uncaught, spammers may generate prodigious amounts of fake best answers, which could have a non-trivial
impact on the quality of the machine learning model.
When the spammer’s agenda is direct advertising, we can often
identify telltale signs of his/her activity: repeated phone numbers
or URLs help us detect much of the best answer spam. However,when the spammer’s intention is to obtain higher status within thecommunity, and the perks that come with it, the spam content maylack easily identifiable patterns. Such a spammer may post low
quality answers to his/her own questions, and select those as best,
despite the presence of other, truly better answers in the thread.
There are clues that may help identify the best answer spammers.
First, the spammers have an incredible high best answer rate, compared to normal users. Screening the best answer rate list of users
can thus help identify them. Second, to be efficient, best answer
spammers tend to answer their own question quickly. We can cluster the users by their time of receiving best answer label to find the
spammers.

Question Spam
Spammers also target questions. First, they find a large document
collection to use as the source, such as an FAQ list or an archive of
historical documents (common sources include history/geographydocuments and government reports). Then, they use one accountto ask questions replicating the headings in the document collection, and use another account to submit corresponding document
sections as the answer. While such question-answer pairs are a perfect match, they do little to help the community beyond boosting
the spammers’ reputation.
Question spam will often result in irregular patterns in the distributions of both timestamps and interaction counts with different
users. These patterns help us identify spammers when performing
manual evaluation. We are considering adopting automatic routines
once the patterns stabilize.

Answerer Targeting
The most prominent characteristics of a Q&A site are anchoredin its answer collection. Without users providing high-quality andtimely answers, the system surely alienates its adopters quickly.
To motivate users to provide high-quality answers, a Q&A systemshould provide certain incentives. In this respect, Wikipedia may
be considered a most successful product (even if not a classicalQ&A system), which attracts volunteers to provide quality content
via authorship honor. However, transferring that success to a Q&A
site faces several challenges. First, Wikipedia decides its “questions” whereas Q&A accepts questions. Second, the quality bar for
Wikipedia is set high, and the low quality content can be edited
away easily. Users who are not qualified to provide quality content
are discouraged and eventually turned away. In comparison, thequality bar for a Q&A site is much lower. Virtually anyone can answer a question. Therefore, both the question flow and the answer
flow of a Q&A system are more “open” than Wikipedia—and this
openness inevitably reduces overall content quality.
To address the quality challenge, one can consider providing
more explicit virtual or monetary incentives. Such incentives often invite spammers. One way to deal with low-quality content
and spam is to strengthen content ranking algorithms so that lowquality questions and answers can be “buried” while high-quality
posts are promoted. Another way to improve quality is to target
experts or friends as potential answerers. Targeting experts cansurely improve quality, but experts may demand incentives. Targeting friends may or may not be helpful, depending on the goal of
the service. If the primary goal is to collect content to augment web
search, targeting friends, which typically yields subjective content,
may not be particularly beneficial. At the same time, if the goal is toincrease usage, targeting friends may turn a Q&A site into a forum
site—trading content quality for higher degree of user interaction.
Nevertheless, no matter what the goal is, our developed algorithms
can be adjusted to meet the system requirements.

6.

CLOSING REMARKSIn this paper, we presented our mature Q&A service along with
its six key subroutines. We pointed out the primary design goalsof Confucius and showed how the subroutines had helped to meetthem. We also discussed how the subroutines could be adjusted
to position the service as being oriented more toward social interactions or toward high-quality content generation.Our future work
will focus on improving the key subroutines.

Acknowledgement
We would like to thank the engineering, research, and product management teams for their contributions. In particular, we would liketo thank Jim Deng, Feng Hong, Greg Coladonato, Jerry Smith, Jun
Wu, Dekang Lin and Shamayn Teh for their innovative ideas.

7. REFERENCES[1] E. Agichtein, C. Castillo, D. Donato, A. Gionis, and
G. Mishne. Finding high-quality content in social media. In
Proceedings of WSDM ’08, pages 183–194, 2008.
[2] D.M. Blei, A.Y. Ng, and M.I. Jordan. Latent Dirichlet
allocation. The Journal of Machine Learning Research,
3:993–1022, 2003.
[3] S. Brin, L. Page, R. Motwani, and T. Winograd. The
PageRank citation ranking: bringing order to the web. In
Proceedings of ASIS98, pages 161–172, 1998.
[4] H. T. Dang, D. Kelly, and J. Lin. Overview of the TREC
2007 question answering track. In NIST Special Publication:
SP 500-274 The Sixteenth Text REtrieval Conference (TREC
2007) Proceedings, 2007.
[5] T. Hofmann. Probabilistic latent semantic analysis. In
Proceedings of UAI’99, 1999.
[6] D. Horowitz and S. D. Kamvar. Anatomy of a large-scalesocial search engine. In Proceedings of WWW’2010, 2010.
[7] J. Jeon, W. B. Croft, and J. H. Lee. Finding similar questions
in large question and answer archives. In Proceedings of
CIKM’05, pages 84–90, 2005.
[8] J. Kleinberg. Authoritative sources in a hyperlinked
environment. Journal of the ACM, 46(5), 1999.
[9] J. Ko, E. Nyberg, and L. Si. A probabilistic graphical model
for joint answer ranking in question answering. In
Proceedings of SIGIR’07, 2007.
[10] Y. S. Lai, K. A. Fung, and C. H. Wu. FAQ mining via list
detection. In Proceedings of the Workshop on Multilingual
Summarization and Question Answering, 2002.
[11] F. Li, X. Zhang, J. Yuan, and X. Zhu. Classifying what-type
questions by head noun tagging. In Proceedings of COLING
2008, 2008.
[12] Zhiyuan Liu, Yuzhou Zhang, Edward Y. Chang, andMaosong Sun. PLDA*: Parallel latent dirichlet allocation
with data placement and pipeline processing. ACM
Transactions on Intelligent Systems and Technology(Accepted), 2010.
[13] V. Murdock, D. Kelly, W.B. Croft, N.J. Belkin, and X. Yuan.
Identifying and improving retrieval for procedural questions.
Information Processing and Management, 43(1), 2007.
[14] A.Y. Ng, A.X. Zheng, and M.I. Jordan. Stable algorithms forlink analysis. In Proceedings of SIGIR’01, 2001.
[15] K. Papineni, S. Roukos, T. Ward, and W. J. Zhu. Bleu: amethod for automatic evaluation of machine translation. In
Proceedings of ACL’02, pages 311–318, 2001.
[16] D. Radev, W. Fan, H. Qi, H. Wu, and A. Grewal.
Probabilistic question answering on the web. In Proceedingsof WWW ’02, 2002.
[17] X. Si, Z. Gyongyi, E. Y. Chang, and M.S. Sun. Scalable
mining of topic-dependent user reputation for improving user
generated content search quality. Technical report, Google,
2010.
[18] M. Surdeanu, M. Ciaramita, and H. Zaragoza. Learning torank answers on large online QA collections. Proceedings of
ACL-08: HLT, 2008.
[19] X. Xue, J. Jeon, and W. B. Croft. Retrieval models forquestion and answer archives. In Proceedings of SIGIR ’08,
2008.
[20] X. Zhang, Y. Hao, X. Zhu, M. Li, and D. R. Cheriton.
Information distance from a question to an answer. In
Proceedings of KDD ’07, pages 874–883, 2007.