Package: clickmodels
ClickModelsClickModels is a small set of Python scripts for the user click models
initially developed atYandex. AClick
Modelis a probabilistic graphical model used to predict search engine
click data from past observations. This project is aimed to deal with
click models used in Information Retrieval (see next section) and
intended to be easy-to-read and easy-to-modify. If it’s not, please let
me know how to improve it :)If you are using this code for your research work, consider citing one
of our papers when appropriate (seeReferencessection below).If you are looking for a general-purpose framework to work with
probabilistic graphical models you might want to examineInfer.NET.
It should also work with IronPython.Quick Startcp clickmodels/config_sample.py config.pyvim config.pypython bin/run_inference.py < data/click_log_sample.tsv 2>inference.logMore details about the config and input data formats below.System-Wide InstallIf you wish, you can install the ClickModels core (parameter inference
and click simulation) to a system-wide location:sudo python setup.py installUninstall:sudo pip uninstall clickmodelsNew!Now, thanks toagrotov, the models can
also be run in a click generation mode and predict relevance (DBN only).
Check outClickModel.get_model_relevances()andClickModels.generate_clicks()methods.N.B.: Use this code with care as it is not fully tested yet.Models ImplementedDynamic Bayesian Network(DBN) model: Chapelle, O. and Zhang,
Y. 2009. A dynamic bayesian network click model for web search
ranking. WWW (2009).User Browsing Model(UBM): Dupret, G. and Piwowarski, B.
2008. A user browsing model to predict search engine click data from
past observations. SIGIR (2008).Exploration Bias User Browsing Model(EB_UBM): Chen, D. et
al. 2012. Beyond ten blue links: enabling user click modeling in
federated web search. WSDM (2012).Dependent Click Model(DCM): Guo, F. et al. 2009. Efficient
multiple-click models in web search. WSDM (2009).Intent-Aware Models(DBN-IA, UBM-IA, EB_UBM-IA, DCM-IA):Chuklin, A. et al. 2013. Using Intent Information to Model User
Behavior in Diversified Search. ECIR
(2013).Format of the Input Data (Click Log)A small example can be found underdata/click_log_sample.tsv. This
is a tab-separated file, where each line has 7 elements. For example,
the line1dd100500  QUERY1  50  0.259109["http://1","http://2","http://3","http://4","http://5","http://6","http://7","http://8","http://9","Http://10"][false, false, false, false, true, true, false, false, false, false]    [0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0]has the following fields:1dd100500— some identifier (currently not used)QUERY1— text of the query. It can contain any UTF-8 characters
except tab sign\t50— integer identifier of the region (country, city) of the user
who submitted the query.If you don’t want this, just put some
constant (e.g., ``0``) in this column. At Yandex user region is
heavily used by ranking, so throughout the code the pair(query, region)is used to identify the query, i.e., if we have
the same query string from two different region we consider them as
two separate queries.0.259109— float value, corresponding to the probabilityP(I = V)that user has averticalintent V, i.e., he or she is
interested more in the vertical documents than in organic web
documents.If you do not want any of this intent stuff just put
``0`` in this column. We assume that the user has one of the two
intents:verticalintent V with probabilityP(I = V)and
regularwebintent with probability1 - P(I = V). For example,
if we want to take into account user’s interest in images and we
somehow know the probability that the user is interested more in
images than organicwebresults, we can make use of intent-aware
click models. SeeChuklin, A. et al. 2013. Using Intent Information
to Model User Behavior in Diversified Search. ECIR
(2013)for more details.jsonlist of the URLs of the documents that make up SERP (search
engine result page). Document’s url is an identifier, so in principle
you can use any (string) id you want.NB: this is not a python
list, so yuo have to use double quotes and no comma after the last
element.jsonlist with thepresentation types(vertical types) of the
documents (seeChuklin, A. et al. 2013. Using Intent Information to
Model User Behavior in Diversified Search. ECIR
(2013).).If you do not want to know thisjust set it to the list offalseof the same length as the previous list.jsonlist of clicks. Each element is the number of times
corresponding URL was clickedIf you need more data to experiment with you can use any publicly
available dataset and convert it to the format described above. For
example, you can use a dataset provided by one of the Yandex challenges
(you need to register to get access to the data): -http://imat-relpred.yandex.ru/en/datasets-http://switchdetect.yandex.ru/en/datasetsFilesREADME.mdThis file.LICENSE, AUTHORS, CHANES.txtSelf explaining.setup.pyPython package installation file:python setup.py--helpbin/Directory with the scripts.clickmodels/Directory with the core code. This is the directory that gets installed
bysetup.py.data/data/directory contains an example of click log (see format
description above) as well as two examples of result pages with fresh
block included (seemakeGluedSERP.pydescription above):data/serp_sample.jsonis used in an example above, whiledata/serp_sample2.jsonwas used to create a picture in the paperChuklin, A. et al. 2013. Using Intent Information to Model User
Behavior in Diversified Search. ECIR
(2013).doc/Automaticaly generated documentation. Also availableonline.html/Directory with the CSS/JS files forbin/generate_serp.py; it outputs
there a number of HTML files for you to examine to get an idea what kind
of SERPs we address with the intent-aware click models.bin/generate_serp.py{not used by the other, does not use other code}Create html of the
SERP containing fresh block item. This is used just to illustrate the
notion ofpresentation typesused by Intent-Aware models. Run as:bin/generate_serp.py < data/serp_sample.json. Output is placed inhtml/directory.WARNING:all previously generated html files in this directory will
be removedbin/compare_click_models.py{not used by other scripts}Script used to compare different models
and output significance of the difference. The pair of models to compare
is specified in the code by modifyingTESTED_MODEL_PAIRSvariable.
Model pair is a text string which is mapped to the pair of functions
returning model objects (seeMODEL_CONSTRUCTORSdict for the
mapping). E.g.UBMvsDBNis used to compare UBM model
(UbmModel()) to the default DBN model
(DbnModel((0.9,0.9, 0.9, 0.9))).NB: we may have a list of
models needed to be compared to each other. For this purpose the same
notion ofpairis abused. For instance,MODEL_CONSTRUCTORS['EB_UBM']contains 3 algorithms to be compared to
each other:UBM, EB_UBM, EB_UBM-IA.Usage:bin/compare_click_models.py directory_with_click_logs 2>run.logInput:directory_with_click_logs— directory containing files
with click logs. Each file is in the format described above. These
files are then sorted alphabetically and split into pairs where first
file is used for training, the second one is used for testing. For
example, if the directory contains filesf01,f02,f03,f04thenf02will be used to test models trained usingf01,f04will be used to test models trained usingf03and so on. Two models are evaluated on the test set and their
performances (Average Perplexity or Log Likelihood) are compared
using appropriate formula (seeperpGainandllGainfunctions
respectively).NB: Multiple train and test files are needed to
calculate confidence interval for the gains (bootstrap.pyis used
for this purpose).Output: some progress output is printend tosys.stderrwhich
might be useful for a long run. Finally the gains of one model over
another is output in the following format:UBM (0, 1) [-0.0115, 0.0659, 0.075, 0.0778, 0.0623, 0.0403, 0.0593, -0.040] (0.0095, 0.0662)It first outputs the name of the “pair”pairNamespecified in theTESTED_MODEL_PAIRS, then pair of indeces(i, j)which mean
that the model compared areMODEL_CONSTRUCTORS[pairName][i]andMODEL_CONSTRUCTORS[pairName][j]which is UBM and UBM-IA in our
example. Next is the list of gains of modeljover modelifor each pair of the train/test logs (in our example we had 8 pairs
of files underdirectory_with_click_logs). The next element is
the confidence interval according to bootstrap test (with 95%
confidence level and 1000 bootstrap samples). This line will be
printed for all the “pairs” listed underTESTED_MODEL_PAIRSand
for both Average Perplexity (PERPLEXITY) and Log Likelihood (LL). For
perplexity measure also the gains for individual position (rank) are
printed. Like this:UBM POSITION PERPLEXITY GAINS: (0, 1) [[average_gain_for_pos1, confidence_interval_for_pos_1], …]bin/run_inference.pyRun click model inference and evaluate the models:Usage:bin/run_inference.py < data/click_log_sample.tsv 2>inference.logInput: click log in the format described above (sys.stdin)Output(assuming thatTRAIN_FOR_METRIC = False):ModelName (LogLikelihood, Perplexity)clickmodels/inference.pyThis file contains implementation of all the click models, probabilistic
inference and helper functions needed to work with them. This is the
core of the codebase. More details about the classes/functions below.clickmodels/bootstrap.pyCopyright ©Ernesto P.
Adorio: code we use
to perform a bootstrap test.clickmodels/config_sample.pyCopy this file toconfig.pyand modify it; it will be used bybin/run_inference.pyandbin/compare_click_models.py. This is
the file with default settings for different parameters of the
inference, input and output data.MAX_ITERATIONS— maximum number of iterations in Expectation
Maximization (EM) algorithm (applicable only for models using EM
algorithm).DEBUG— perform some additional tests when running algorithm
(makes it slower)PRETTY_LOG— make log output prettier. IfFalsethen more
information is put into log.USED_MODELS— list of model names to be tested in__main__section of the script. Possible names are['Baseline', 'SDBN', 'UBM','UBM-IA','EB_UBM','EB_UBM-IA','DCM','DCM-IA','DBN','DBN-IA'].
Please refer to the__main__section ofinference.pyto see
how these names are expressed in terms of our class hierarchy (all
those nastyif 'XXX' in USED_MODELS).MIN_DOCS_PER_QUERY,MAX_DOCS_PER_QUERY– number of documents
per query. Set to 10 by default as most of search engines return list
of 10 doucments.SERP_SIZE- size of the search engine result page (SERP). Used if
we want to model clicks beyond the first result page. See the section
namedBeyond the First Result Pagebelow for more details.EXTENDED_LOG_FORMAT- if set toTruethe urls, layout and
clicks are dicts instead of lists (seeFormat of the Click Logsection above). Example:data/click_log_sample_extended_format.tsv.TRANSFORM_LOG- transform the click log by inserting the fake
documents for pagination button (currently works only withEXTENDED_LOG_FORMAT = True). See the section namedBeyond the
First Result Pagebelow.QUERY_INDEPENDENT_PAGER- used to switch betweenSDBN(P)/SDBN(P-Q). Only used withTRANSFORM_LOG = True.TRAIN_FOR_METRIC– ifTruethe model will be trained such
that its parameters can be used in a metric (likeChuklin, A. et al.
2013. Click Model-Based Information Retrieval Metrics. SIGIR
(2013).).
See the section below for more details.PRINT_EBU_STATS— ifTruethe parameters of the EBU metric
will be printed first (Yilmaz, E. et al. 2010. Expected browsing
utility for web search evaluation. CIKM. (2010)).DEFAULT_REL- default (prior) relevance (attractiveness,
satisfaction) parameter values used in click models like DBN or EBU.clickmodels/input_reader.pyA class used for reading input data in a click log format described
above.Class HierarchyAlso see epydoc-generateddocumentation.Click ModelsThe base class for all the click models is the class calledClickModel. In order to define a new click model you should create a
clas inherited from it and re-define methodstrainand_getClickProbs.trainfunctionNote, thattestmethod is already implemented and uses_getClickProbsfunction. If you redefine__init__method, then
be sure to invoke the__init__of the parent class to set theignoreIntentsandignoreLayoutparameters (they should be set toTrueunless you are usingIntent Awaremodel)ClickModelclass by itself represents a baseline click model which
sets probability 0.5 to any click event.DbnModelThis class is, in fact an implementation of a more generalDBN-IAmodel (Chuklin, A. et al. 2013. Using Intent Information to Model User
Behavior in Diversified Search. ECIR
(2013).) that makes use of intent and presentation type of the documents whenignoreIntentand/orignoreLayoutis set toFalse. Thetrainmethod is a probabilistic EM inference.If all what you want is just original DBN model by Chapelle et al. you
should creat it asDbnModel((0.9,0.9, 0.9, 0.9))(ignoreIntentandignoreLayoutisTrueby default).SimplifiedDbnModel (DbnModel)This is the same asDbnModel((1.0,1.0, 1.0, 1.0), ignoreIntents, ignoreLayout), buttrainmethod is just counting instead of EM algorithm. SeeChapelle, O. and Zhang, Y. 2009. A Dynamic Bayesian Network click model
for web search ranking. WWW (2009)., Section 5 (Algorithm 1).UbmModelThis is the most general case for all UBM-like intent-aware models.
ChangingignoreIntents,ignoreLayoutandexplorationBiasparameters you can get different models:UBM, UBM-intent, UBM-layout,
UBM-IA, EB_UBM, EB_UBM-intent, EB_UBM-layout, EB_UBM-IA(for the
names seeChuklin, A. et al. 2013. Using Intent Information to Model
User Behavior in Diversified Search. ECIR
(2013).).EbUbmModel (UbmModel)Just a shortcut forUbmModel(ignoreIntents, ignoreLayout, explorationBias=True)which
correspond to the model calledExploration Bias UBMinChen, D. et
al. 2012. Beyond ten blue links: enabling user click modeling in
federated web search. WSDM (2012).DcmModelThis model is, again, more generalDCM-IAmodel which reduces toDCMwhenignoreIntents = True,ignoreLayout = True.trainmethod is a simple counting, no EM algorithm.Please note, thatgetGammamethod invokesDbnModel.getGamma, so
be careful when changing that.InputReaderThis class intented to read input (click log) in the format described
above. To save memory, it maps queries and urls to ids. It means, that
you need to use the same instance of theInputReaderclass even if
you read multiple click log files. Otherwise you will end up with two
different ids assigned to the same query.Performance IssuesIf you experience performance issues consider usingPyPyinstead of regular cPython. It may lead to
10x spead up. You can also install and usesimplejsonmodule instead
ofjson.TRAIN_FOR_METRICIf you setTRAIN_FOR_METRIC = Truethe code will expect you to
provide document relevances instead of URLs. We make an assumption, that
document attractiveness and/or satisfaction probability only depends on
its human-assigned relevance grade. A model will then be trained to
assign the same attractiveness / satisfaction probabilities to all the
documents with the same relevance.Input FormatThe format in this case is similar to the one descirbed above with only
difference that URLs should be replaced by the relevance grade of the
corresponding document to the corresponding query. The query field will
be ignored in that case. The relevance grade should take one of the
following values:IRRELEVANT— lowest relevance scorn, document is not relevant to
the queryRELEVANT— combines marginally relevant and just relevant
documentsUSEFUL— document is more than just relevant, it is really usefulVITAL— highest relevance score, the document is essentialPlease note, that if you also havePRINT_EBU_STATSset toTrue,
then the parameters of the EBU / rrDBN metric will be printed out first
(these ones can be computed directly without need to train a model).OutputFor each model corresponding parametres will be printed out:UBM— attractiveness probabilitiesalphaand position
discount parametersgammaDCM— attractiveness probabilitiesalpha(named asurlRelevancesin the code) and position discount parametersgammaFor more conceptual details about converting click models into
evaluation metrics please refer to the paperChuklin, A. et al. 2013.
Click Model-Based Information Retrieval Metrics. SIGIR
(2013).Beyond the First Result PageIf you want to model the clicks beyond the first result page you may
want to model pagination button separately. We implemented the models
described in the paperA. Chuklin, P. Serdyukov, and M. de Rijke.
Modeling Clicks Beyond the First Result Page. In CIKM. ACM,
2013..
Namely, by setting the following config options you will get:TRANSFORM_LOG = True,QUERY_INDEPENDENT_PAGER = False:SDBN(P)modelTRANSFORM_LOG = True,QUERY_INDEPENDENT_PAGER = True:SDBN(P-Q)modelTRANSFORM_LOG = False: reqularSDBNmodelPlease, refer to the paper for more details. ***ReferencesA. Chuklin, P. Serdyukov, and M. de Rijke. Using Intent Information
to Model User Behavior in Diversified Search. In ECIR, 2013.[pdf]A. Chuklin, P. Serdyukov, and M. de Rijke. Click model-based
information retrieval metrics. In SIGIR. ACM, 2013.[pdf]A. Chuklin, P. Serdyukov, and M. de Rijke. Modeling Clicks Beyond
the First Result Page. In CIKM. ACM, 2013.[pdf]Copyright and LicenseCopyright ©Yandex2012-2013,varepsilon2012-∞Published under the BSD license.