The	  NTCIR-­‐11	  IMine	  Task	  Kickoﬀ	  Mee4ng
Yiqun	  Liu,	  Ruihua	  Song,	  Min	  Zhang,	  Zhicheng	  Dou,	  	  Takehiro	  Yamamoto,	  Makoto	  P.	  Kato,	  Hiroaki	  Ohshima,	  	  Ke	  Zhou	  http://www.thuir.org/IMine/
September,	  2013@NII,	  Tokyo	  Diversiﬁed	  search

SERP	  (Search	  Engine	  Result	  Page)	  • Given	  an	  ambiguous/underspeciﬁed	  query,	  produce	  a	  single	  Search	  Engine	  Result	  Page	  that	  sa4sﬁes	  diﬀerent	  user	  intents!	  
• Challenge:	  balancing	  relevance	  and	  diversity
Highly	  relevant	  near	  the	  top	  Cover	  many	  intents	  Give	  more	  space	  to	  popular	  intents?	  
Give	  more	  space	  to	  informa4onal	  intents?	  IMine
Understanding	  user	  intents	  in	  Web	  search	

Intent	  Mining	曖昧	  （ambiguous） 	

The	  IMine	  task
Three	  subtasks	
• Subtopic	  Mining	  (SM):	  Chinese,	  English,	  Japanese	  INPUT	  	  	  	  :	  query	  (e.g.	  “harry	  poYer”)	  OUTPUT:	  ranked	  list	  of	  subtopic	  string	  (e.g.	  “harry	  poYer	  book,	  harry	  poYer	  ﬁlm,	  harry	  poYer	  the	  character…”)	  • Document	  Ranking	  (DR):	  Chinese,	  English	  INPUT:	  query	  (e.g.	  “harry	  poYer”)	  OUTPUT:	  diversiﬁed	  ranked	  list	  of	  web	  pages	  • Search	  Task	  Mining	  (TASKMINE):	  Japanese	  INPUT	  :	  query	  (e.g.	  “pollen	  allergy	  treatment	  ”)	  OUTPUT:	  ranked	  list	  of	  task	  string	  which	  sa4sﬁes	  the	  given	  query	  （e.g:	  “Laser	  surgery”，”an4allergic	  drug”，”allergy	  mask”）	  	  More	  User	  Behavior	  Data• THUIR@INTENT/INTENT2:	  More	  user	  behavior	  data	  led	  to	  beYer	  performance.	  
• More	  raw	  click-­‐through	  data:	  SogouQ	  has	  doubled	  its	  size	  to	  include	  click-­‐through	  data	  collected	  from	  Sogou.com	  in	  2011	  • 1.85GB	  =>	  3.85GB,	  over	  40M	  user	  clicks	  • More	  subtopic	  candidates	  generated	  from	  	  more	  recent	  user	  behavior	  data.	  
• Search	  engine	  data	  provided	  by	  two	  major	  	  Chinese	  search	  engines	  will	  be	  adopted	  	  • Find	  out	  whether	  more	  logs	  help	  improve	  	  SM/DR	  performance

Intent	  Annota4on	  Using	  Logs
• SM	  results	  in	  pools	  will	  be	  clustered	  with	  click-­‐
through/pseudo	  RF	  data	  at	  ﬁrst	  to	  generate	  preliminary	  candidate	  intent	  groups.	  	  • Query	  frequency	  informa4on	  will	  be	  taken	  	  into	  considera4on	  during	  subtopic	  	  importance	  vo4ng	  process.	  
• Data	  source:	  recently	  collected	  data	  	  from	  Sogou	  for	  Chinese	  SM,	  Bing	  for	  	  English/Japanese	  SM	  • More	  credible	  SM	  qrels	  with	  less	  annota4on	  eﬀorts	  • Perhaps	  the	  reusability	  of	  results	  could	  be	  increased

Crowd-­‐Sourcing	  based	  Evalua4on• A	  search-­‐engine-­‐like	  annota4on	  interface	  to	  collect	  feedback	  informa4on	  from	  a	  rela4vely	  large	  number	  of	  unprofessional	  users	  (e.g.	  50+	  undergraduate	  students)	  • Data	  collected	  from	  the	  interface:	  query,	  click,	  examina4on	  • High	  correla4on	  with	  examina4on	  • Preliminary	  results	  on	  10	  users.	  	  • KAPPA:	  0.65,	  Accuracy:	  0.83.	  
• Find	  out	  whether	  D#	  measures	  accords	  	  with	  user	  sa4sfac4on

INTENT	  with/for	  Knowledge	  Graph• Fuji	  in	  Wikipedia:	  • Fuji	  the	  actress	  • Fuji	  the	  Mountain	  • Fuji	  the	  apple	  • Ambiguous	  queries:	  NTCIR	  INTENT	  #0205:	  功夫(kung	  fu)，#0206:	  ⽣生⽇日快樂樂樂樂(happy	  birthday)	  • Mining	  and	  evalua4ng	  hierarchical	  user	  intents	  • SM	  task:	  return	  a	  two-­‐level	  hierarchical	  list	  of	  subtopics	  • First	  level:	  at	  most	  10	  major	  categories	  of	  user	  intent	  (“Microsoq	  windows”	  for	  “windows”)	  • Second	  level:	  other	  minor	  subtopics	  under	  the	  categories	  (“windows	  update”,	  “windows	  8.1	  installa4on”)	  Task	  Mining	  (TASKMINE)	  subtask
Find	  tasks	  that	  saDsﬁes	  the	  given	  query	
OUTPUT	

query	

move	  to	  a	  new	  house	

Stop	  gas	
Stop	  electricity	
Submit	  document	  of	  move	  out	Submit	  document	  of	  move	  in		  	  Organisers:	  	  Takehiro	  Yamamoto,	  Makoto	  Kato,	  Hiroaki	  Ohshimahttp://www.dl.kuis.kyoto-u.ac.jp/ntcir-11/taskmine/Summary	  of	  IMineThank you
http://www.thuir.org/IMine/
http://www.dl.kuis.kyotou.ac.jp/ntcir-11/taskmine/