Detecting “In-play” Photos in Sports News Photo Database AKio Kiłahara and Keiji Yanai Univ, of Electro-Communications, Tokyo, JAPAN PCM 2009 Dec. 16th 2009
› There are many news photos on the Web › By collecting them regularly, we can build our own news photo DB. We have been crawling Web photo news since 2005. • About half million news photos in our lab file server. L however › Not classified precisely • Just classified in terms of broad genre such as domestic", "international" "economy" and "sports" • So not easy to find out news photos a user want to see
› Focus on  sports photo news  in this work › Classify news photos in terms of sports categories and player’s state (“in-play” or not ) • "In-play" =  a player are playing in the game田内台風14 9月26日名風14 X人が行え経済トル道9月26日9月26エンターテインメント9月26日< appaniいるだろスポーツ大リー 9月26日エンゼルntc- |大リーグ・イチロー、日本記録へ6本9月26日15時49分配信時事通信エンゼルス戦の7回、内野安打を放つイチロー。日米通算3079安打とし、張本勲の日本最高記録まで残り3試合であと6本とした。2得点を加えて今季99得点となり、200安打に続く8年連続100得点にも迫った本とした。2得点差加photo articleルワーズナイン。ナンリンら生活していてい“Baseball” is not shown！動画地域リサーチ雑誌/ブログランキングスポーツアクルロジー 地域ニュース日間社[PR] ≪おすすめカードローン≫⇒来店不要•24時間受付！ title iTunes
▷ Quattoni et al. (CVPR 2007) News classification of Web photo news using visual features and textual features ▷ Jain eł al. (CVPR 2008) • Sports classification of sports photos taken by normal people (not for news photo taken by professional phołographers) No work to classify photos regarding sports categories and "in-play" simultaneously so far
› Propose two methods to estimate sports genre and player’ s state for Web news photos • Use supervised learning • Use textual features and visual features › [1] two-step method • [Ist step] sport classification with textual info. 。[2nd step] "in-play" classification with visual info, › [2] one-step method • Simultaneous sport-and- "in-play"  classification with both textual and visual information
Web sports photo news ] st Step Sports classifi– Cation 2nd step “in-plav classifi– cation MKL…Multiple Kernel Learning
• Simultaneous sport-and- "in-play"  classification using  both textual and visual information Web sports photo news Textual and visual features bag-of-words bag-of-features Gabor Region-based bag-of-features Color histogram Sports and “in-play” classification by feature fusion
› Textual features (from news articles) Bag-of-words · Word histogram with top N frequent words › Visual features (from news photos) • (normal) Bag-of-featuress • Region-based bag-of-features • Gabor features • Color histogram
› Represent an image as a set of local features Sample 2000 points randomly 1. Describe local patterns around sampled points 2. with the SIFT descriptor [Lowe 2004] Generate codebooks by K-means   (k=1000) 3. 4. Convert images into BoF vectors by voting to the nearest codewords
1. 2. Region segmentation (JSEG [Deng 2001]) Select the most informative region (mi-SVM [Andrew 03]) ·  Estimate most informative regions using multiple instance learning (mi-SVM) · "in-play" photos are positive, and "not-in-play" are negative samples. ·  The region with the largest SVM value is regarded as "most informative"  region Convert the selected region into a bag-og-features (BoF) vector. Not foreground detection but informative (background) detection 3. Feature fusion by Multiple Kernel Learning (MKL) [Sonnenburg et al. 2006] › Is an extenstion of a SVM . › Can handle “a combined Kernel” which is a linear combination of Kernels. › Can estimate Kernel weights and SVM model parameters simultaneously. ▷ Can integrate features by assigning one feature to one Kernel,
▷ We built an Web photo news dataset. › Dała source: Yahoo! Japan Phoło News › 6-sports: baseball, golf, car race, soccer, tennis, sumo › We gave ground truth label (sports, in-play) by hand.
▷ Evaluation:   classification rate= # correctly classified photos # all the classified photos (Equivalent to the average of diagonal of confusion matrix)
› [Ist step] sports category classification • Classifier:  standard SVM with chi-square Kernel, • Feature:  textual features (2800-dim bag-of-words) › [Znd step] “in-play” classification • Classifier:  Multi Kernel Learning (MKL) with chi-square ( x ² ) Kernels, k(x,x')=exp(−γ ∑ ∥ x − x'∥ z ) • Features:  BoF, region-based BoF, Gabor, Color In all the experiments, we optimized kernel param. (γ ) and cost param. (C) by grid search.
76.50% 80.75% “Car race” photos classified as "in-play" Correctly classified Failed “Car race” photos classified as "not-in-play" Success Failue “Golf” photos classified as "in-play" Success Failure “Golf” photos classified as "not-in-play" Success Failue
0% 20% In case of “car race”, “tennis” and “sumo”, two kinds of weights were obtained, because we have the two best paremeter settings in the CV of training data. .
› 12-class classification  by  MKL : b sports × {in-play, not-in-play} Web sports photo news Textual and visual features bag-of-words bag-of-features Gabor Region-based bag-of-features Color histogram Sports and “in-play” classification by feature fusion 12 classes Results of the one-step method
› Incorrectly classified into “not-in-play”
› Propose łwo mełhods ło classify Web sporłs news photos into "in-play" or "not-in-play" ▷ Results by the two-step method • Sports classification:      99.33 % • “in-play” classification: 80.75% ▷ Results by the one-step method • Sports classification :     100.0 % • “in-play” classification : 77.08 %
▷ Add new sports categories. • Use Amazon Mechanical Turk ? • Transfer learning might be possible, (e.g. baseball vs softball) › Add new player’s states such as “interview” and close-up faces ▷ Add new visual features ▷ Recognize "pose" of players › Introduce sport-category-dependent methods. • e.g.  "golf club/bat/tennis racket detection" Thank you for your attention,