EXHIBIT B

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 1 of 34
(12) United States Patent 

Gross 

USOO9192861 B2 

(10) Patent No.: US 9,192,861 B2 

(45) Date of Patent: Nov. 24, 2015 

(54) 

(71) 

(72) 

(73) 

(*) 

(21) 

(22) 

(65) 

(62) 

(60) 

(51) 

(52) 

MOTION, ORIENTATION, AND 

TOUCH-BASED CAPTCHAS 

Applicant: The John Nicholas and Kristin Gross 

Trust, Berkeley, CA (US) 

Inventor: John Nicholas Gross, Berkeley, CA 

(US) 

Assignee: John Nicholas and Kristin Gross 

TrustCA (US), Trust U/A/D Apr. 13, 

2010 

Notice: Subject to any disclaimer, the term of this 

patent is extended or adjusted under 35 

U.S.C. 154(b) by 0 days. 

Appl. No.: 14/223,848 

Filed: Mar 24, 2014 

Prior Publication Data 

US 2014/O295927 A1 Oct. 2, 2014 

Related U.S. Application Data 

Division of application No. 12/494,083, filed on Jun. 

29, 2009, now Pat. No. 8,752,141. 

Provisional application No. 61/076,495, filed on Jun. 

27, 2008. 

Int. C. 

G06F I7/00 (2006.01) 

A63F I3/30 (2014.01) 

A63F 9/06 (2006.01) 

A63 F.3/04 (2006.01) 

GO6F 21/31 (2013.01) 

U.S. C. 

CPC ................. A63F 13/12 (2013.01); A63F 13/30 

lmage Data 

accordance 

components 

suggestions 

(2013.01); A63F 3/0423 (2013.01); A63F 

2009/0661 (2013.01); A63F2300/558 

(2013.01); A63F2300/572 (2013.01); A63F 

Gamts) 

image is 332 

Preset 

Word Topic 

2300/638 (2013.01); A63F2300/8094 

(2013.01); G06F 21/31 (2013.01); G06F 

222 1/2133 (2013.01) 

(58) Field of Classification Search 

USPC .................. 463/9-13.16, 20; 726/3–5, 17, 21 

See application file for complete search history. 

(56) References Cited 

U.S. PATENT DOCUMENTS 

4,441.201 A 4, 1984 Henderson et al. 

5,465,982 A 11/1995 Rebane 

5,774,837 A 6/1998 Yeldener et al. 

5,897,616 A 4/1999 Kanevsky et al. 

5,933,803 A 8/1999 Ojala 

(Continued) 

FOREIGN PATENT DOCUMENTS 

EP 2045754 4/2009 

WO 2O08025O19 2, 2008 

OTHER PUBLICATIONS 

Aaron, Andy, et al., "Conversational Computers.” Scientific Ameri 

can, Jun. 2005, 6 pages. Aaron, Andy, et al., "Conversational Com 

puters.” Scientific American, Jun. 2005, 6 pages. 

(Continued) 

Primary Examiner — Ronald Laneau 

(74) Attorney, Agent, or Firm — PatentBest; Andrew 

McAleavey 

(57) ABSTRACT 

A CAPTCHA system uses images/pictures and/or motion for 

granting access to a computing system. The images can be 

culled from examples used in pictorial games, and can pro gressively presented to increase the strength of the 

CAPTCHA challenges. Speech recognition, motion and 

touch sensing can also be employed as parts of the challenge. 

20 Claims, 16 Drawing Sheets 

Advertises 

840 Auction (dual) 

Manual mode 

with Emage Supplemental 

Option and tinning Selected? 

Accelerate for manual mode 

388 data (of downloads 

image 

382 

Yes 

in thanual 

mode) Retrieve secondary image database information for image D . . . . . . . . . . . . . 384 

Present secondary image data to participants (or hints to artist in mania mode) 385 

SYNHSSS SUGGESTIONMODULE 

36 

Suggestion Option Selected 

366 

Normal picture 

node 367 

33) 

Search engine Result Web Page 

epict 

Stock Ad Stock 

Ad/Epict Combination 

/ 

 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 2 of 34
US 9,192,861 B2 

Page 2 

(56) 

5,940,799 

5.991,617 

6,014,441 

6,157,912 

6, 191,777 

6, 195,698 

6,208,970 

6,397, 181 

6,546,369 

6,681,205 

6,697,769 

6,699,127 

6,763,373 

6,851,051 

6,878,062 

6,879,956 

7,222,072 

7,505,946 

7,516,220 

7,603,706 

7,679,604 

7,707,251 

7,725,395 

7,785,180 

7,797,191 

7,827,265 

7,841,940 

7,887,058 

7,909,695 

7,917,508 

7,929,805 

7,945,952 

8,005, 197 

8,019, 127 

8,032,481 

8,032,483 

8,036,902 

8, 166,388 

8,380,503 

8.403,761 

8,601,538 

8,752,141 

2002fOO 13708 

2002fOO161.96 

2002/0090993 

2003,0004719 

2003/0022717 

2003/OO33145 

2003/009328O 

2003/O125944 

2003. O163316 

2003/01821.19 

2003. O194211 

2003/0214553 

2003/0217336 

2004, OO19485 

2004/01O2975 

2004/O125877 

2004/O158495 

2004/02496.28 

2004/O254793 

2005, 0071.163 

2005/0105712 

2005/O114137 

2005/O114705 

2005/O1592O6 

2005/O15995.0 

2005/0171851 

2005. O184461 

2005.0185847 

2005/0228641 

2005/0267752 

2006, OO74677 

2006/0248019 

2006/0277045 

References Cited 

U.S. PATENT DOCUMENTS 

A 

A 

A 

A 

B1 

B1 

B1 

B1 

B1 

B1 

B1 

B1 

B2 

B1 

B2 

B1 

B2 

B2 

B1 

B2 

B2 

B2 

B2 

B1 

B2 

B2 

B2 

B2 

B2 

B1 

B2 

B1 

B2 

B2 

B2 

B1 

B1 

B2 

B2 

B2 

B2 

B2 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

A1 

8, 1999 

11, 1999 

1, 2000 

12, 2000 

2, 2001 

2, 2001 

3, 2001 

5, 2002 

4, 2003 

1, 2004 

2, 2004 

3, 2004 

T/2004 

2, 2005 

4, 2005 

4, 2005 

5/2007 

3, 2009 

4, 2009 

10, 2009 

3, 2010 

4, 2010 

5, 2010 

8, 2010 

9, 2010 

11, 2010 

11, 2010 

2, 2011 

3, 2011 

3, 2011 

4, 2011 

5, 2011 

8, 2011 

9, 2011 

10, 2011 

10, 2011 

10, 2011 

4, 2012 

2, 2013 

3/2013 

12, 2013 

6, 2014 

1, 2002 

2, 2002 

T/2002 

1, 2003 

1, 2003 

2, 2003 

5/2003 

T/2003 

8, 2003 

9, 2003 

10, 2003 

11, 2003 

11, 2003 

1, 2004 

5, 2004 

T/2004 

8, 2004 

12, 2004 

12, 2004 

3, 2005 

5/2005 

5/2005 

5/2005 

7/2005 

7/2005 

8, 2005 

8, 2005 

8, 2005 

10, 2005 

12, 2005 

4, 2006 

11, 2006 

12, 2006 

Bruckert et al. 

Powell 

Mark 

Kneser et al. 

Yasuhara et al. 

Lillibridge et al. 

Ramanan 

Li et al. 

Buth et al. 

San Martin et al. 

Goodman et al. 

Lobb et al. 

Shigi 

Bolle et al. 

Bjorklund et al. 

Honda et al. 

Chang 

Chellapilla et al. 

Stiert 

Donnelly et al. 

Uhlik et al. 

Birch 

Rui et al. 

Von Ahn et al. 

Cotten et al. 

Cheever et al. 

Bronstein 

Ritter et al. 

Phelps, Jr. et al. 

Baluja et al. 

Wang et al. 

Behforooz 

Baird et al. 

Misra 

Pinckney et al. 

Haveliwala et al. 

Strom et al. 

Gounares et al. 

Gross 

Kira et al. 

Qvarfordt et al. 

Gross 

Walker et al. 

Orak 

Koshiro et al. 

Yuschik 

Bjorklund et al. 

Petrushin 

Oudeyer 

Wohlsen et al. 

Addison et al. 

Junqua et al. 

Abecassis 

Dodge 

Gounares et al. 

Kobayashi et al. 

Eide 

Chang et al. 

Gennaro et al. 

Chelba et al. 

Herley et al. 

Aaron et al. 

Williams et al. 

Saito et al. 

Reshefet al. 

Bjorklund et al. 

Roth et al. 

Applebaum et al. 

Cogliano et al. 

Rowe 

Chelba et al. 

Navratil et al. 

DeSimone 

Rajakumar 

Gleason 

2007, OO16689 A1 1, 2007 Birch 

2007, OO18393 A1 1/2007 Ritter et al. 

2007.0043681 A1 2/2007 Morgan et al. 

2007/0071206 A1 3/2007 Gainsboro et al. 

2007/OO73630 A1 3/2007 Greene et al. 

2007/O112570 A1 5/2007 Kaneyasu 

2007/0113294 A1 5/2007 Field et al. 

2007/0162761 A1 7/2007 Davis et al. 

2007. O165821 A1 7/2007 Altberg et al. 

2007/0213111 A1 9, 2007 Maclver et al. 

2007/027183.0 A1 11/2007 Holt et al. 

2007/0280436 Al 12/2007 Rajakumar 2007/0282605 A1 12/2007 Rajakumar 2008.OO72293 A1 3, 2008 DUrSO 

2008/O1273O2 A1 5/2008 Qvarfordt et al. 

2008. O140420 A1 6/2008 Lee 

2008/O175174 A1 7/2008 Altberg et al. 

2008/0216163 A1 9, 2008 Pratte et al. 

2009/0012855 A1 1/2009 Jamal et al. 

2009 OO13391 A1 1, 2009 Ernst 

2009 OO42648 A1 2/2009 Phelps, Jr. et al. 

2009/005.5.193 A1 2/2009 Maislos et al. 

2009, OO77629 A1 3/2009 Douceur et al. 

2009.0113294 A1 4/2009 Sanghaviet al. 

2009. O150983 A1 6/2009 Saxena et al. 

2009/0204819 A1 8, 2009 Parker 

2009,0241174 A1 9/2009 Rajan et al. 

2009,0249476 A1 10, 2009 Seacat 

2009,0259468 A1 10, 2009 Schroeter 

2009/0280895 A1 11/2009 Yamaoka et al. 

2009,0297.064 A1 12/2009 KOZiolet al. 

2009/0309698 A1 12/2009 Headley et al. 

20090313694 A1 12/2009 Mates 

2009031927O A1 12/2009 Gross 

20090319271 A1 12/2009 Gross 

20090319274 A1 12/2009 Gross 

2009/0325661 A1 12/2009 Gross 

2009/0325696 A1 12/2009 Gross 

2009/0328150 A1 12/2009 Gross 

2010.0049526 A1 2/2010 Lewis et al. 

2010.0077210 A1 3/2010 Broder et al. 

2011/0085700 A1* 4/2011 Lee ............................... 382,103 

2011/0098117 A1 4/2011 Tanaka 

2011/02O1396 A1 8, 2011 Ritter et al. 

2012/0100910 A1 4/2012 Eichorn et al. 

2012/0179742 A1* 7/2012 Acharya et al. ............... TO9,202 

2013/0166726 A1* 6/2013 Boldyrev et al. ............. TO9,224 

2014.0020084 A1 1/2014 Gross 

2014/0289828 A1 9/2014 Gross 

2014/0295927 A1 10, 2014 Gross 

OTHER PUBLICATIONS 

Anonymous; "Speaker Verification Testing Final Report Draft 1.0.” 

International Biometric Group, 2004, 31 pages. 

Arbel, Eran, et al., “An automated method for distinguishing between 

humans and computers using voice.” U.S. Appl. No. 60/891,042, 

filed Feb. 22, 2007, 4 pages. 

Arrenius, Lisa; "Speech Synthesis on the World WideWeb: Evalua 

tion of the Uppsala University Internet Portal.” Master's thesis in 

Computational Linguistics, Jun. 4, 2004.61 pages. 

Baird, Henry, et al., “Human interactive proofs and document image analysis,” in Proc., IAPR 2002 Workshop on Document Analysis 

Systems, (Princeton, NJ), Aug. 2002, 12 pages. 

Beutnagel, M., et al.; "The AT&T Next-Gen TIS System.” http:// www.research.att.com/projects/tts, 1999.4 pages. 

Bigham, J.P. et al.; “Evaluating existing audio CAPTCHAs and an 

interface optimized for non-visual use.” in proceedings of ACM CHI 

2009 Conference on Human Factors in Computing Systems, Apr. 

2009, pp. 1829-1838. 

Bohr, et al.; “Improving auditory CAPTCHA security.” ISR Tech. 

rep., University of Maryland, College Park, MD, 2008. 

Chan, T.: “Using a test-to-speech synthesizer to generate a reverse 

Turing test.” Tools with Artificial Intelligence, 2003. Proceedings. 

15th IEEE International Conference on, vol., no. pp. 226-232, Nov. 

3-5, 2003. 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 3 of 34
US 9,192,861 B2 

Page 3 

(56) References Cited 

OTHER PUBLICATIONS 

Golle, P; "Machine learning attacks against the Asirra CAPTCHA.” 

in proceedings of the 15th ACM Conference on Computer and Com 

munications Security, Alexandria, Virginia, USA, Feb. 28, 2008, 

535-542. 

Gossweiler, Rich, et al.: “What's Up CAPTCHA2 a CAPTCHA 

Based on Image Orientation.” WWW 2009, Apr. 20-24, 2009, 

Madrid, Spain, 10 pages. 

Gossweller, Rich, et al.: “What's Up CAPTCHAP A CAPTCHA 

Based on Image Orientation.” WWW 2009, Apr. 20-24, 2009, 

Madrid, Spain, 10 pages. 

“Guess-a-Sketch.” web.archive.org/web/. ./GAS.html, Nov. 24. 

2002, 5 pages. 

“GWAP” http://bloggwap.com/2008/05/hellow-world.html, May 

13, 2008, 3 pages. 

International Search Report and Written Opinion for PCTIUS2009/ 

047413, mailed Oct. 15, 2009, 18 pages. 

Kochanski, G., Lopresti. D., and Shih, C.; "A Reverse Turing Test 

Using Speech.” Seventh International Conference on Spoken Lan 

guage Processing, Sep. 2002. 

Lopresti, Daniel: “Leveraging the Captcha Problem.” Presented at 

the Second International Workshop on Human Interactive Proofs, 

Bethlehem, PA, May 2005, 15 pages. 

Markkola, Anu, et al.; "Accessible Voice CAPTCHAs for Internet 

Telephony.” Symposium on Accessible Privacy and Security 

(SOAPS) 2008, Jul. 23, 2008, Pittsburgh, PA USA, 2 pages. Mori, Greg, et al., “Recognizing Objects in Adversarial Clutter: 

Breaking a Visual CAPTCHA.” University of California Berkeley 

Computer Science Department, 2003, 8 pages. 

Nass, Clifford, et al., “Does Computer-Synthesized Speech Manifest 

Personality? Experimental Tests of Recognition, Similarity-Attrac 

tion, and Consistency-Attraction.” Journal of Experimental Psychol 

ogy: Accepted Apr. 3, 2001, vol. 7, No. 3, 171-181. 

Pope, Clark, et al.; Abstract of “Is It Human or Computer? Defending 

E-Commerce with Captchas.” IT Professional, vol. 7, No. 2, Mar. 

1Apr. 2005, 1 page. Reynolds, Douglas, et al., “Automatic Speaker Recognition Recent 

Progress, Current Applications, and Future Trends.” Presented at the 

AAAS 2000 Meeting Humans, Computers and Speech Symposium, 

Feb. 19, 2000.36 pages. Reynolds, Douglas; “Automatic Speaker Recognition: Current 

Approaches and Future Trends.” This paper is based in part on the 

tutorial "Speaker Verification: From Research to Reality” by DA 

Reynolds and L.P. Heck, ICASSP 2001, 6 pages. 

Russ, John C.; "The Image Processing and Analysis Cookbook.” 

Copyright 1996-2001, p. 1,37.38,53,54.241,242. Schlaikjer; "A Dual-Use Speech Captcha: Aiding Visually Impaired 

Web Users while Providing Transcriptions of Audio Streams.” Tech 

nical Report CMU-LTI-07-014, Carnegie Mellon University, Nov. 

2007. 

Schuckers, Stephanie; "Spoofing and Anti-Spoofing Measures.” 

Information Security Technical Report, vol. 7, No. 4. Dec. 10, 2002, 

pp. 56-62. 

Stone, Jonathon; "A Guide to Speaker Verification.” Produced by ICR 

Speech Solutions & Services, Oct. 28, 2003, 11 pages. Thuras, Dylan; "The Turing Test and Machine Intelligence.” http:// htmltimes.com/turing-test-machine-intelligence.php, Nov. 4, 2008, 

3 pages. 

Von Ahn, Luis, et al., “Peekaboom: a game for locating objects in 

images.” Proceedings of the SIGCHI conference on Human Factors 

in computing systems, Apr. 22-27, 2006, Montreal, Quebec, Canada, 

23 pages. 

Von Ahn, Luis, et al.: “Labeling Images with a Computer Game.” CHI 

2004, Apr. 24-29, 2004, Vienna, Austria, 8 pages. 

* cited by examiner 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 4 of 34
U.S. Patent Nov. 24, 2015 Sheet 1 of 16 US 9,192,861 B2 

Wired or 

Wireless 

Link 

125 

/ Electronic 

Tablet/Pad Display/Control Engine 

Stylus 

115 

Broadcast Receiver 

147 

Music Matching 

Logic 150 

Remote Server 

145 

es 

c will 

. O 

. 

C 

G 

C 

s 

Data 

Feed 

 

 

 

 

 

 

 

 

 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 5 of 34
U.S. Patent Nov. 24, 2015 Sheet 2 of 16 US 9,192,861 B2 

Fig. 2 *Artist 206 

Contributions 

*Image 

Database 

Challenge Words/Epicts image Libraryl Designs 

Current Group Rateable Epicts Tabulating Module 

23 

Presentation Engine \ 

240 

Synthesis/Suggestion Engine 

260 

Speech Recognition Module i Community Votes 

*current epicts 

best epicts 

best artists 

most challenging Advertising Module 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 6 of 34
U.S. Patent Nov. 24, 2015 Sheet 3 of 16 US 9,192,861 B2 

internet 305 Artist Submissions i Image Library Builder 

Content Collections 

Libraryl image solicitations 

Content 

Builder Community Suggestions 321 Dictionary/Encyclopedia : Pry references 306 mage 

: Database 

324 

Native DB 

e age Formatting Interface 

323 

322 

Secondary 

image 

Database 

324' 328 

Native or 

Image processed : 

Metadatal supplemental TT to other routines? 

descriptors X modules 326 image data 

www.waw 322 t 

32O 

Semantic 

Supplementer 

327 325 

329 

Image Timing Related Image Origin ID 

Components Components Components 

 

 

 

 

 

 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 7 of 34
U.S. Patent Nov. 24, 2015 

Fig. 3B 

Identify Game Participants Rule, 

Initiate game across all participants 342 

Present Images (or suggestions) to 

participant devices in accordance 

with Image Control data/game rules 

343 

Collect response/ratings data from 

participants 344 

s 

Determine participant winners/ 

prizes 345 aaaaaaaa 

Adjust game status/score 

— 

346 

Tabulate performance of epicts and 

participants/journal game 347 

ACCount to Artists/content providii 

Recommend other epicts/artists 349 

Sheet 4 of 16 US 9,192,861 B2 

Presentation Engine 

Rotating? 

Community Random topic Ratings Rotating? 

Timely/ Randon artist 

Contemporary CAPTCHA test 

Challenge Image selection logic, O 

Present Images on website in 

accordance with image Control datal 

game rules 351 

ynthesis/Suggestion logic - Supplemental images 

Publish to 

HSocial network sites.-bSearch engines, etc. 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 8 of 34
U.S. Patent Nov. 24, 2015 Sheet 5 of 16 US 9,192,861 B2 

Fig. 3C 

G D 

image Score DB Image Ds 

332 333 

Participant Participant IDs 

334 

SCOre DB 

332 

Image Recognized? 

Recognition time Calculate Image/Participant Score 

Presentation 335 

mode 

made ratin 9 9 Author SCOre OB 

Calculate Author Scores 336 332" 

Publish (Optional). image difficulty rankings 

image ratings 

group image ratings 

author ratings 

participant ratings 337 

game scores r 

image Presentation% identified Mean Standard Ratings Difficulty Demo 

D mode within time recognition deviation Score Rating graphic ime (RT R data 

SD Ratings Favorite image (RT)|Score Authors IDs 

authored 

Author D Average images Mean Working Recognition Recognition On image 

Score ine challenge 

Tabulatind Module 

 

 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 9 of 34
U.S. Patent 

Present 

Image Data 

in 

aCCOrdance 

with Image 

Components 

and timing 

data (or 

downloads 

image 

Suggestions 

in manua 

mode) 363 

360 

Nov. 24, 2015 Sheet 6 of 16 

Image IDs 

332 

US 9,192,861 B2 

Manual mode 

36 

Supplemental 

Option 

Selected? 

Accelerate for 

manual mode 

368 

Suggestion 

Option 

Selected 2 

366 

362 

Retrieve secondary image 

database information for image ID 

364 

Present secondary image data to 

participants (or hints to artist in 

manual mode) 365 

SYNTHESIS - 

SUGGESTION MODULE 

Normal picture 

mode 367 

 

 

 

 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 10 of 34
U.S. Patent Nov. 24, 2015 Sheet 7 of 16 

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

Solicit participant voice inputs 

376 

Repeat as Develop participant voice prints| 

necessary templates 

377 

Test voice print/template discrimination 

(multi-voice input differentiator) 378 

US 9,192,861 B2 

Collect test 

Words/ 

utterances 

Excited Words 

(prosody 

Compensation) 

Crowd mode or 

distributed mode 

Amplitude 

Compensation 

/normalization 

Set Up Mode 

Receive image ID Metadata (answer tag, 

Semantic answer tags) 38 

Load grammarlacoustic model for answer 

Words 382 

Receive answer input(s) 

(deconvolve as necessary) 83 

Perform speech recognition 

(on each utterance source) with time 

Stamping 384 

Identify winner (if any) 85 

May be single microphone input 

Source May be multiple sources 

Game Mode 380 

locat, distributed, or server 

side 

N 370 Speech Recognition Module 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 11 of 34
U.S. Patent Nov. 24, 2015 Sheet 8 of 16 US 9,192,861 B2 

FIG. 3F 

Keywords/topics 

391 

Determine epict 

393 

Defer Ad Option 

Sponsored epicts 

Serve Ad 396 

390 ADVERISING MODULE 

 

 

 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 12 of 34
U.S. Patent Nov. 24, 2015 Sheet 9 of 16 US 9,192,861 B2 

Fi 9. 3G 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 13 of 34
Sheet 10 of 16 US 9,192,861 B2 

3H 

U.S. Patent NOV. 24 , 2015 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 14 of 34
U.S. Patent Nov. 24, 2015 Sheet 11 of 16 US 9,192,861 B2 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 15 of 34
U.S. Patent Nov. 24, 2015 Sheet 12 of 16 US 9,192,861 B2 

Fig. 4 

E-Picture Games 

Solitaire Challenge 

411 

Start a Team Game 

410 

Upload an Epict 

Watch a Game 415 p p 420 

Submit a Challenge Word 

42 

Top N lists 

Best Teams 

Best individuals 

Top Scores 

Top Artists 

Enroll a team 430 Find/Rate an Epict 433 

Find team mates 431 Find/Rate an Artist 432 

/ Challenge Epict it 

4 O r Challenge Epict #2 

Challenge Epict #3 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 16 of 34
U.S. Patent Nov. 24, 2015 Sheet 13 of 16 US 9,192,861 B2 

Fig. 5 

individuals 

515 

Benchmark Sets (s) (s) 

20 

500 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 17 of 34
U.S. Patent Nov. 24, 2015 Sheet 14 of 16 US 9,192,861 B2 

Fig. 6 

Challenge Word for team A 

10 

(Optional) Team B 

votes (eliminates) one 

Or more alternatives 

620 

Epict image atternatives presented to team 8 

615 

Select Epict image atternative to be 

presented to team A 625 (Optional) Randomizer 

630 

(Optional) Destructive 

interference (remove 

selected features) 635 

Select final form of Epict image alternative 

and present to team A 640 

Prediction Module 

600 

 

 

 

 

 

 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 18 of 34
U.S. Patent Nov. 24, 2015 Sheet 15 of 16 US 9,192,861 B2 

Fig. 7 

User Submissions Artist Submissions 

Z31. 717 721 

Picture Selected Time Remaining Artwork Selected 

Process artwork 

(draw image in 

AnSWer original style, or 

use decomposed 

Randonize 

picture display 

data within Canvas 

(fill randomly over 

Hint Board 

Advertisement 

715 1234 

Game Progress / 

71.6 Game Score 

 

 

 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 19 of 34
U.S. Patent Nov. 24, 2015 Sheet 16 of 16 US 9,192,861 B2 

Fig. 8 

Advertisers 

840 Auction (dual) 

830 

Search Engine Result? 

Web Page 

Epict Ad Stock Stock 

8 awawawaw 1 5 

Ad/Epict Combination 

800 

 

 

 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 20 of 34
US 9,192,861 B2 

1. 

MOTION, ORIENTATION, AND 

TOUCH-BASED CAPTCHAS 

RELATED APPLICATION DATA 

The present application is a divisional of U.S. patent appli 

cation Ser. No. 12/494,083, filed Jun. 29, 2009, which claims 

the benefit under 35 U.S.C. 119(e) of the priority date of 

Provisional Application Ser. No. 61/076,495 filed Jun. 27, 

2008. Both of those applications are incorporated by refer 

ence in their entirety. The application is further related to the 

following applications, all of which are incorporated by ref erence herein: 

Pictorial Game System & Method: Ser. No. 12/494.013, 

filed Jun. 29, 2009 

Internet Based Pictorial Game System & Method: Ser. No. 

12/494,065, filed Jun. 29, 2009 

FIELD OF THE INVENTION 

The present invention relates to challenge systems which 

include pictorial and/or motion features. 

BACKGROUND 

A popular board game introduced many years ago goes by 

the name of PictionaryTM. In this game teams of players 

compete against each other by having one member of the 

team manually draw an object (which could be a person, 

place, thing, action, etc.) while the other members of the team 

attempt to guess the identity of the object. A time factoris also 

accounted for in that the team must determine the object within a certain time period. PictionaryTM is one form of pictorial game that has enjoyed 

great Success in the market, but it has not been rendered into 

a form suitable for Internet accessibility. One example of a 

drawing game that has been implemented in electronic form 

is depicted in US Publication No. 2007/0018393, incorpo 

rated by reference herein. It is apparent that there is a need for 

other similar types of games with different rules, objectives 

that can be enjoyed in electronic form, and particularly on 

sites available on the Internet (including Social network sites) 

and other hand held devices (including advanced phones). 

CAPTCHAs (Completely Automatic Public Turing Tests 

To Tell Humans And Computers Apart) are well-known for 

controlling access to computers and resources on the Internet. 

A recent example by Sanghavi et al. (US Publication No. 

2009/0113294) incorporated by reference herein demon 

strates the use of a progressive CAPTCHA. The use of a 

progressive technique increases the cost to spammers seeking 

to use human capital to circumvent text based CAPTCHAs. 

While Sanghavi etal propose their approach for conven 

tional (distorted) text type challenges, they fail to appreciate the potential for image based challenges. In particular, rec 

ognition of images can be incorporated as part of a picture 

based game to develop appropriate challenges, unlike text 

approaches. 

SUMMARY OF THE INVENTION 

An object of the present invention, therefore, is to reduce 

and/or overcome the aforementioned limitations of the prior 

art. 

A first aspect of the invention concerns a method of con 

ducting a pictorial based electronic game comprising the 

steps: presenting an electronic image authored by a human 

composer within a graphical interface; wherein the electronic 

10 

15 

25 

30 

35 

40 

45 

50 

55 

60 

65 

2 

image is presented in a visual sequence that Substantially 

tracks arendering of such image as performed by Such human 

composer, associating the electronic image with one or more 

concepts; providing a rule set for a user to use to predict an 

identity of the one or more concepts; processing responses 

from the user in compliance with the rule set to determine if 

the user has accurately determined an identity of the one or 

more concepts associated with the image; wherein the elec 

tronic image is presented to the user during at least a portion 

of the visual sequence substantially without explicit textual 

information describing the one or more respective concepts. 

The electronic image can be presented on a user's televi 

sion system, a browser, a portable computing device Such as 

a PDA, cell phone, etc. The images can be created automati 

cally or by human composers with an online graphical com 

position tool. In some embodiments the responses from the 

user are tabulated to compute a performance of the user, the 

electronic image, and an artist associated with the electronic image. 

Some embodiments may also provide one or more audio 

visual cues and/or clues to the user to assist in decoding the 

electronic image. An electronic advertisement can be pro 

vided as part of the one more audiovisual cues and/or clues. 

The audiovisual cues and/or clues can be derived from images associated with a second set of concepts semantically related 

to the one or more concepts. 

For some applications the electronic image is presented at 

different time intervals with differing levels of resolution, 

noise and/or pixelation. 

In other applications the responses are provided as speech 

utterances and recognized by a speech recognition engine. 

The speech recognition engine can be configured to listen to 

multiple users simultaneously to determine a winner. 

A graphical tool can be provided to permit the user to create 

a second electronic image to be decoded by a second user. 

Users can also avail themselves of a file uploading tool in 

Some cases for permitting the user to upload a photographic 

file image to be decoded by a second user. 

The invention also supports team play for some embodi 

ments. Different teams can compete against each other, even 

from at different physical locations. In team play the teams 

can influence an identity and/or a composition of the elec 

tronic image during a competition. 

In some embodiments the ability to repeat the game for a 

plurality of second electronic images constituting a bench 

mark set is allowed, so that the user's responses are measured 

against other users to determine a performance rating relative 

to the benchmark set. A recommendation engine can be used 

to recommend at least one of an artist or a second electronic 

image to the user based on the responses. 

Still further embodiments derive the electronic image auto 

matically based on a content of an Internet page. In some 

portable applications a series of different electronic images 

are presented in response to a change in orientation of a 

portable device which is presenting the images. 

Another aspect of the invention concerns conducting a 

pictorial based electronic game comprising the steps: present 

ing an electronic image within a graphical interface, the elec 

tronic image being derived from a scene captured by a 

machine and embodied in a digital and/or photographic form; 

wherein the electronic image is presented in a visual sequence 

that varies so that content is gradually revealed for the image 

over a first time period; associating the electronic image with 

one or more concepts; providing a rule set for a user to use to 

predict an identity of the one or more concepts; processing 

responses from the user in compliance with the rule set to 

determine if the user has accurately determined an identity of 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 21 of 34
US 9,192,861 B2 

3 

the one or more concepts associated with the image; wherein 

the electronic image is presented to the user during at least a 

portion of the first time period substantially without textual 

information describing the one or more respective concepts. 

An additional step can be performed to combine the digital 

and/or photographic form with other metadata for the scene in 

to the electronic image. A computing system can be trained 

and employed to automatically find digital and/or photo 

graphic form images in one or more Internet accessible image 

repositories. The electronic images can include scene com 

ponent, Scene timing information and device orientation for 

the scene as part of the electronic image. Advertising infor 

mation for the scene can also be incorporated as part of the 

electronic image. Still another aspect concerns conducting a pictorial based 

electronic game comprising the steps: presenting an elec 

tronic image within a graphical interface, the electronic 

image being derived from a scene captured by a machine and 

embodied in a digital and/or photographic form; wherein the 

electronic image is presented in a visual sequence that varies 

so that content is gradually revealed for the image over a first 

time period; associating the electronic image with one or 

more concepts; providing a rule set for a user to use to predict 

an identity of the one or more concepts; processing responses 

from the user in compliance with the rule set to determine if 

the user has accurately determined an identity of the one or 

more concepts associated with the image; wherein the elec 

tronic image is presented to the user during at least a portion 

of the first time period substantially without textual informa 

tion describing the one or more respective concepts. 

Another aspect of the invention concerns operating an 

Internet based pictorial game with a computing system com 

prising: providing a database of game images related to asso 

ciated topics, the game images being generated by human 

composers; associating the electronic image with one or more 

concepts; providing a rule set for a user to use to give a 

prediction of an identity of the one or more concepts; present ing the game image with the computing system in a visual 

sequence that Substantially tracks a rendering of Such image 

as performed by Such human composer; automatically pro 

cessing responses from the user in compliance with the rule 

set with the computing system to determine if the user has 

accurately determined an identity of the one or more concepts 

associated with the image; wherein the electronic image is 

presented to the user substantially without explicit textual 

information describing the one or more respective concepts. The game image is preferably presented within a web page 

loaded in an Internet browser, and can be part of a game website, and/or integrated as part of a personal profile page on 

a social networking site. 

As with the other approaches users can participate as a 

team or in competition against each other in providing the 

predictions. Their scores can be based on a time required to 

give an accurate prediction for the image. Additional visual 

clues semantically related to the respective concepts can be 

presented simultaneously to the user with the game images. 

Advertisements can be matched to the game images based 

on a plurality of bids provided in an auction by multiple 

advertisers. The ads can be matched to artists based on a 

plurality of bids provided in an auction by multiple advertis 

ers for artists composing the game images. 

Still another aspect concerns operating an Internet based 

pictorial game with a computing system comprising: provid 

ing a database of game images related to associated topics; associating the game image with one or more concepts; pro 

viding a rule set for a user to use to give a prediction of an 

identity of the one or more concepts; presenting the game 

10 

15 

25 

30 

35 

40 

45 

50 

55 

60 

65 

4 

image with the computing system in a visual sequence that 

varies so that content is gradually revealed for the image over 

a first time period; automatically processing responses from 

the user in compliance with the rule set with the computing 

system to determine if the user has accurately determined an 

identity of the one or more concepts associated with the 

image; wherein the electronic image is presented to the user 

during at least a portion of the first time period Substantially 

without explicit textual information describing the one or 

more respective concepts. The game image is preferably 

revealed in a spatially random sequence. 

A further aspect concerns operating an Internet based pic torial game on a social networking site with a computing 

system comprising: providing a database of game images 

related to associated topics; associating the electronic image with one or more concepts; wherein the game images are 

contributed by members of the social networking site; pre senting the game image with the computing system within a 

Social networking site page in a visual sequence that varies so 

that content is gradually revealed for the image over a first 

time period; wherein the electronic image is presented to the 

user during at least a portion of the first time period substan 

tially without explicit textual information describing the one 

or more respective concepts; automatically processing 

responses from the user with the computing system to deter 

mine if the user has accurately determined an identity of the 

one or more concepts associated with the image; selectively publishing results of the user's performance on the Social 

networking site. In some embodiments multiple members of 

the Social networking site can play the game in a collaborative 

mode. 

Yet another aspect is directed to a pictorial based electronic 

game that includes one or more Software modules imple 

mented in a computer readable medium to effectuate the 

above aspects. 

Another aspect concerns a method of testing recognition of 

images for a CAPTCHA (Completely Automatic Public Tur 

ing Tests To Tell Humans And Computers Apart) system comprising: automatically electing an electronic image for a 

participant with a computing system, wherein the electronic 

image comprises imagery other than words; associating the 

electronic image with one or more concepts; progressively 

revealing information over a first time period for the elec 

tronic image within an interface for the participant using the 

computing system; automatically processing responses from 

the participant with the computing system to correlate an 

accuracy of guesses from the participant for the one or more 

concepts associated with the electronic image. 

Preferred embodiments use a step of calculating a percent 

age of participants able to decode and accurately guess the 

one or more concepts within a predetermined time. The elec 

tronic image can be presented at different time intervals with 

differing levels of resolution, noise and/or pixelation. The 

electronic images can then be randomly selected as needed 

within a CAPTCHA system for granting access to resources 

of a second computing system, including a portable electron 

ics device. 

A further aspect concerns providing a visual CAPTCHA 

(Completely Automatic Public Turing Tests To Tell Humans 

And Computers Apart) challenge comprising: automatically 

electing an electronic image for a participant with a comput ing system, wherein the electronic image comprises imagery 

other than words; associating the electronic image with one or 

more concepts; progressively revealing information over a 

first time period for the electronic image within an interface 

for the participant using the computing system; automatically 

processing responses from the participant with the computing 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 22 of 34
US 9,192,861 B2 

5 

system to detect if a guess from the participant for the one or 

more concepts associated with the electronic image meets a 

required threshold. 

In some embodiments the responses are received inaudible 

form and processed by a speech recognition device adapted to 

recognize human voices. The visual CAPTCHA challenge 

can be combined with a physical CAPTCHA challenge requiring the user to manipulate the portable computing 

device in a prescribed manner. The physical CAPTCHA chal 

lenge can also interact with elements of the electronic image 

and/or other objects presented in the interface. 

Another aspect is directed to providing a motion and/or 

orientation based CAPTCHA (Completely Automatic Public 

Turing Tests To Tell Humans And Computers Apart) chal 

lenge on a portable computing device comprising: automati 

cally electing a manipulation challenge for a user with the 

portable computing device, wherein the manipulation chal 

lenge includes a set of physical manipulation instructions for 

the portable computing device for the user; automatically 

processing sensory data associated with user movements and/ 

or manipulations of the portable computing device; automati 

cally determining if the user movements and/or manipula 

tions from the user meet a required threshold for satisfying the 

set of manipulation instructions. 

The set of physical manipulation instructions for the 

motion and/or orientation based CAPTCHA challenge can be 

presented in an electronic image, and/or in audible form by a 

text to speech engine operating on the portable computing 

device. The set of physical manipulation instructions typi 

cally require the user to move the portable computing device 

So as to manipulate objects presented in an interface. For 

example, the set of physical manipulation instructions may 

require the user to orient a needle of a virtual compass in a 

specified direction, or explain (in spoken form) what direc 

tion the needle is pointing to at any moment in time. Alterna 

tively the physical manipulation instructions can require the 

user to speak or blow into a microphone. 

Still another aspect concerns providing a touch based 

CAPTCHA (Completely Automatic Public Turing Tests To 

Tell Humans And Computers Apart) challenge on a portable computing device comprising: automatically electing a 

manipulation challenge for a user with the portable comput 

ing device, wherein the manipulation challenge includes a set 

of physical touch instructions for the portable computing device for the user; automatically processing touch sensory 

data associated with user movements and/or manipulations of 

objects within a graphical interface of the portable computing 

device; automatically determining if the user touch sensory 

data meets a required threshold for satisfying the set of physi 

cal touch instructions. 

In some embodiments the set of physical touch instructions 

for the motion and/or orientation based CAPTCHA challenge 

are presented in an electronic image, and/or in audible form 

by a text to speech engine operating on the portable comput 

ing device. The instructions may require the user to arrange a 

set of letters on the graphical interface, or orient an object in 

a proper direction. 

A further aspect concerns the fact that the various 

CAPTCHA challenges are embodied in a computer readable 

form and implemented as one or more routines executing on 

a portable computing device which also has motion, orienta 

tion, touch and/or microphone sensors. 

DESCRIPTION OF THE DRAWINGS 

FIG. 1 is an illustration of a first embodiment of a pictorial 

game implemented on a stand alone computer or console; 

10 

15 

25 

30 

35 

40 

45 

50 

55 

60 

65 

6 

FIG. 2 is an illustration of a second embodiment of a 

pictorial game implemented on an Internet website; 

FIG. 3A illustrates a preferred embodiment of a pictorial 

game image builder, 

FIG. 3B illustrates a preferred embodiment of a pictorial 

game image presentation engine; 

FIG. 3C illustrates a preferred embodiment of a pictorial 

game tabulating module: 

FIG. 3D illustrates a preferred embodiment of a pictorial 

game synthesis-Suggestion module: 

FIG. 3E illustrates a preferred embodiment of a pictorial 

game speech recognition module; 

FIG. 3F illustrates a preferred embodiment of a pictorial 

game advertising module: 

FIGS. 3G, 3H and 3J show examples of user contributed 

content that can be artistically styled and formatted for 

embodiments of the present invention: 

FIG. 4 depicts a preferred embodiment of a pictorial game 

website; 

FIG. 5 illustrates a preferred embodiment of a pictorial 

game team rating module: 

FIG. 6 illustrates a preferred embodiment of a variant of a 

rule set used in a pictorial game; 

FIG. 7 depicts a preferred embodiment of a game screen 

used in a pictorial game; 

FIG. 8 depicts a process for integrating advertisements and 

epicts in accordance with the present invention. 

DETAILED DESCRIPTION 

FIG. 1 is an illustration of a first embodiment of an elec 

tronic pictorial game 100 implemented on a standalone com 

puter or console. In this embodiment an input device 110. 

which may be an electronic tablet, pad, or equivalent canvass 

is used by a human artist to create electronic pictures, gener 

ally referenced herein as "epicts.” Again, as noted above, the 

epicts may represent human authored viewable renditions of 

human perceivable concepts, which, in the broadest sense 

may be in the form of ideas, people, places, things, actions, 

emotions, etc. In other cases the epicts may be conventional 

electronic photographic images which have been formatted to 

be compatible with an accompanying presentation system. 

A stylus 115 is preferably used to aid with such process, 

and is adapted for human manipulation Such as in the form of 

a handheld pen. Other types of manipulable tools can also be 

used of course depending on the particular game and desired 

manipulation (eye, head, foot, etc.) required to provide input. 

The epict is displayed for viewing by the artistand other game participants on a display 120 which may be coupled to or part 

of a computing device with a Software/hardware engine adapted for rendering the image inappropriate form. Thus the 

display may be part of a television, a computer monitor, etc., 

and the accompanying computing device may be a personal 

computer, a game console or other similar device. 

Data is communicated to and from the input device 110 

through a communications channel 125 which is preferably a 

wireless link. In some embodiment it may be desirable to 

provide feedback to the artist in the form of game rules, 

Suggestions, etc. to the tablet for consideration. The image 

data can be transferred over the channel in any desired form. 

Other variations of such architecture are of course possible, such as the use of multiple tablets/pads 110 (and accompa 

nying stylus devices 115) to allow for multi-player simulta 

neous play. In Such instances it may be useful to divide dis 

play 120 into distinct regions and partitions for displaying the 

various artistic renditions. For example, with 4 artists the 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 23 of 34
US 9,192,861 B2 

7 

display could be divided into 4 quadrants, which would allow 

participants to view 4 simultaneous renditions of an object. 

In other embodiments a handheld pointer 130 may be used 

to draw directly on the screen display 120. The pointer may be 

of the type used in the Wii gaming platform which is a well 

known system for this type of interactivity. For some embodi 

ments it may be possible to use an electronic PDA or cell 

phone that is synced to the gaming console 120 to provide the 

epicts. 

In its most basic form the pictorial game 100 can be con 

figured to use a set of rules and scoring options that imitates 

the functionality of the Pictionary game noted earlier. For 

example the participants could be required to rotate the artist 

function, the participants could be required to move a visible 

token across a virtual playboard projected on display 120, the 

participants required to guess the concept in a particular time 

period, etc. In this version of the game, the participants play 

on teams which compete against each other to guess what 

concept is being drawn by a co-team member. The concepts 

may be supplied by the game engine 120, or may be supplied 

by a remote server 145. It would be relative easy for these 

functions to be implemented electronically and this option 

may be attractive to persons who like Such game and want to 

duplicate the experience in enhanced form with an electronic 

platform. Nonetheless a variety of additional or modified 

rules and options for drawing, scoring, guessing, etc., could 

be elected depending on the desired game format, and Such 

features could be adopted on a game by game basis by the 

game host and participants. 

As an example of Such modified rules, the game could be 

run without requiring players/participants to provide 

responses orguesses. In one Such case artists could contribute 

their epicts in connection with an ongoing event that they are 

attending (e.g., a concert, sporting event, trade show and the 

like), viewing (as in a television broadcast or gaming experi 

ence seen on receiver 147) or hearing. The epicts could be 

directed to different topics and aspects of the vent, and could 

form part of a collective artistic gallery that is presented to a 

larger audience of the event simply for viewing and entertain 

ment. Again, the organizers of such event could use human or 

machine editors to Solicit and select particular ones of the 

epicts that appear Superior to others. For example at a sports 

stadium/arena a fans live artistic contribution could be dis 

played on the scoreboard and the like. 

In other embodiments the invention could be used at vari 

ous events to Solicit and encourage crowd and audience par ticipation. For example, at a music concert, attendees could 

be given a chance to enter their sketches of the events occur 

ring at the concert, such as the make up of the band, carica 

tures of the band members, etc. These sketch submissions 

could be uploaded and viewed by event managers to identify 

and select particular sketches for recognition. The recogni 

tion in turn could consist of financial or non-monetary incen 

tives for the contributor. In other instances fan contributions 

could be displayed and viewed dynamically (along with attri 

bution to the fan by name, seat number, etc.) as they are 

created for further enjoyment. This same format could be 

used at Sporting events and other large public gatherings 

where it would be desirable to report graphical data in real 

time for public consumption. 

For some games it may be desirable for the participants to 

upload the game data to an offsite location for storage, review, 

etc., using a data feed 140. The epicts, concepts and partici 

pant information could thus be integrated into the feed 140 

which can be incorporated within a social network site page 

or broadcast to other authorized friends and third parties. For 

10 

15 

25 

30 

35 

40 

45 

50 

55 

60 

65 

8 

example many Social networking sites now include specific 

areas for allowing user contributions to be added dynamically 

to a personalized page. 

Similarly the epicts may be compiled by concept, topic, 

etc. by a remote server 145 for presentation to other network 

users. The remote server may be programmed with additional 

game reward logic to give prizes, recognition, etc. to partici 

pants and teams in real-time to facilitate and increase game 

interest. If desired, the entire game could be journalled and 

replayed (at higher speeds as well) at a later time for viewing. 

It will be understood by those skilled in the art that other 

features could be implemented into the present embodiment 

to add to the interest/excitement of the game. For example, an 

express option to include musical cues could be added, so that 

the artists rendition is accompanied by a song or other 

audible clue that assists the participants to guess the concept in question. 

To assist with this function automated music matching 

logic 150 could be used to coordinate the visual rendering of 

the epict with the music chosen by the artist. For example if 

the artist took 30 seconds to sketch a picture of a bird, the 

music matching logic could be used to automatically select 

and extract an equivalent (an relevant) 30 seconds of music 

(perhaps with lyrics masked to avoid explicit rendering of the 

answer) from a song such as “Blackbird” by the Beatles to 

present with the epict. The music excerpts can be obtained 

from any one of a number of online sources, and may be 

further cropped as needed to accommodate a particular epict. 

To make their incorporation easier, they can be catalogued by 

different topic tags, so that they can be identified and selected 

easily by a human artist. By keeping the music excerpts short 

the issues of copyright and fair use can be minimized. 

As further explained, speech recognition (as described 

below in more detail) could also be incorporated to recognize 

participant utterances and assist with scoring. 

A second embodiment of a pictorial game 200 imple 

mented on an Internet website is shown in FIG. 2. The com 

ponents of this system generally include a game module 210 

which has access to a database of images 205 for creating a set 

of epicts 215. 

The images 205 can be derived from a number of sources, including community artists contributions, pictures/graphi 

cal content contributed by members, and other image data 

automatically retrieved from online image repositories (e.g. 

Such as from the Google image search engine). In some 

embodiments the images could be linked from members 

other Social networking pages (e.g. Facebook, MySpace, 

etc.), online photograph accounts (e.g., such as Flickr, Pic 

assa). Details of such compilation are described below in 

connection with FIG. 3A. 

Returning to FIG. 2, the game module 210 includes a 

number of components, including an image library 220, a 

tabulating module 230, a presentation engine 240, a synthe sis/suggestion engine 260, and a speech recognition module 

270. An advertising module 280 can also be incorporated if 

needed. Each of these routines is preferably implemented as 

one or more Software routines adapted for executing on an 

Internet accessible server, and capable of interacting with 

input from client browsers, back end databases, website pages 

and other similar processing/data resources. 

The aforementioned routines are specifically responsible 

for supporting an Internet website page 250, which includes a 

number of features and functions for interacting with users of 

an online gaming site. For example the gaming page 250 

could include a set of challenge words/epicts 290; a set of 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 24 of 34
US 9,192,861 B2 

ratable epicts 294, and a community voting section 296. Each 

of these is discussed briefly here and in more detail below in 

connection with FIG. 4. 

In FIG. 2, the challenge words/epicts 290 represent a set of 

words—to be drawn by contributing artists and a set of 

epicts—drawings to be guessed at by other participants. As 

seen below, selecting this feature allows the member to either 

invoke a drawing canvass/tool (to create an epict for a specific 

word) or invoke an image presentation screen (to guess what 

word is appropriate for an epict concept). Both of these func 

tions are described in more detail below. The ratable epicts 294 feature allows members to examine specific epict-con 

cept pairings, and rate them on their artistic/creative merit. 

Thus, for example, this section of the site may be used to view 

specific artistic renditions of particular concepts. Members 

may be encouraged or requested to contribute epicts on par 

ticular themes, Subjects, objects, etc. The ratings are then 

captured and compiled for community perusal and other pur 

poses described herein. A number of different data collection 

schemes exist for this purpose and could be used for Such 

purpose, including conventional polling software. 

Furthermore the gaming page may include options and 

features for presenting the various tabulations accumulated 

by the site for user entertainment. As an example, particular epicts, artists, concepts could be recognized, along with top N 

lists and the like. 

It will be understood that all of the above applications (challenge words/epicts 290, ratable epicts 294, community 

voting 296) and others could also be implemented by mem 

bers on an individual basis on their respective web pages. For 

example the feature could be implemented as a separate appli 

cation for inclusion within pages of a social networking site, 

in the same way that Facebook users can incorporate other 

games or features on their profiles or data feeds. This would 

allow the concepts of the present invention to be perceived 

and disseminated to a wider audience. 

As seen in FIG. 2, the epicts 215 are preferably stored 

according to concept, artist, etc. to create a master community 

database of accessible image content. Each epict is preferably 

tagged with an artist name (if available), one or more concept 

(word) tags, and other data described below. 

The individual modules responsible for carrying out the 

main functions of the invention are now described in more 

detail in connection with FIGS. 3A-3F. A preferred embodi 

ment of an image library builder module 320 shown in FIG. 

3A includes a number of components for constructing a data 

base of epicts. As seen there candidate images 305 for epicts 

can be received either through manual contributions or auto 

matic selection from a variety of sources, including Internet 

collections, private collections, artist Solicitations, artist Sub 

missions, etc. Candidate concepts 306 can be similarly received manually (through member/community explicit 

contributions) or through automatic collection/generation 

from online resources such as dictionary and encyclopedia 

references. In the latter case, local/topical themes (regional events/happenings) or contemporary themes (holidays, sea 

Sonal events) may be employed to augment a list of concepts 

for artists to consider. Other sources of such data will be 

apparent to those skilled in the art. 

A library/content builder routine 321 retrieves and pro 

cesses these image and concept/word contributions. The 

native image data 322 is examined to see if it is compatible with a presentation engine (described below) format, and, if 

not, it undergoes further formatting treatment by a routine 

323 to convert it into acceptable form. A product currently 

offered by Studiomee titled “Pocket Artist’ provides this type 

of functionality, and other similar products could be used. The 

5 

10 

15 

25 

30 

35 

40 

45 

50 

55 

60 

65 

10 

Pocket Artist product and Pocket Sketch products are avail 

able from the iTunes app store. A similar feature is offered by Sketchmyphoto also available from their website. 

For example in Some embodiments of the invention an 

epict may consist not of artistic drawings, but solely of an 

actual photographic image of an object, Such as a bird in a 

tree. The image file may be filtered and reduced (in terms of 

visual features) to render a simplified version of the image 

that can be broken down into discrete components. For 

example all the background elements of the image could be 

stripped out to show only the bird itself. Then the image could 

be divided—logically or physically—into distinct regions 

that are further reduced in complexity, so that, for example, 

only basic features and outline of the shape of the bird remain. 

These basic features thus make up and are then used to Syn thesize the image in later renditions of the epict. Various 

forms of conventional picture processing software are 

capable of this type of operation and could be used for this 

purpose. For example the Adobe Photoshop Elements soft 

ware includes several image “filtering features (for example 

the “poster edges' and “fresco' look found under the artistic 

category, and several aspects of the sketch category (photo 

copy for example)) that provide this type of capability. The 

advantage of Such format is that it can take up less space and 

is somewhat more consistent in terms of giving the game an 

artistic look. An example of an original JPEG photo and an 

artistically formatted counterpart is shown in FIG. 3G. Even 

in the black and white reduced version the image is still 

recognizable. 

Nonetheless it is possible to use simple JPEG (or similarly 

compressed data formats) for visual images if desired. Again 

the precise format of the epicts will be a function of the 

specific design goals and is expected to vary from application 

to application. 

The images are also processed by an image Supplementa 

tion module 325 which is used generally to locate other asso 

ciated and relevant content for the candidate image as part of 

the pictorial game. For example metadata routine 326 is 

adapted to identify/extract appropriate content tags. The con 

tent tags represent correct responses or answers that may be 

associated with the epict in question. For example a picture of 

a cat may be tagged with the descriptors “cat, “pet” “purr” 

and so on. In some cases Such data may be explicitly provided, 

in others it may be determined by reference to various soft 

ware tools available on the market for identifying the nature 

of objects shown in an image. 

A semantic Supplementer routine 327 examines the tags/ 

descriptors and other concepts and determines other seman 

tically related concepts and words. Thus for example, in the 

above example, the “pussycat' and “feline' may be deter 

mined (with reference to a reference semantic tool Such as 

WORDNET) to be semantically related to “cat.” Based on 

Such association additional text tags could be appended by a 

routine 322 along with other native or processed Supplemen 

tal image data associated with the content of Such additional 

text tags. Thus, for example, if image data already exists for 

Such semantic relatives, it can be associated with the candi 

date epict in question for use in the game. The completed epict is then Stored in a primary image 

database 324, along with the other associated metadata/pa 

rameters noted above. A secondary image database 324 con 

tains related images that can be accessed/related as desired to 

the entries in the main database. A database interface 328 is 

used to query and access epict content as needed by other 

routines as described below. A preferred embodiment of an 

epict record 329 is also shown in FIG. 3A. Each epict is 

preferably coded with the following parameters: 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 25 of 34
US 9,192,861 B2 

11 

a unique image ID (a numerical designation of any appro priate length): 

an image name—(a listing of the appropriate content tags, descriptors, etc.); 

image components (some images may be stored in a format 

where the artist’s rendering of the image is journaled 

into distinct components; in other cases the contribu 

tions can be automatically rendered into components by formatting routine 323); 

image component timing (in instances where multiple 

components exist, timing information can be provided to 

indicate what component should be drawn first, second, 

etc., and at what speed); related image components (as noted above, a pointer to 

other secondary images in database 324 that can be 

associated with the epict); 

Origin ID (a reference to an artist, data source or the like 

from which the epict was derived) 

The preferred approach uses a scheme in which the elec 

tronic image is presented in a visual sequence that Substan 

tially tracks a predicted rendering of Such used by a human 

composer. In other instances where the content is not based on 

a human artist, the image can be presented visually as the 

system predicts a human artist would render it, or in a manner 

that is logically related to the Subject matter of the image. 

Those skilled in the art will appreciate that other param 

eters can be included as well depending on the application and 

desired functionality. For example some epicts could be 

tagged with automatic metadata (geocoding, time, etc.) as 

desired, along with additional pointers to appropriate musical 

relatives (as noted above). 

A preferred embodiment of the process used in a pictorial 

game image presentation engine 340 is depicted in FIG. 3B. 

As alluded to earlier, the present invention can be imple 

mented in various forms. For example, in an automated mode, 

the epicts can be drawn from database 324 and presented 

within a portion of an interface for a webpage to be reviewed 

and evaluated by individual members. The members prefer 

ably have the ability to select (and/or subscribe to) the epicts by category or topic. The members could then participate in a 

solo-type interaction with the game through these individual 

epicts. Individual members may elect for the epicts to be 

presented on their respective profile pages as a customizable 

widget in accordance with one or more programmed themes 

of their selection. For example a particular user may direct the 

epict engine to randomly select topics based on the news of 

the day, so as to cause the system to dynamically present 

epicts directed to such concepts on his/her site as such content 

develops over time. 

In other applications nonetheless it may be desirable for 

members to participate in epict presentations/guessing in a 

team based mode. In Such instances members may elect to 

join and sign up as a team in accordance with a set of rules as 

shown at 341. Invitations and sign-ups can be implemented 

with any number of known tools. The rules may be based on 

the particular game chosen so that for example teams may 

play directly against other teams, or in Some cases, may simply compete against a reference set of epicts. Other for 

mats are of course possible. The rules may further specify scoring and other game options shown below as well. 

The game is then initiated at step 342 for the participants in 

question. The participants may enter the game directly by 

reference to a game ID, an authorization token or similar 

mechanism known in the art. In one embodiment the teams 

are expected to review/create a number of separate epicts on 

a rotating basis. That is, a first team would create a first epict 

for a second team to guess within a certain time period, and 

10 

15 

25 

30 

35 

40 

45 

50 

55 

60 

65 

12 

Vice versa. The game can consist of a plurality of rounds 

(again selectable by the participants) in which different epicts 

are provided to the respective teams. The game final scoring 

can be based on any desired mechanism, such as by deter 

mining the most correct guesses made for a set of N epicts. 

Other rules and variations can be used as well. 

In a preferred embodiment the game members can view the 

epicts on their respective client devices (which may be a PC, 

a cellphone, a PDA, a game console, etc.) in accordance with 

image data/game formatting requirements at step 343. As 

noted earlier the participant may also be shown additional 

Supplemental images derived from a main epict to assist in 

guessing the concept in question. For some embodiments of 

the games the teams can be presented epicts that come from a 

common library database 324, while in others the epicts are 

created during game time by the participants themselves. 

Again other modes are contemplated by the present teachings 

and the participants can be given latitude to select a particular 

rule set. 

In those cases where game participants are required to 

create their own epicts on the fly, a variety of different soft 

ware tools can be used to assist them as needed to comply with a game selected concept that they are asked to compose. 

These types of tools are again well-known in the art; one 

example is a software application known as Graffitti that is 

offered on the Facebook site, and which presents a graphical interface for creating drawings for community review. See 

e.g. (h-t-t-p://)apps(dot) facebook(dot).com/graffitiwall/ and 

h-t-t-p://apps(dot) facebook(dot).com/graffiti/ An adaptation 

of this product also allows replay/journaling features and this 

function could be incorporated as well with the present inven 

tion. Thus this product (or something equivalent) could be 

used both for creating and storing epicts for later use (as seen 

in FIG. 3A) or for permitting participants to create content as 

required for a particular game. In the case of using “canned' 

epicts the individual replay components and timing could be 

stored as part of epict record 329 as noted above. For example there may be variable parameters on how quickly the epict is 

drawn. That is the ratio of how much drawing time is involved 

relative to a maximum time allowed for recognition of the 

image can be adjusted based on a desired game difficulty. 

Another example of a drawing tool that may be appropriate 

for the present invention is described in US Publication No. 

2007/0016689 incorporated by reference herein. 

On certain platforms, such as the iPhone, additional sen 

sors could be exploited to render the image in different 

modes/styles. For example the iPhone contains motion sen sors which can be used to determine the attitude of the device. 

The epicts could be encoded to have motion based compo 

nents that are only expressed as the device is moved. For 

example, a picture of a tree could be coded so that it moves in 

response to the phone being moved. This could help denote 

the concept of “wind,” which is otherwise difficult to concep tualize in a static drawing. A picture of a hockey puck could 

be shown in different positions as the device is moved, to 

denote the concept of sliding, and so on. A series of different 

images, representing scenes active at a different orientation/ 

rotation angle of the device, could be used to denote the 

pouring of water, Such as from one container to another. The 

use of such variable position technique would be most advan 

tageous in connection with concepts involving verbs, actions, 

etc. The possibilities are endless in this respect, and other 

examples will be apparent to those skilled in the art. 

The artist participants may similarly be given clues/cues 

and other forms of explicit assistance to help them create the 

pictorial images. The Suggestions may be as simple as pro 

viding additional Semantic word cues that are perhaps easier 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 26 of 34
US 9,192,861 B2 

13 

to depict visually. Alternatively they may be as complex as 

providing specific objects as templates to work from. For 

example for a dog the system may provide a basic body of an 

animal with four appendages, allowing the user to construct 

the rest of the image on his/her own. This technique can help 

solve the case of “drawing/artist block for people who don’t 

know what to use as a starting point. 

As alluded to above, some embodiments may benefit from 

the inclusion of musical clues that are associated with the 

epict, and this feature can be further used during the presen tation to assist in decoding the images. 

At step 344 the responses are logged from the game par 

ticipants (in the form of guesses, epicts, etc.) It will be appar 

ent that in Some cases the raw ratings data feed to be tabulated 

for the game can originate from a standalone architecture (see 

data feed 140) as described in FIG. 1 above. Moreover it 

should be apparent that the participants do not have to be 

physically in the same location to participate as a team. 

Again in FIG. 3A, the participant winners and associated 

prizes are determined at 345 by a decision software module 

using any desired set of rules, i.e., such as by guessing the 

exact word (topic/concept) or a semantic variant as noted 

below. At 346 the game status/score for each participant team 

is adjusted as well in accordance with the rules elected for the 

game. The performance of the participants/teams/epicts/art 

ists is then logged and tabulated at 347. This data can then be 

published as an update or feed if desired to search engines, profiles and other social networking sites. 

The contributors of the content (artists, image sources, etc.) 

are then recognized at step 348 based on the use of the epict 

and/or a rating provided therefore. In embodiments of the 

game where some form of accounting is given to such entities a database can be maintained of credits or funds for each 

contributor. 

As another option the presentation engine can use a col 

laborative filtering or corroborative filtering routine to iden 

tify other epicts or artists that would be of interest to the game participants based on their rating or adoption of an epict/ 

artist. In otherwords, the system can correlate epicts based on 

artist, content tag, score, etc., so that participants can be 

shown other examples of epicts likely to be of interest based 

on their scoring for one or more game epicts. Similarly the 

participants can be given information on other artists likely to 

be of interest based on scores given to a particular contributor 

by other participants. This information could be compiled on 

a topic by topic basis as well for further refinement. Recom 

mendation routines are well-known in the art and can be 

modified to accommodate any and all of the data types refer 

enced above for the epict entries 329. 

Again all of the feedback data above, including scoring can 

be used in the architecture of FIG. 1 above as well. 

The right hand side of FIG. 3B shows other game entry points for the presentation engine 340 captured from a game site (or participant) webpage. For example image challenge 

selection logic 350 is responsible for selecting a particular 

epict to be used within a particular game or widget. To make 

this determination it can examine collective community rat 

ings for one or more epicts, consider contemporary/local 

topics, etc. to select one from database 324. In other embodi 

ments the epicts could be rotated or randomized on a topic basis, an artist/source basis, etc. In still other applications the 

challenge selection logic 350 can make use of geospatial 

information to tailor the content of the challenges to a par 

ticular location where the user happens to be (in the case of a 

portable application). For example, in a baseball stadium the 

challenges could be geared to guessing the identity of players, 

and so on. 

10 

15 

25 

30 

35 

40 

45 

50 

55 

60 

65 

14 

In some applications a CAPTCHA challenge system (not 

shown) could introduce visual content to be reviewed by humans to determine what appropriate tags/concepts are pre 

sented for the image in question. This information can then be 

used to augment a library of challenge test images. For 

example the challenge system could determine that an (inten 

tionally visually distorted) image A is nonetheless identified 

as either X or Y by 90% of the population, thus making is 

suitable for inclusion in a CAPTCHA system. The image 

could be re-presented at various times to different partici 

pants/Internet viewers—and with different levels of degrada tion—until it is no longer recognized by a certain percentage 

of the population. This process could be repeated itera 

tively—with progressively degraded versions of the image— 

as needed to meet the CAPTCHA system's requirements. The 

recognition times could also be measured for each iteration. 

In this manner the present invention can act as a form of 

testing ground for these types of challenge systems. 

An optimized set of image challenges could be presented 

on an iPhone like device as part of a CAPTCHA to gain access 

to accounts, data, computing resources, etc. The user would 

be asked to provide the identity of the image either through a 

keyboard, or, if desired, audibly using speech recognition. 

The latter affords an additional layer of protection against 

would-be fraudulent accesses. 

In still another embodiment, a CAPTCHA challenge for an 

iPhone type portable device (capable of motion/orientations 

sensing of the device) can be based on providing hints or clues 

to a user on how to move or orient such device. That is, the 

CAPTCHA could provide either text on a display, or audible 

instructions to the user on how to manipulate the portable device, such as rotating it to the right, to the left, in a complete circle, picking it up and putting it down N times, shaking it in 

different directions, etc., etc. as part of a CAPTCHA chal 

lenge. For example the current version of the iPhone has a 

built-in compass; the user could be asked to orient the needle 

to point to a particular location (N, NE, E, SE, S, SW, W. NW 

and so on.) In other instances the user may be asked to move 

the device so as to cause some action in a scene, such as to 

make water pour from a cup, and so on. Thus the manipulation 

of objects within the screen can be done as needed in a 

challenge based on sensing the movement of the device. 

In other cases the CAPTCHA can include a visible image indicating the movement to be imparted to the device. Such as 

showing a person rotating it 90 degrees, picking it up and 

down, moving it in a FIG. 8 pattern, etc. Again the image 

could be progressively revealed or degraded if desired. 

Skilled artisans will appreciate that there are many options for 

the physical manipulations and the invention is not so limited. 

The touchscreen on an iPhone like device could also be 

integrated into the challenge so that the user is required to 

physically touch or manipulate icons or other objects to 

achieve some result. For example the user could be asked to 

arrange a set of letters in a sequence to spella particular word 

in a particular location on the screen. Alternatively the user 

could be asked to orient an object into its correct position by moving it with their fingers such as described generally in the 

article “What's Up CAPTCHA? A CAPTCHA Based On 

Image Orientation” by Gossweiler et al. (WWW 2009, Apr. 

20-24, 2009, Madrid, Spain) incorporated by reference 

herein—or by moving the device. Those skilled in the art will 

appreciate that other physical manipulations could be 

employed, including blowing into the phone's microphone 

(as is done in Some games), etc. The challenges could be used, 

as noted, for accessing computing resources, accounts, etc., 

and even for applications such as unlocking/accessing the 

portable device (i.e., like a secret handshake). 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 27 of 34
US 9,192,861 B2 

15 

Returning to FIG.3H, an example of the types of degraded 

images is shown therein. The same image as FIG.3G is shown 

with different degrees of pixelation, again using the basic 

operational tool available with Adobe Photoshop Elements. 

At one level of extreme pixelation (created using a variable 

colorhalftone filter feature) the image is not recognizable; but 

at lower levels the image can easily be distinguished as a 

bunkbed as in the original. Similar effects can be achieved 

using filters such as crystallize, meZZotint, mosaic (and sev 

eral image noise and threshold options) all of which can be 

experimented with using routine skill to devise samples which are recognizable by humans but (preferably) not 

machines. For example FIG. 3J shows an original JPEG 

image that has been subjected to various levels of noise fil 

tering, either through removing noise (at different pixel lev 

els) or addition (at differing percentages). It can be seen that 

at some thresholds a human would have a difficult time per ceiving the object in the image, but there are clearly levels at 

which the image is decipherable. The images therefore could 

be subjected to different levels of pixelation/signal process 

ing treatment on the fly, or they could be stored as Successive 

original files in the different degraded forms for ease of com 

putation/presentation. 

At step 351 the selected epict is presented on the game 

website (or member widget) in accordance with a set of game rules applicable to Such site. User ratings and responses are 

collected in a similar fashion at step 344 as noted before for 

the team embodiment. 

In this manner the presentation engine 340 can accommo 

date and incorporate content contributed by participants 

across a wide variety of input platforms and game environ 

ments. All of the aforementioned steps in FIG. 3B can be 

implemented using one or more software routines using ordi 

nary skill in the art based on the present teachings. 

FIG. 3C illustrates a preferred embodiment of a pictorial 

game tabulating module 330. This module is generally responsible for compiling data and scoring for the epicts, participants, teams, etc., publishing results and accounting 

for results of epict games. 

As seen in seen FIG. 3C, a routine assigns each pictorial 

game an ID 331 to identify it uniquely for tracking purposes. 

For each game, a set of epictIDs 333 and participant IDs 334 

are maintained. The epict scores and participant scores are 

further stored by a routine in respective databases 332 and 

332. A routine 335 also calculates epict and participant 

scores. The scoring can be based on any number of criteria, 

Such as in the case of an epict, the fact that it was recognized, 

the amount of time required, what rating was given to it, and 

the mode of presentation used (static or dynamic replay). The 

participants can be scored in a similar fashion, such whether 

they successfully recognized an epict (by identifying one or 

more correct concepts), the amount of time required by the 

participant to make the guess, what rating was given by the 

participant and the mode of presentation used (static or 

dynamic replay). These are but examples of course, and other 

types of data could be collected as needed. 

A separate routine 336 also calculates author/source scores 

to be stored in an artist database 332". This scoring can be 

based on the same parameters described above, so that, for 

example, artists are rewarded based on positive ratings given by participants. 

The results from the pictorial games (whether they be home 

based, or based on team games, or derived from Internet sites, 

widgets, etc.) are published as desired by a routine 337. The 

results may include other aggregate calculations, including image difficulty rankings, image ratings, group image ratings, 

author ratings, participant ratings, team scores, game scores 

5 

10 

15 

25 

30 

35 

40 

45 

50 

55 

60 

65 

16 

and similar data. Again the results can be made available to 

any of the platforms from which the game originates, includ 

ing an online gaming site or social network profile, home 

based console system, etc. 

The format of the data records maintained in the databases 

is shown generally in FIG. 3C as well. For example for an 

image database 332, an epict record 338 may include the 

following fields: 

Image ID (as noted above) 

Presentation mode (whether it was displayed statically or 

using a slow motion replay) 

% identified within time (number of participants who cor 

rectly guessed the concept of the epict divided by total 

viewers) 

Mean recognition time (RT) (the average and/or mean 

times for recognizing the epict) 

Standard deviation RT (basic statistical databased on rec 

ognition times) 

Ratings Score (average score across all participants) Difficulty Rating (numerical rating given by participants to 

indicate how hard it was to decipher the epict) Demographic data (information about the participants) 

Note that the above is an example of an aggregate record. In 

addition some of the above fields (where applicable) may be 

compiled on an individual basis, with each game/rating of the 

epict given a unique record. Other fields could be maintained 

in Such case within database 332 or an associated database, 

for example the actual time may be recorded for each epict 

evaluation. The types of data collected could include other 

parameters as well. 

In addition for the participants, an aggregate record 338 

stored in database 332 can include: 

Participant ID (a numerical designation of any appropriate length); 

Images Perceived (which epicts were evaluated) 

Images Recognized (which ones were successfully identi 

fied) 

Mean Recognition Time (basic mathematical computation 

of times required by participant to review epicts) 

SD (RT) (as above) 

Ratings Score (as above, on an epict basis) 

Favorite Authors (list of artists by ID) 

Image IDs authored (list of epicts contributed identified by epict ID) 

As above, note the data/fields are for an aggregate record. 

Some of the above fields (where applicable) may be compiled 

on an individual basis, with each epict rated given a unique 

record by score, actual review time, etc. and are maintained 

within database 332 or an associated database. The types of 

data collected could include other parameters as well. 

Finally, for the individual contributors/artists, the follow 

ing type of aggregate record 338" could be maintained within 

database 332": 

Author ID (as above) 

Image IDs (as above, indicating which epicts created by 

this artist) 

Average Images Recognition Score (compiled across all 

epicts) 

Mean Recognition Time (across all epicts) 

SD (RT) (across all epicts) Author Score (an average rating provided by participants) Working on image challenge (indicating what contest they 

are working on) 

As above, note the data/fields are for an aggregate record. 

Some of the above fields (where applicable) may be compiled 

on an individual artist basis, with each artist having a unique 

record by epict rated, so that an actual epict score, review 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 28 of 34
US 9,192,861 B2 

17 

time, artist score etc. are maintained within database 332" or 

an associated database. The types of data collected could 

include other parameters as well. 

While not expressly shown, it should be noted that team 

scores/data can be maintained as well in essentially the same 

fashion as for individual participant scores noted above. The 

data for such teams can be published as desired as well to 

appropriate authorized entities. 

Again it should be noted that these data fields may be varied 

from application to application. It may be desirable to com 

pile averages rather than means in some cases, and so on. The 

particular parameters will vary depending on the desired 

functionality. All of the above data can be published in appro 

priate form by routine 337 in any number of different loca 

tions, formats, etc., for participant enjoyment. 

FIG. 3D illustrates a preferred embodiment of a pictorial 

game synthesis-Suggestion module 360. This feature of the 

invention is responsible for assisting game participant artists 

to compose epicts, and to facilitate presentation of so-called 

preformed/canned epicts to participants. 

As seen in this figure, based on a game ID 331 a determi 

nation is made whether the game is to be run in an automatic 

or manual mode. In the former case image/epict IDs 332 are 

obtained automatically from a database 324, while in the 

latter case a manual mode 361 requires a game participant to 

create the epicts in accordance with the details noted above 

(see FIG. 3B). 

In either event the game participants have the option at Step 

362 to elect a supplemental presentation. This feature, as 

noted above in connection with FIG.3A, allows for additional 

secondary images and other cues (from database 332) to be 

retrieved at 364 and presented as part of the reference epictat 

step 365. In a manual mode this information can be hidden 

from the participants and shown only to the artist to assist the 

latter in formulating appropriate images. If the Supplemental 

option is not elected the image data is presented at 363 in 

accordance with any component/timing information associ 

ated with the epict. It should be noted that the automatic mode 

noted here would be the default option for any epict games/ presentations made in connection with a website/web profile implementation since artists are typically associated with 

team play. 

In the event a manual mode is used at 361 for the game the 

artist can nonetheless invoke a Suggestion option for the con 

cept at 366. If the user selects such feature the game play 

routes through the same steps above, except that in addition to 

the Supplemental option features the designated artist is given 

specific Suggestions on images that could be drawn to com 

municate the concept in question. If the option is not chosen 

the image is simply presented in a normal picture mode at 

367. 

All of the aforementioned steps in FIG. 3D can be imple 

mented using one or more software routines using ordinary skill in the art based on the present teachings. 

Next, FIG. 3E illustrates a preferred embodiment of a 

pictorial game speech recognition module that can be used in some embodiments of the invention. While this module is 

depicted in the context of a pictorial game it will be apparent 

to skilled artisans that it can be used in connection with any number of online games. 

As noted above in FIG. 3B, participants guessing on the 

conceptembodied in the epict typically provide input on Such 

using a keyboard or mouse. Another option however is the use 

of speech data from the participants. This feature has the 

advantage of most duplicating the prior art game experiences 

and frees the participants from the task of being distracted 

while analyzing the image data. Other benefits are apparent as 

10 

15 

25 

30 

35 

40 

45 

50 

55 

60 

65 

18 

well, including the fact that the participant Voice prints col 

lected by the module can be used for authentication and access verification as well. 

The recognition module 370 executed two basic proce 

dures: a set up mode 375 and a game mode 380. In the first 

mode, a routine Solicits participant Voice inputs at Step 376. 

This can be done in any conventional fashion, including by collecting test words/phrases, examining for prosody, and 

accounting for background noise attendant at the partici 

pants site. In situations where more than one participant is 

present at the client device for presenting guesses a “crowd 

mode can be employed. Furthermore depending on the pro 

cessing resources available to the participant they may elect to 

have the bulk of the speech recognition performed by a 

remote server associated with the pictorial game system. 

Other options will be apparent to those skilled in the art. 

At step 377 a set of voice prints/templates are thus devel 

oped for each of the individual team members in the case of 

a solo game of course, only the user invoking the game need 

provide a template. If a number of players are present at a 

particular location, it may be desirable to provide some form 

of amplitude compensation/normalization to account for dif 

ferent participant speaking Volumes. 

An optional step 378 can be used to confirm that the system is accurately discriminating between individual participants. 

Each participant would be permitted to speak a test phrase, for 

example, and the system would indicate which person it 

believes gave the utterance. In the event the system needs 

further training the procedure could be repeated. During a game mode 380, the speech recognition module 

receives metadata for one or more epicts at step 381. This 

metadata includes a number of content tags describing the 

object of the epict, and may include a number of different 

semantically related words. 

Based on these “answer responses an appropriate gram mar and/or customized acoustic model can be loaded for the 

speech recognizer at step 382. At this point therefore the 

speech recognizer need not be capable of recognizing Sub 

stantial numbers of words. In fact it need only focus on a 

handful of words, and thus it can be substantially reduced in 

computationally complexity because it can ignore all other 

inputs that do not match one of the appropriate answers. 

During step 383 the SR module listens for answers input by 

the various game participants. In the case of team members 

distributed across multiple locations the data input to the SR 

is thus received from multiple sources. In such instances it 

may be desirable to allocate a dedicated SR resource to each 

channel to improve performance. The SR module then rec 

ognizes and time-stamps each accurate recognition at step 

384 to give appropriate attribution to the first participant accurately guessing the epict concept. 

Nonetheless even in “crowd scenarios where the partici 

pants are at a single location, recent advances have made it 

possible to make reasonable resolutions of individual speak 

ers who are speaking simultaneously. See e.g. h-t-t-p://tech nology(dot)newscientist(dot).com/article/dn14105-robot 

asimo-can-understand-three-voices-at-once(dot)html 

incorporated by reference herein. By using a similarly con 

figured console or game platform with appropriately located 

microphones a SR module could distinguish between mul 

tiple human speakers articulating their responses at the same 

time. Other known tracking techniques for identifying the 

correct speaker can also be employed. Such as by using video 

and other input. 

Upon receiving a correct response the routine would iden 

tify a winner at step 385. This feature allows the participants 

to express their answers rapidly and in free form without the 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 29 of 34
US 9,192,861 B2 

19 

need for slower, more cumbersome data entries. It is expected that this form of input would be far more preferable in 

embodiments of the invention where individual players desired to play the game over a portable device, Such as an 

Iphone or similar device that includes speech capability built 

in. In Such instances the grammars/dictionaries can be down 

loaded directly to the device (since they need only pertain to 

the universe of words associated with the concepts) to allow 

untethered (non-network) play in situations where the recog 

nizer functions can also reside on the portable device. 

FIG. 3F illustrates a preferred embodiment of a process employed by a pictorial game advertising module 390. This 

process basically can be an adaptation of a conventional key 

word auction based algorithm such as the type used by Such 

entities as Google, Yahoo!, Microsoft, etc. In this instance, 

therefore advertisers are permitted to bid on a keyword or 

topic basis at step 391. In the present invention the keywords/ 

topics are preferably based on the epicts and related metadata 

(concepts/words and semantic relatives) stored in databases 

324, 324 (FIG. 3A). Thus the advertising stock 392 is pref erably served on an epict by epict basis at step 393. In this 

fashion a content based matching system can be implemented 

between advertising stock and the content of the epicts. 

Since it may be undesirable to serve ads which have content 

that may communicate (and thus spoil) the epict recognition 

process by players (since it may be related to the topics/ 

concepts of the epict), it is possible to switch off the serving 

of ads as noted at step 394 until the epict has been recognized 

or the recognition time has passed. This would avoid the 

problem of players being given hints as to the identity of the 

epict. Other techniques could be used. At step 395 the ad is 

served based on the advertiser's stock 392. It should be noted 

that in Some cases it may nonetheless be desirable to incor 

porate advertising while the epict is being drawn as a mecha 

nism for assisting the recognition process. In this embodi 

ment, the ad is in fact assisting the player to solve an aspect of 

the game, a feature unlike other advertising modules used in 

the art. 

This concept could be further enhanced by integrating 

other user contributed content directly into the ads. For 

example one mode of the game, as noted above, involves a 

Suggestion component intended to assist players determine 

epict concepts. An ad which directly incorporates an epict 

determined to be related to (or Suggestive of) the epict in 

question could be used to present the Suggestion explicitly on 

the participants screen while they are playing the game. 

This collaboration between advertisers and artists can be 

extended at step 396 so that particular artists/epicts can be 

essentially “sponsored.” For example, as seen in FIG. 2 (ep 

icts 294) and FIG. 4 (epicts 440) the identity/stock of epicts 

chosen for presentation within the web pages can be deter 

mined by advertisers who match their advertising Stock to one 

or more epicts that they want to be associated with. In this 

respect, therefore, the epicts effectively act as web assets 

(akin to a web page in an Adsense campaign) that advertisers 

can bid to be associated with. By allowing advertisers to bid 

on specific assets (instead of queries, or webpage content) the 

invention provides another angle for monetization of Such 

assets and opportunities for advertisers. 

The advertisers could in some cases be allowed to intro 

duce their own proprietary/customized epict content as well 

(to the image library) to enhance the appearance of the game 

and increase their exposure. They may also contribute their 

own list of concepts, topics, etc. to be used as part of the game. 

Combined with the dynamic serving of ads related to an epict 

the invention allows for advertising to be used as an integral and dynamic component of an interactive game. 

5 

10 

15 

25 

30 

35 

40 

45 

50 

55 

60 

65 

20 

FIG. 4 depicts a preferred embodiment of a pictorial game 

website 400 that embodies aspects of the present invention. 

Specifically, this figure shows a an example of a main web 

page that would be seen by Internet users accessing a game 

site. It should be noted that many conventional aspects of Such 

types of pages have been omitted to make the description of 

the invention clearer. 

In particular, the game site offers users a chance to set up and 

initiate a pictorial game with all the features and functions 

described above in connection with FIGS. 2 and 3A to 3E. To 

wit, different portions of Such page as presented within a 

client GUI interface include selectable URLs which invoke 

routines to permit the user to: 

Start a team game (option 410)—allows for a team to start 

a game against another team using the procedures noted 

in FIGS. 3A-3E: 

Start a solitaire challenge (option 411) allows for an indi 

vidual to start a game using the procedures noted in 

FIGS. 3A-3E; 

Submit a challenge word (option 412)—used for allowing members to offer concepts/words to be expressed picto 

rially by other members; 

Watch a game (option 415)—allows a member to act as a 

form of voyeur and observe a team game by topic, locale, 

team name, etc., 

Upload an epict (option 420)—allows a user to submit their 

own pictorial content and descriptor tags for the enter 

tainment of the community; View top N lists for the community/site (option 425)— 

allows users to see the highest rated epicts, the most 

difficult epicts, team ratings, artist ratings, participant 

ratings, game scores, etc.; the difficulty of epicts can be 

determined by computing a total number of times the 

epict has been presented versus how many time it has 

been recognized; artist ratings can be similarly com 

puted; the most difficult concepts to recognize (either 

individually or by topic) can also be compiled 

Enroll a team (option 430)—allows a user to set up a new 

team for play on the site; 

Find teammates (option 431)—allows user to find other 

persons to be part of a team; 

Find/rate an artist (option 432)—allows a user to search for 

specific artists by name, topic, concepts, etc. and rate 

them if desired 

Search/Rate epicts (option 433)—allows a user to search 

for specific epicts by artist, topic, concepts, etc. and rate 

them if desired 

Review/guess on challenge epicts (option 440)—allows 

users to pick specific epicts/words to evaluate and play 

with in accordance with game rules. 

Again not all of these options are necessary, and other 

options could be presented and offered as well, and the par ticular implementation will vary according to game specifi 

cations and requirements. 

Some embodiments of the pictorial game can include a 

rating module 500 as seen in FIG. 5. The purpose of this 

module is to allow teams 510 and individuals 515 to access 

and play with a set of benchmark epicts (S1, S2, ... Sn) 520 

and evaluate their performance relative to such standard for 

competitive purposes. The benchmark epict sets can be 

selected by human editors, or automatically by rating module 

500 by evaluating ratings/performances of epicts compiled by 

tabulating module 230. 

In the former case, a human editor may select a number (X) 

of epicts from database 324 based on a ratings list of X 

different topics. In the latter case the identity of the individual 

epicts within the sets is determined by reference to some 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 30 of 34
US 9,192,861 B2 

21 

threshold rating for score, difficulty, etc. In either scenario the 

epicts are preferably selected to expose participants to a cross 

section of the available inventory. 

The participants then work through one or more of the epict 

sets (e.g. S1) and then have their times/accuracy evaluated by a routine 530 Such as that described above for FIG. 3A. Their 

scores, and the ratings (if provided) for the epicts are then 

compiled by a routine 540 that, again can be similar to that 

already discussed above. 

This aspect of the invention therefore allows members (and 

teams) to evaluate themselves relative to some test bench 

marks for competitive recognition. The results can be pub 

lished, as noted earlier, for the entertainment of the overall 

community, to individual profiles, the main game webpage, 

or other Social networking sites. 

FIG. 6 illustrates a preferred embodiment of a variant of a 

rule set 600 used in a pictorial game. Basically this figure 

describes a format that can be employed in Some variants of 

the game to make it more interesting for Some participants in 

team play applications. 

In this embodiment a challenge word/concept is selected 

for a first team (A) by a routine at 610. This may be done 

randomly, by topic, or by another rule choice opted by the 

members. 

At 615, and without letting the member of team. A see such 

word, the members of team Bare allowed to visualize various 

epict alternatives selected by the system for presenting Such 

concept. The alternatives may be selected randomly, or, alter 

natively, may be selected based on a variety of Scores/ratings 

for Such epicts. The latter may be used, for example, to pro 

vide a reasonable cross section of challenges ranging from 

easy to difficult. 

The members of team B can then (optionally) vote to 

eliminate (or select) one or more of the alternatives to be used 

by the system for team A at step 620. In some variants of the 

games the team members may be allowed to see the difficulty ratings of the epicts as measured by the performance of other 

individuals/teams. 

During step 625 the final epict is then selected for team A 

based on the selection made by team B. In some cases it may be desirable to incorporate a randomizer at 630 to select in 

situations where the selection is not specifically defined. 

At 635 the module further allows an option that gives team 

B yet another opportunity to impair or handicap team A's 

chances of Success in decoding the epict. In particular the 

routine arbitrarily selects aspects/components of the image 

which can be selectively removed from the epic to make it 

more difficult to decipher. Thus team B is again presented 

with degraded variants of the epict to be shown to A, and they 

then make a final selection which is presented at step 640. 

As an ancillary feature the teams (or individuals for that 

matter) can make use of a prediction component 650 which 

allows for review of epicts and their associated concepts to 

provide a prediction or vote as to whether Such epict was 

recognized by Some (system definable) percentage of the 

community, and/or a prediction of what rating such epict 

garnered. For example the module may ask the participant a 

simple yes or no question about whether the epict was recog 

nized within the allotted time by more than 50% of the popu 

lation? (or more than 75%, etc.) The module can then further 

request the participants estimate of the rating given by the 

community to the epict. The participants can then see how 

close or far off their predictions are for the epict in question. 

This aspect of the invention therefore further allows partici 

pants to glean insights into the collective thinking of their 

fellow gamers. 

10 

15 

25 

30 

35 

40 

45 

50 

55 

60 

65 

22 

These rule variants are not necessary of course, but provide 

an additional element of play that may be attractive to some 

participants. Other examples of such variants will be apparent 

to skilled artisans from the present teachings. FIG. 7 depicts 

a preferred embodiment of a game screen/interface 700 used 

by a participantina pictorial game. This screen may be part of 

an interface that is presented within a conventional browser 

(or similar Internet browsing utility for a handheld device or 

game console) to interact with game websites. This same 

interface may be used as well, as noted above, in connection 

with user profile pages and the like maintained at Social 

networking sites. 

Within the interface the participant is shown an electronic 

canvass 705 in which the epict is presented visually. Again, 

the epict may be shown all at once, or it may be staggered as 

noted earlier. The canvass is preferably scaled in size so as to 

allow participants to visualize the image without noticeable 

compression, artifacts, etc. 

At portion 710 the user is allowed to enter his/her guess of 

the object, by keying in an answer, selecting an answer by 

mouse, or speaking (in the case of a speech recognition 

embodiment). Other forms of input could be used as well 

(touch for example). 

Section 715 of the interface informs the user of his/her 

progress in the game. For example the game may consist of a 

set number of epicts that must be completed to satisfy a game 

objective, and a time factor may be introduced as well. The 

participants score is also displayed at 716. 

A time remaining for guessing the epict is shown at 717 in 

any convenient visual form (digital, analog, hourglass, etc.) 

and in accordance with user preferences for the interface. In 

fact, preferably all elements of the interface can be scaled, 

arranged and formatted in accordance with a desired aesthetic 

of the user and to give a sense of customization. 

The category selected by the user is identified at region 718 

of the interface. 

At portion 719 of the interface the game may display the 

answer if the user has selected it correctly (a semantic equiva 

lent may be acceptable). In some instances it may be desirable 

to “recognize' partial correction portions of a users answer, 

so that if he/she inputs “cruise liner the invention would 

nonetheless acceptand display the “liner part of the provided 

answer as acceptable. This acts as a further clue for the par 

ticipant that the prefix must be different. Other variants will 

be apparent to those skilled in the art. 

The stock of epicts presented within interface 700 can 

come from multiple sources, including artist Submissions 

720, and user submissions 730. These are differentiated pri marily on the basis of how the origin of the image, rather than 

any significant technical basis, since both can be accessed 

from a database 324 (FIG. 3A). Nonetheless the process by which epicts come into existence can be used to augment the 

accompanying metadata and presentation mode for the par ticipant. 

In a first case, artist submissions 721 preferably consist of 

images created by original artists using one of the drawing 

tools mentioned above. In such cases the original artwork 722 

is selected, and then processed by a routine 723 in the manner 

noted above for FIGS. 3A and 3B. Thus the image is prefer 

ably presented all at once—in a static mode—or using some 

form of timing, as in a replay mode based on using an actual 

artist rendering time. The latter could be accelerated or 

slowed depending on other game parameters. 

In other embodiments the epict could be presented in a 

degraded form (as noted above) that gradually improves with 

time, based on a variable number of iterations, time, etc. For 

example the image could be given incremental improvements 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 31 of 34
US 9,192,861 B2 

23 

every 10 seconds until a final image is reached. The partici 

pants score could be weighted according to how quickly they 

decoded the image (i.e. what level of degradation). 

In similar fashion user submissions 731, which may be 

basic JPEG images, can also be selected at 732 as a form of 

epict. This type of contentis perhaps easier to contribute since 

many users have electronic cameras, tools for uploading images, online picture accounts (Shutterfly, Flickr, etc.) that 

can be tapped into for content. The images are preferably 

pre-treated (see discussion for FIGS. 3A and 3B above) to 

render them into more artistic form, and/or to create different 

levels of degradation as noted above. Alternatively, in some 

embodiments it may be possible to impart this type of visual 

appearance on the fly within the interface, since Sufficient 

time may be given between iterations to permit this degree of 

processing. In any event the images are presented, again, 

using one of the modes above, except of course that in Such 

instances an original “artist' mode would not be possible in 

the case of a photograph instead of a drawing. Nonetheless it 

is possible that an artistic rendering could be mimicked in 

Some cases using additional image processing software. 

For some applications it may be desirable to simply divide 

the image file into physical regions, and present different sets 

of pixels within regions on a random basis. As an example, a 

set of N different iterations could be used. At the first itera 

tion, X% of the pixels are revealed. At the seconditeration, an 

additional set of Y% of the pixels for each region are revealed, 

and so on. This would give the image the appearance of 

randomly filling in with time. Other variants are of course 

possible and useable with the present invention. 

As with the other embodiments above, the participant can 

also opt to use musical hints during the presentation of the 

epict. In addition a hint board 750 may also be displayed 

within the interface to give further clues to the participant. As 

noted earlier, the hint board may integrate advertising as well, 

and the adstock itselfmay form part of the visual clue, by way 

of images, text, music/audio, etc. Because the participants 

focus may shift within the interface to pickup and detect such 

clue, the chances of an ad being perceived (and thus selected 

eventually) by the user increase dramatically. While the time 

is running for the epict the hint board images could also be 

changed if desired. In this respect the stock could be rotated 

randomly, or based on Some programmed logic. In an 

embodiment where the epict deciphering is timed, the user's 

selection of an ad within the hint board can be used as a 

control signal to stop the game clock so that the participant is 

not penalized for viewing and interacting with an advertise 

ment. For applications where timing is not used this would not 

be necessary of course. 

The advertisers can then also be given options forbidding 

on what times their ads are presented within the hint board 

during game play. For example the right to present a visual 

hint/ad during an earlier portion of the game could be auc 

tioned/priced at a different price than the right to offer the 

same hint/adata second time later in the process. An end-of 

epict time option can also be offered, to allow an advertiser to 

occupy the interface for alonger period at the end of the game. 

Other options can also be offered, such as the right to opt in or 

out of an epict in which the participant was successful or not. 

All of these factors can be used to adjust an appropriate auction and compensation scheme. 

The hint board ads could also be tailored based on a per formance level achieved by the participant, and thus a pre 

dicted mood of the user. The user's mood could also be 

determined with reference to prosodic elements in a speech embodiment. For example a first type of ad might be shown if 

the user was successful in the current/priorepict, compared to 

5 

10 

15 

25 

30 

35 

40 

45 

50 

55 

60 

65 

24 

a second type of ad if the user was unsuccessful. The inven 

tion can monitor the progress and performance of the user to 

increase (or decrease) the level of visual assistance through 

the hint board as needed to maintain an entertainment level 

for the participant. Most players will not want to play a game 

that is too easy or too hard, and this balance can be struck in 

part by adjusting the types of Supplemental hints given within 

the interface. Again, too, the advertisers can participate and 

assist with Such process by providing ad stock/associated 

epicts of various levels of explicitness for a particular con 

cept. The advertisers may also opt to sponsor or present the 

musical hints and cues associated with the epict. The adver 

tisers can then also be given options forbidding on different 

levels of visual/audio hints, so that the right to offer a first 

level of visual assistance is priced at a different price than the 

right to offer a second higher/lower level of assistance. Since 

it is more likely a participant will have a positive association 

with a hint that helps them solve the visual puzzle, this factor 

can be used to adjust an appropriate compensation Scheme. 

While some conventional online ads are presented in the 

form of a game (e.g., they require the user to interact with 

them, Such as using a mouse) it can be seen that the hint board 

750 instead behaves as a form of additional artist/participant 

for the game the user is engaged in. Those skilled in the art 

will appreciate that this feature may be used advantageously 

in other game environments in which it is desirable to impart 

clues/cues to a game player using an interactive ad. 

While not shown explicitly the game can of course use and 

exploit demographic and other profile/behavioral data col 

lected for participants to enhance game play. The epict data 

base 324 of artist and author contributions can be made avail 

able to news aggregators, search engines, etc., so that relevant 

content is presented to web surfers looking for pertinent infor 

mation on particular concepts. 

As shown in FIG. 8, in some embodiments a search engine, 

blog, newsfeed or news site 800 may offer accents or other 

adornments of a page based on epicts contributed by an artist 

community. For example, a Google news story about a per 

son/place may be coupled with one or more artist caricatures 

of such person/place. 

Similarly, in a search engine page 810, a person looking for 

the Tower of London may be shown an epict 815 for such as 

part of the search results page. As with the above embodi 

ments, the epict 815 (preferably compressed to thumbnail 

size) may be combined with an ad 820 to form a new ad 825, so that the attractiveness of ad stock can be increased for a 

general searcher using a search engine, and the chances of a 

click-through increased by virtue of the additional visual 

aesthetic. This mechanism further allows artists to tap into 

and benefit from advertising dollars as they can be rewarded 

for their success in creating incremental benefits over con 

ventional text only ads. 

Search engine operators such as Google can thus also 

maintain a stock of epicts/artists that can be bid upon with a 

conventional auction engine 830 by advertisers 835, much in 

the same way keywords are bid upon in an Adwords cam 

paign. The performance of individual epicts in clicks and 

conversions can be tracked as with any other keyword. This 

again would allow for a more efficient marketplace to 

acknowledge and recognize Successful advertising copy con 

tributed by artists. 

The same advertising and epict coupling can be used of 

course in an Adsense campaign as well, so that members of an 

advertising network can have pages 810 Supplied by a com 

bination of text and images proven to be attractive to web 

Surfers. Since AdSense campaigns are based on the content of 

the webpage 810 in question, a similar matching scheme 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 32 of 34
US 9,192,861 B2 

25 

could be done here. If desired, the ad and epict auctions could 

be de-coupled. That is, both would be evaluated separately for 

inclusion on a page, so that one matching engine would 

determine if the advertiser's ad is appropriate for the page, while a separate pictorial matching engine could use a sepa 

rate algorithm to match metadata for the epict and thus select 

it that way. For example a site devoted to fishing may be 

served with Adsense ads on fishing Supplies, while the epict 

for the page may be in the form of a fishing boat, a hook, a boy fishing off a pier, etc. Other examples will be apparent to 

those skilled in the art. 

Consequently epicts can be the Subject of auctions based 

on keywords in queries, by which an advertiser first wins the 

right to present within the page, and then presents an epict 

along with the ad. In addition to this the advertiser may separately have to bid on the right to use aparticular epict with 

an ad, so this could be part of a separate auction that is 

conducted after the first ad auction. Alternatively epicts can 

be presented based on the content of the pages, the search 

results, or the ads. 

Other content sites and programs could access the epict 

database in question as well depending on a desired visual 

effect. Some music providers, such as Itunes, may offer 

accompanying epicts for their songS/albums for example. In 

auction sites, members who do not otherwise have visual 

images for an offering can elect to use an epict instead at 

nominal cost. Members of Social networking sites may elect 

to adorn their profile pages with static or rotating epicts 

devoted to particular set themes, or dynamically based on 

topics of discussion presented within a discussion wall. The 

invention can thus mimic the behavior of conventional ad 

serving systems which present ads based on the content of a 

web page. In other words, the content of a web page can be 

analyzed to determine one or more related topics which are 

then matched to one or more topics associated with image in 

a database of images/topics. 

All of the above schemes could be implemented under a 

royalty scheme that rewards the artists/authors in question. 

It will be understood by those skilled in the art that the 

above are merely examples and that countless variations on 

the above can be implemented in accordance with the present teachings. A number of other conventional structures/steps 

that would be included in a commercial application have been 

omitted, as well, to better emphasize the present teachings. 

It will be apparent to those skilled in the art that the mod 

ules of the present invention, including those illustrated in the 

figures can be implemented using any one of many known 

programming languages Suitable for creating applications 

that can run on large scale computing systems, including 

servers connected to a network (such as the Internet). The 

details of the specific implementation of the present invention 

will vary depending on the programming language(s) used to 

embody the above principles, and are not material to an 

understanding of the present invention. Furthermore, in some 

instances, a portion of the hardware and software will be 

contained locally to a members computing system, which 

can include a portable machine or a computing machine at the 

users premises, such as a personal computer, a PDA, digital 

video recorder, receiver, etc. 

Furthermore it will be apparent to those skilled in the art 

that this is not the entire set of software modules that can be 

used, or an exhaustive list of all operations executed by Such 

modules. It is expected, in fact, that other features will be 

added by System operators in accordance with customer pref 

erences and/or system performance requirements. Further 

more, while not explicitly shown or described herein, the 

details of the various software routines, executable code, etc., 

10 

15 

25 

30 

35 

40 

45 

50 

55 

60 

65 

26 

required to effectuate the functionality discussed above in 

Such modules are not material to the present invention, and 

may be implemented in any number of ways known to those 

skilled in the art. Such code, routines, etc. may be stored in 

any number of forms of machine readable media. The above 

descriptions are intended as merely illustrative embodiments 

of the proposed inventions. It is understood that the protection 

afforded the present invention also comprehends and extends 

to embodiments different from those above, but which fall 

within the scope of the present claims. 

What is claimed is: 

1. A method of providing a motion and/or orientation 

based Completely Automatic Public Turing Test To Tell 

Humans And Computers Apart (CAPTCHA) challenge on a 

portable computing device comprising: 

a. automatically electing a manipulation challenge for a 

user with the portable computing device, wherein said 

manipulation challenge includes a set of physical 

manipulation instructions for the portable computing 

device for the user; 

b. automatically processing sensory data associated with 

user movements and/or manipulations of the portable 

computing device; and 

c. automatically determining if said user movements and/ 

or manipulations from the user meet a required threshold 

for satisfying said set of manipulation instructions to 

qualify said user as a human. 

2. The method of claim 1, wherein the set of physical 

manipulation instructions for the motion and/or orientation 

based CAPTCHA challenge are presented in an electronic 

image. 

3. The method of claim 2, wherein said electronic image is 

presented in degraded form. 

4. The method of claim 1, wherein the set of physical 

manipulation instructions for the motion and/or orientation 

based CAPTCHA challenge are presented in audible form by 

a text to speech engine operating on the portable computing 

device. 

5. The method of claim 1, wherein said set of physical manipulation instructions requires the user to move the por table computing device so as to manipulate objects presented 

in an interface. 

6. The method of claim 1, wherein said set of physical manipulation instructions require the user to orient a needle 

of a virtual compass in a specified direction. 

7. The method of claim 1, wherein said set of physical manipulation instructions require the user to speak or blow 

into a microphone. 

8. The method of claim 7, wherein a speech response from 

the user is processed and used to determine if it matches a 

human Voice or a machine Voice, and further whether it con 

tains a description of an image presented in an interface of the 

portable computing device. 

9. The method of claim 1, wherein the motion and/or ori 

entation based CAPTCHA challenge is embodied in com 

puter readable form and implemented as a routine executing 

on the portable computing device. 

10. The method of claim 1, wherein the user is required to 

move the portable computing device so as to cause some 

action between objects in a scene presented in an interface of 

the portable computing device. 

11. The method of claim 1, wherein said manipulation challenge is used for unlocking and/or accessing the portable 

device. 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 33 of 34
US 9,192,861 B2 

27 

12. A method of providing a touch-based Completely Automatic Public Turing Test To Tell Humans And Comput 

ers Apart (CAPTCHA) challenge on a portable computing device comprising: 

a. automatically electing a manipulation challenge for a 

user with the portable computing device, wherein said 

manipulation challenge includes a set of physical touch 

instructions for the portable computing device for the 

user, 

b. automatically processing touch sensory data associated 

with user movements and/or manipulations of objects within a graphical interface of the portable computing 

device; and 

c. automatically determining if said user touch sensory data 

meets a required threshold for satisfying said set of 

physical touch instructions to qualify said user as a 

human. 

13. The method of claim 12, wherein the set of physical 

touch instructions for the touch-based CAPTCHA challenge 

are presented in an electronic image. 

10 

15 

14. The method of claim 12, wherein the set of physical 2O 

touch instructions for the touch-based CAPTCHA challenge 

are presented in audible form by a text to speech engine operating on the portable computing device. 

28 

15. The method of claim 12, wherein said set of physical 

touch instructions require the user to arrange a set of letters on 

the graphical interface. 

16. The method of claim 12, wherein the touch based 

CAPTCHA challenge is embodied in computer readable 

form and implemented as a routine executing on the portable 

computing device. 

17. The method of claim 12, wherein the user is required to 

move the portable computing device so as to cause some 

action between objects in a scene presented in an interface of 

the portable computing device. 

18. The method of claim 12, wherein said manipulation 

challenge is used for unlocking and/or accessing the portable 

device. 

19. The method of claim 12 wherein said electronic image is presented in degraded form. 

20. The method of claim 12, wherein a speech response 

from the user is processed and used to determine if it matches 

a human Voice or a machine Voice, and further whether it 

contains a description of an image presented in an interface of 

the portable computing device. 

k k k k k 

Case 6:20-cv-00310-ADA Document 1-3 Filed 04/22/20 Page 34 of 34