EXHIBIT I 

 

Case 6:20-cv-00505-ADA Document 27-9 Filed 11/05/20 Page 1 of 12
(12)

(54)

(75)

(73)

(*)

(21)

(22)

(65)

(63)

(51)

(52)

(58)

(56)

USOO8812993B2

United States Patent

Goertz

(10)Patent No.: US 8,812,993 B2

(45)Date of Patent: Ang. 19, 2014

USER INTERFACE

Inventor: Magnus Goertz, Lidingo (SE)

Assignee: Neonode Ine., Santa Clara, CA (US)

Notice:

Appl. No.: 13/310,755

Filed: Dec. 4, 2011

Prior Publieation Data

References Cited

U.S. PATENT DOCUMENTS

4,243,879

4,301,447

4,790,028

4,847,606

A

A

A

A

1/1981

11/1981

12/1988

7/1989

(2006.01)

(2013.01)

(2013.01)

Int. CL

G06F 3/00

G06F 3/0481

G06F 3/0488

U.S. Cl.

CPC

Subject to any disclaimer, the term ofthis

patent is extended or adjusted under 35

U.S.C. 154(b) hy 〇 days.

Carroll et al.

Funk et al.

Ramage

Beiswenger

R ated U.S. Application Data

Continuation of application No. 10/315,250, filed on

Dec. 10, 2002, now Pat. No. 8,095,879.

US 2012/0094723 Al Apr. 19, 2012

G06F 3/0488 (2013.01); G06F 3/0481

(2013.01)

USPC 715/863; 715/764; 715/864

Field of Ciassification Search

USPC 715/863, 864, 764

See applieation file for complete seareh history.

5,036,t87 A 7/t99t Yb lidaetal.

5,053,758 A t〇/t99t Cornett et al.

5,194,863 A 3/t993 Barker et al.

5,283,558 A 2/1994 Chan

5,305,435 A* 4/1994 Bronson 715/777

5,406,307 A* 4/1995 Hirayama et al 715/800

5,463,725 A 10/1995 Henckel et al.

5,561,758 A 10/1996 Hocker et al.

5,577,733 A 11/1996 Downing

5,594,471 A* 1/1997 Deeran et al 345/173

5,603,053 A * 2/1997 Gough et al 710/5

5,612,719 A 3/1997 Beernink et al.

5,618,232 A 4/1997 Martin

5,644,628 A * 7/1997 Schwarzer et al 379/93.19

(Continued)

FOREIGN PATENT DOCUMENTS

EP

EP

〇 330 767 Bl 9/1989

0513694 A2 11/1992

(Continued)

OTHER PUBLICATIONS

Carlson, Jeff, Visual Quickstart Guide Palm Organizers. Peachpit

Press. 200〇 Berkeley, CA. pp. xiii, 12,25,26, 28-30,40,47,246 and

253.

(Continued)

Primary Examiner Ryan Pitaro

(74) Attorney, Agent, or Firm Soquel Group LLC

(57) ABSTRACT

A non-transitory computer readable medium storing computer program code whieh, when exeeuted by a mobile handheld deviee that has a touch sensitive display, instructs the

device (i) to display a notification of an incoming phone call,

and (ii) to accept the call in response to a mufti-step operation

comprising an object touching the touch sensitive display,

and the object gliding along the toueh sensitive display in a

specific direction.

8 Claims, 4 Drawing Sheets

212

21

Case 6:20-cv-00505-ADA Document 27-9 Filed 11/05/20 Page 2 of 12 US008812993B2
US 8,812,993 B2

Page 2

(56) References Cited

U.S. PATENT DOCUMENTS

715/764

5,745,116 A 4/1998 Pisutha-Arnond

5,748,185 A 5/1998 Stephan et al.

5,785,439 A 7/1998 Bowen

5,821,930 A 10/1998 Hansen

5,821,933 A 10/1998 Keller et al.

5,825,352 A 10/1998 Bisset et al.

5,889,236 A 3/1999 Gillespie et al.

5,898,434 A 4/1999 Small et al.

5,900,875 A 5/1999 Haitani et al.

5,907,327 A * 5/1999 Ogura et al......................

5,914,709 A 6/1999 Graham et al.

5,943,039 A 8/1999 Anderson et al.

5,943,044 A 8/1999 Martinelli et al.

5,956,030 A 9/1999 Conrad et al.

5,988,645 A 11/1999 Downing

6,037,937 A 3/2000 Beaton et al.

6,052,279 A 4/2000 Friend et al.

6,085,204 A 7/2000 Chijiwa et al.

6,088,032 A * 7/2000 Mackinlay ....................

6,292,179 Bl 9/2001 Lee

6,304,261 Bl 10/2001 Shields et al.

6,310,610 Bl 10/2001 Beaton et al.

6,340,979 Bl 1/2002 Beaton et al.

6,346,935 Bl 2/2002 Nakajima et al.

6,356,287 Bl 3/2002 Ruberry et al.

6,529,920 Bl 3/2003 Arons et al.

6,542,191 Bl 4/2003 Yonezawa

6,549,217 Bl *

Bl *

4/2003 De Greef et al.................

6,573,883 6/2003 Bartlett ..........................

6,597,345 B2 7/2003 Hirshberg

6,618,063 Bl 9/2003 Kurtenbach

6,639,584 Bl 10/2003 Li

6,646,633 Bl 11/2003 Nicolas

6,664,983 B2 12/2003 Ludoplh

6,690,365 B2 2/2004 HincUey et al.

6,690,387 B2 2/2004 Zimmerman et al.

6,707,449 B2 3/2004 Hinckley et al.

6,727,917 Bl 4/2004 Chew et al.

6,734,883 Bl 5/2004 Wynn et al.

6,757,001 B2*

Bl *

6/2004 Allport ..........................

6,757,002 6/2004 Oross et al......................

6,765,559 B2 7/2004 Hayakawa

6,788,292 Bl 9/2004 Nako et al.

6,812,940 B2* 11/2004 Arnold ...........................

6,833,827 B2 12/2004 Lui et al.

6,874,683 B2 4/2005 Keronen et al.

6,925,611 B2 8/2005 SanGiovanni

6,988,246 B2 1/2006 Kopitzke et al.

7,006,077 Bl 2/2006 Uusimaki

7,007,239 Bl * 2/2006 Hawkins et al.................

7,030,861 Bl 4/2006 Westerman et al.

7,046,232 B2* 5/2006 Inagaki et al....................

7,155,683 Bl * 12/2006 Williams .......................

7,159,120 B2* 1/2007 Muratov et al..................

7,159,763 B2 1/2007 Yap et al.

7,199,786 B2 4/2007 Suraqui

7,225,408 B2 5/2007 O'Rourke

7,231,231 B2* 6/2007 Kokko et al.....................

7,283,845 B2* 10/2007 De Bast .........................

7,286,063 B2 10/2007 Gauthey et al.

7,304,638 B2 12/2007 Murphy

7,343,565 B2* 3/2008 Ying et al........................

7,441,196 B2 10/2008 Gottftircht et al.

7,450,114 B2 11/2008 Anwar

7,665,043 B2* 2/2010 Kho ...............................

7,818,691 B2* 10/2010 Irvine ............................

7,880,724 B2 2/2011 Nguyen et al.

8,120,625 B2 2/2012 Hinckley

8,127,141 B2 2/2012 Hypponen

2001/0000668 Al* 5/2001 Bodnar ..........................

2001/0002694 Al 6/2001 Nakazawa et al.

2001/0017934 Al* 8/2001 Paloniemi et al...............

2001/0022579 Al 9/2001 Hirabayashi

2001/0026268 Al 10/2001 Ito

2001/0028344 Al 10/2001 Iwamoto et al.

715/848

715/745

345/156

715/840

715/864

715/817

715/780

345/158

715/816

713/182

455/566

455/566

715/780

715/864

715/856

345/339

382/107

2001/0028365 Al*

2001/0030641 Al

2001/0043189 Al*

2001/0043198 Al

2001/0055006 Al

2002/0002326 Al

2002/0015064 Al*

2002/0027549 Al

2002/0029341 Al*

2002/0046353 Al

2002/0054153 Al*

2002/0060699 Al*

2002/0063696 Al

2002/0075244 Al

2002/0109677 Al

2002/0135619 Al*

2002/0171691 Al

2002/0173300 Al*

2002/0191029 Al*

2003/0013483 Al*

2003/0016253 Al*

2003/0043207 Al

2003/0076306 Al

2003/0095102 Al*

2003/0122787 Al

2003/0206162 Al*

2004/0001101 Al*

2004/0021643 Al

2004/0021681 Al*

2004/0100510 Al

2004/0119763 Al*

2004/0125143 Al

2004/0141010 Al

2005/0035956 Al

2005/0253818 Al

2007/0263822 Al*

2009/0285383 Al*

10/2001 Ludolph 345/764

10/2001 Suzuki

11/2001 Brisebois et al 345/156

11/2001 Ludtke

12/2001 Sano et al.

1/2002 Causey, III et al.

2/2002 Robotham et al 345/863

3/2002 Hirshberg

3/2002 Juels et al 713/184

4/2002 Kishimoto

5/2002 Arnold 345/810

5/2002 D'Agostini 345/783

5/2002 Kubo et al.

6/2002 Tani et al.

8/2002 Taylor

9/2002 Allport 345/810

11/2002 Currans et al.

11/2002 Shtivelman et al 455/418

12/2002 Gillespie et al 345/8t0

t/2003 Ausems et al 455/556

1/2003 Aoki et al 345/863

3/2003 Duarte

4/2003 Zadesky et al.

5/2003 Kraft et al 345/168

7/2003 Zimmerman et al.

11/2003 Roberts 345/173

1/2004 Trajkovic et al 345/781

2/2004 Ho lino et al.

2/2004 Liao 345/702

5/2004 Milic-Frayling et al.

6/2004 Mizobuchi et al 345/863

7/2004 Deaton et al.

7/2004 Fitzmaurice et al.

2/2005 Sinclair et al.

11/2005 Nettamo

11/2007 Chang et al 379/202.01

11/2009 Tsuei 379/242

FOREIGN PATENT DOCUMENTS

EP 0618528 At 10/1994

EP 0703525 At 3/1996

WO 8600446 At t/t986

WO 8600447 At t/t986

OTHER PUBLICATIONS

Streitelmeier, Julie, "Palm mlOO." The Gadgeteer. 200〇 <http://

www.the-gadgeteer.eom/review/palm_mlOO_review> pp. 1-8.

Venolia et al., “T-Cube: A Fast, Self-Disclosing Pen-Based Alphabet.^^Apr. 24, 1994. pp. 265-27〇

Dulberg, et al.. An Imprecise Mouse Gesture for the Fast Activation

ofControls. Interact 1999. pp. 1-8.

Pogue, David, Palm Pilot: The Ultimate Guide, 2nd Edition, 1998.

O'Reilly and Associates, Inc. pp. I-I7.

C. Plaisant and D. Wallace, “Touchscreen tog e design", CHI '92

Proceedings ofthe SIGCHI Conferenee on Human Factors in Computing Systems, May 3-7, t992, pp. 667-668, ACM NewYork, NY,

USA.

Harrison, B. L., Fishkin, K. P., Gujar. A., Mochon, C. and Want, R., 

Squeeze Me, HoldMe, Tilt Me! An Exploration ofManipulative User

Interfaces, Proceeding ofthe CHI '98 Conferenee on Human Factors

in Computing Systems, ACM, Los Angeles, CA, Apr. 18-23, 1998,

pp. 17-24.

Kurtenbach, G. P., Sellen, A. J. and Buxton, W. A. S., An Empirical

Evaluation of Some Artieulatory and Cognitive Aspeets ofMarking

Menus, Human-Computer Interaction, vol. 8, Issue 1, L. Erlbaum

Associates Inc., Hillsdale, NJ, Mar. 1993, pp. 1-23.

MacKenzie, 1. S. and Soukoreff, R W., Text Entry for Mobile Computing: Models and Methods, Theory and Practice, Human-Computer Interaction, L. Erlbaum Associates Inc. Hillsdale, NJ, 2002,

vol. 17, pp. 147-198.

Mynatt, E. D., Igarashi, T., Edwards, W. K. and LaMarca, A.,

Flatland: New Dimensions in Office Whiteboards, Proceeding ofthe

CHI '99 Conferenee on Human Factors in Computing Systems,

ACM, NewYork, NY May t5-20, 1999, pp. 346-353.

Case 6:20-cv-00505-ADA Document 27-9 Filed 11/05/20 Page 3 of 12
US 8,812,993 B2

Page 3

(56) References Cited

OTHER PUBLICATIONS

Bier, E. A., Fishkin, K., Pier, K. and Stone, M. C., A Taxonomy of

See-Through Tools: TheVideo, Proceeding ofthe CHI '95 Mosaieof

Creativity, ACM, NewYork, NY May 7-t t, t995, pp. 4tt-4t2.

Bier, E. A., Stone, M. C., Fishkin, K., Buxton, W. and Baudel, T., A

Taxonomy of See-Through Tools, Conference on Human Factors in

Computing Systems, CHI '94, Boston, MA, ACM, New York, NY,

Apr. 24-28, 1994, pp. 358-364.

Damm, C. H., Hansen, K. M. and Thomsen, M., Tool Support for

Cooperative Object-OrientedDesign: Gesture Based Modeling on an

Electronic Whiteboard, Proceeding of the CHI '00 Conferenee on

Human Factors in Computing Systems, ACM, New Ybrk, NY, Apr.

t-6, 2000, pp. 518-525.

Jermyn, 1., Mayer, A., Monrose, F., Reiter, M. K. and Rubin, A. D.,

The Design andAnalysis ofGraphical Passwords, Proceedings ofthe

8th USENIX Seeurity Symposium, Washington, D.C., USA, Aug.

23-26,1999.

Kurtenbach, G. and Buxton, W., Gordon Kurtenbach and William

Buxton, The Limits ofExpert Performance Using Hierarchic Marking Menus, Proceedings from INTERCHI 93 Conference on

Human Factors in Computing Systems, ACM, New Ybrk, NY, Apr.

24-29, t993,pp. 482-487.

Pirhonen, A., Brewster, S. and Holguin, C., Gestural and Audio

Metaphors as a Means ofControl for Mobile Deviees, Proeeedings of

the SIGCHI Conferenee on Human Factors in Computing Systems:

Changing our World, Changing Ourselves, ACM New^ork, NY, pp. 

291-298.

SwitchHack 1.62",Jan. 17, 2001, downloaded from htlp://web.

archive.org/web/200101170650/littp://www.deskfi'ee.com/switchhack.html.

swipe_readme.txt. May 8, 2001, downloaded in swipe.zip from

htlp://web.archive.org/web/20010508013439/http://www.

samsungshop.co.yu/palm/swipe.zip.

“Desk Aeeessories", et. 18, 2000, downoaded from http: web.

archive.org/web/20001018025151/littp://www.doublebang.com/resourees/da.html.

"Launch 'em 1.53", Dec. 11, 1998, downloaded from http://www.

5star-shareware.com/PDA/Palm/Utilities/launchem.html.

Pogue, David, Palm Pilot: The Ultimate Guide, 2nd Edition, 1999,

O'Reilly and Assoeiates, Inc., pp. 19, 27-36, 38-47, 80, 113, 494,

507, 509,519-52〇

Brad A. Myers, "Mobile Deviees for Control", Mobile HCl 2002, F.

Paterno (Ed.), LNCS 2411, pp. 1-8, 2002. SpringerVerlag, Berlin,

Heidelberg.

BradA. Myers, Kin Pou Lie andBo-ChiehYMg, '*Two-Handed Input

Using a PDA and a Mouse", CHI '2000, Apr. I-6, 2000, CHI Letters,

vol. 2, Issue I, pp. 41-48, 2000, ACM, NewYork, NY, USA.

Brad A. Myers, “Using Handhelds and PCs fogether". Communications ofthe ACM, Nov. 2001, vol. 44, No. tt,pp. 34-41, ACM, New

York, NY, USA.

Gordon Kurtenbach and William Buxton, "User Learning and Performance with Marking Menus", CHI '94 Proceedings of the

SIGCHI Conferenee on Human factors in Computing Systems, pp. 

258-264, ACM, NewYork, NY, USA.

Gordon Paul Kurtenbach, “The Design and Evaluation of Marking

Menus", 1993. Doctoral Thesis, Graduate Department ofComputer

Science, University ofToronto.

David A. Carr, '*Specification ofInterface fnteraction Otgects", CHf

'94 Proceedings of the SIGCHI Conferenee on Human factors in

Computing Systems, pp. 372-378, 1994, ACM, NewYork, NY, USA.

David A. Carr, Ninad Jog, Harsha Prem Kumar, Marko Teittinen and

Christopher Ahlberg, "Using Interaction Otgect Graphs to Specify

Graphical Widgets", Tech. Rep. ISR-TR-94-69, Sep. t994. Institute

for Systems Research, University ofMaryland, College Park, MD.

Catherine Plaisant and Daniel Wallaee, “Touchscreen Toggle

Switches: Push or Slide? Design Issues and Usability Study", Tech.

Rep. CAR-TR-52t, Nov. t990, Human-Computer Interaction

Laboratory,Center for Automation Research, Dept. of Psychology,

University ofMaryland, College Park, MD.

Mark A. Tapia and Gordon Kurtenbach, “Some Design Refinements

and Principles on the Appearanee and Behavior ofMarking Menus",

UIST '95, pp. 189-195, ACM NewYork, NY, USA.

Michael McGuffm, Nicolas Burtnyk and Gordon Kurtenbach, “FaST

Sliders: Integrating Marking Menus and the Adjustment ofContinuous Values", Proc. Graphics Interface, May 2002, pp. 35-42.

Harrison, Beverly L., Fishkin, Kenneth P., Gujar, Anuj, Mochon,

Carlos andWant, Roy, Squeeze Me, Hold Me, Tilt Me! An Exploration of Manipulative User Interfaces, Proceeding of the CHI '98

Conferenee on Human Factors in Computing Systems, Apr. t8-23,

t998, pp. t7-24, ACM, Los Angeles, CA.

Kurtenbach, Gordon P., Sellen, Abigail J. and Buxton, WilliamA. S.,

An Empirical Evaluation of Some Artieulatory and Cognitive

Aspeets of Marking Menus, Human-Computer Interaction, vol. 8,

Issue 1, Mar. 1993, pp. 1-23, L. Erlbaum Associates Inc., Hillsdale,

NJ.

MacKenzie, 1. Scott and Soukoreff, William, Text Entry for Mobile

Computing: Models and Methods, Theory and Practice, HumanComputer Interaction, 2002, vol. 17, pp. 147-198, L. ErlbaumAssociates Inc., Hillsdale, NJ.

Mynatt, Elizabeth D., Igarashi, Takeo, Edwards, W. Keith and

LaMarca, Anthony, Flatland: New Dimensions in Office

Whiteboards, Proceeding of the CHI '99 Conferenee on Human

Factors in Computing Systems, May t5-20, 1999, pp. 346-353,

ACM, Pittsburgh, PA.

Bier, Eric A., Fishkin, Ken, Pier, Ken and Stone, Maureen C., A

Taxonomy ofSee-Through Tools: The Video, Proceeding ofthe CHI

'95 Mosaie ofCreativity, May 7-tt, 1995, pp. 4tt-4t2, ACM, Denver, CO.

Bier, Eric A., Stone, Maureen C., Fishkin, Ken, Buxton, William and

Baudel, Thomas, A Taxonomy ofSee-Through Tools, Conference on

Human Factors in Computing Systems, CHI 1994, Boston, MA, Apr.

24-28, 1994, Proceedings, pp. 358-364. ACM, NewYork, NY.

Damm, Christian Heide, Hansen, Klaus Marius and Thomsen,

Miehael, Tool Support for Cooperative Otgeet-Oriented Design:

Gesture BasedModeling on anElectronicWhiteboard, Proceeding of

the CHI '00 Conferenee on Human Factors in Computing Systems,

Apr. 1-6, 2000, pp. 518-525, ACM, NewYork, NY.

Jermyn, Ian, Mayer Alain, Monrose, Fabian, Reiter, Michael K. and

Rubin, Aviel D., The Design and Analysis of Graphical Passwords,

Proceedings ofthe 8th USENIX Seeurity Symposium, Washington,

D.C., USA, Aug. 23-26, 1999.

Kenwood KVT-9ttDVD Instruction Manual, 2001, Japan.

Kurtenbach, Gordon and Buxton, William, The Limits of Expert

Performance Using Hierarchic Marking Menus, Proceedings from

INTERCHI 93 Conference on Human Factors in Computing Systems, pp. 482-487, ACM, NewYork, NY.

Pirhonen, Antti, Brewster, Stephen and Holguin, Christopher, Gestural andAudio Metaphors as aMeans ofControl forMobile Deviees,

Proceedings ofthe SIGCHI eonferenee on Human factors in computing systems: Changing OurWorld, Changing Ourselves, pp. 291­

298, ACM NewYork, NY.

Launch 'em 3.02, Oct. 8, 2001, Synergy Solutions Inc., pp. 1-2〇

Launch 'em. Synergy Solutions Inc., Oct. 8, 2001.

* cited by examiner

Case 6:20-cv-00505-ADA Document 27-9 Filed 11/05/20 Page 4 of 12
U.S. Patent Aug. 19, 2014 Sheet 1 of 4 us 8,812,993 B2

Fig. 1. 212

21 Fig. 3.

-222

Fig. 2.

Fig. 4.

Fig. 5. 22

Case 6:20-cv-00505-ADA Document 27-9 Filed 11/05/20 Page 5 of 12
U.S. Patent Aug. 19, 2014 Sheet 2 of 4 US 8,812,993 B2

233

231

i

¢^'taskmanager J

10 rPHONE

02 SMS

I © 3 CALCULATOR

I fS 4 CAMERA

C? 5 CHAT /------ヽ

tS 6 NOTEBOOK

S 〇 7 MP3 PLAYER

I e a RECORDER

© 9 E-M L

Fig. 6.

23

231 231a

Fig. 8.

231

4

Case 6:20-cv-00505-ADA Document 27-9 Filed 11/05/20 Page 6 of 12
U.S. Patent Aug. 19, 2014 Sheet 3 of 4 US &812,993 B2

«7)T*SMniANACEn

〇 1FHOHC

231 32

ME *

Fig. 9. 3b

@asM

at 5 CALCUAFkヽヽ c N

〇 C«MCf<A ヽヽ、 ゝ I

Q b CHAT ヽーー

¢•MOTEBOOK

〇 TMFa rLATCH

〇 • Mcconp»

ifl a ________

α *NV

〇 NOTE /-----

〇|*

〇 9 CALCULATOR

〇 MUSIC

Q 〇*»¢

n TtTRia

D) p*cni

P

Fig. 10.

Fig. 11.

Fig. 12.

Case 6:20-cv-00505-ADA Document 27-9 Filed 11/05/20 Page 7 of 12
U.S. Patent Aug. 19, 2014 Sheet 4 of 4 US &812,993 B2

Fig. 13.

5

Fig. 14.

Case 6:20-cv-00505-ADA Document 27-9 Filed 11/05/20 Page 8 of 12
US 8,812,993 B2

USER INTERFACE

PRIORITY REFERENCE TO RELATED

APPEICATIONS

This application is a continuation ofU.S. application Ser.

No. 10/315,250, entitled USER INTERFACE EORMOBIEE

HANDHELD COMPUTER UNIT, filed on Dec. 10, 2002

now U.S. Pat. No. 8,095,879 by inventor Magnus George

Goertz.

TEGHNIGAL EIELD

The present invention relates to a user interface for a

mobile handheld eomputer unit, whieh eomputer unit eomprises a touch sensitive area, andwhichtouch sensitive area is

divided into a menu area and a display area.

The eomputer unit is adapted to run several applications

simultaneously and to present any active applieation on top of

any other application on the display area.

The present invention also relates to an enclosure for a

handheld eomputer unit

The present invention also relates to a computer readable

medium. A eomputer program product with computer program code is stored within the eomputer readable medium,

whieh eode, when read by a computer, will make it possible

for this computer to present a user interface aeeording to the

invention.

DESCRIPTION OF BACKGROUND ART

Mobile handheld eomputers are known in various embodiments. One kind ofhandheld eomputer is the personal digital

assistant (PDA), which is getting more and more powerful.

Another kind of handheld eomputer unit is the mobile

phone, whieh also is getting more and more powerful. There

are also examples ofwhere the mobile phone and the PDA are

merging into one unit.

A third kind ofhandheld eomputer is the laptop computer,

which is getting smaller and smaller, even eompeting in size

with the PDA's.

The need to manage more information has led the development towards new solutions regarding user interfaces and

navigation. The PDA's and mobile phones are getting larger

and larger in order to provide a user-friendly interfaee.

Since the users have gotten used to small handheld units, it

is hard to move towards larger units. This has led to foldable

keyboards, different kinds ofjoy sticks and different kinds of

touch sensitive displays and pads intended to help in providing a user interface that is suitable for small handheld computer units.

SUMMARY OF THE PRESENT INVENTION

Technical Problems

It is a problem to provide a user-friendly interfaee that is

adaptedto handle a large amount ofinformation and different

kinds oftraditional computer-related applieations on a small

handheld eomputer unit.

It is a problem to provide a user interface that is simple to

use, even for inexperienced users of computers or handheld

deviees.

It is a problem to provide a small handheld eomputer unit

with an easily accessible text input function.

2

It is also a problem to provide a simple way to make the

most commonly used funetions for navigation and management available in the environment of a small handheld eomputer unit.

Solution

Taking these problems into consideration, and with the

staring point from a user interface for a mobile handheld

eomputer unit, which eomputer unit comprises a touch sensitive area, which touch sensitive area is divided into a menu

area and a display area, which eomputer unit is adaptedto run

several applications simultaneously and to present an active

applieation on top of any other application on the display

area, the present invention teaches that the menu area is

adapted to present a representation of a first, a second and a

third predefined fimetion, where the first function is a general

application dependent funetion, the seeond funetion is a keyboard function, and the third function is a task and file manager. The present invention also teaches that any one ofthese

three funetions can be activated when the touch sensitive area

detects a movement ofan object with its starting point within

the representation ofthe funetion on the menu area and with

a direction from the menu area to the display area.

With the purpose ofproviding a simple way ofmanaging

any application or the operations system, the present invention teaches that ifthe first function is activated, the display

area is adapted to display icons representing serviees or settings, depending on the eurrent aetive application. One ofthe

ieons always represents a "help''-service, regardless ofapplication. The ieons are adapted to represent serviees or settings

ofthe operations system ofsaid computer unit, such as background picture, clock, users, help, etc. if no application is

currently active on the eomputer unit.

Selections ofpreferred serviee orsetting is done by tapping

on corresponding icon

With the purpose of providing the aeeess to a text input

funetion in any application in the eomputer unit, the present

invention teaches that when the second function is activated,

the display area is adapted to display a keyboard and a text

field.

Ifa text passage in an active applieation is highlighted, then

this text passage is displayed in the text ロ d for editing

through the keyboard and that the highlighted text passage is

replaeed by the edited text passage when the seeond function

is deactivated.

If no text passage in an active applieation is highlighted,

then the text field is available for inputting and editing oftext

through the keyboard.

In the ease ofthe latterthe first function can be activated, or

the seeond function can be closed, inwhieh a choice ofsaving

or deleting the inputted text is given. The ehoiee ofsaving the

inputted text results in an activation ofthe first function. In

this case the first function will present serviees or settings

available forthe inputted text, sueh as saving the inputted text

for later use, using the inputted text as telephone number in a

telephone applieation, orsending the inputted text as message

in eommunieations application.

In order to provide a task and file management in a user

interface for a handheld mobile eomputer, the present invention teaches that, ifthe third function is activated, the display

area is adapted to display a list with a library of available

applieations and files on the eomputer unit. A selection of an

application will start the application, and a selection of a file 

will open the file in an application intended for the file.

A seleetion ofan application or a file is done by moving the

objeet so that the representation ofdesired applieation or file 

5

10

15

20

25

30

35

40

45

50

55

60

65

Case 6:20-cv-00505-ADA Document 27-9 Filed 11/05/20 Page 9 of 12
US 8,812,993 B2

3

is highlighted, removing the ohjeet from the toueh sensitive

area, and then tapping on the toueh sensitive area.

According to the present invention a navigation in the list is

performed hymoving the ohjeet in a direction towardsthe top

ofthe list ortowards the bottom ofthe list. This will eause the

marking to move in the same direetion. The speed of the

movement of the marking is lower than the speed of the

movement of the ohjeet, with the purpose of making the

navigation easier.

The user interface ofthe present invention is specifically

adapted to be used with a small computer unit where the size

ofthe toueh sensitive area is in the order of 2-3 inches. The

user interface is also adapted to be operated by one hand,

where the ohjeet eanhe a finger, such as the thumb, of a user

ofthe eomputer unit.

Advantages

Those advantages that can be primarily associated with a

user interface or a computer readable medium aeeording to

the present invention reside in the ability to establish a userfriendly interfaee forsmall handheld eomputers, both regarding general application set-up functions, text input funetions,

and file and task management.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will now be deseribed inmore detail

with reference to the aeeompanying drawings, in which

FIG. 1 is a schematic and highly simplified view ofa touch

sensitive area on a mobile handheld eomputer unit; FIG. 2 is

a schematic side view illustrating the activation ofa function;

FIG. 3 is a schematic illustration of a first function;

FIG. 4 is a schematic side view illustrating the selection of

a service or setting represented by an icon;

FIG. 5 is a schematic illustration of a second function;

FIG. 6 is a schematic side view illustrating the selection of

a third function;

FIG. 7 is a schematic illustration of an application or file;

FIG. 8 is a schematic illustration on how navigation is

performed;

FIG. 9 is a schematic illustration ofhow the eontent ofthe

display are is ehanged;

FIG. 10 is a schematic side view further illustrating how

navigation is performed;

FIG. llisa schematic illustration ofmoving forwardsin an

application;

FIG. 12 is a schematic illustration ofmoving backwardsin,

or closing, an application;

FIG. 13 is a schematic illustration of an enclosure

FIG. 14 shows a computerreadable medium in the form of

a solid state memory.

DESGRIPTION OF EMBODIMENTS AT

PRESENT PREFERRED

FIG. 1 illustrates a user interface for a mobile handheld

eomputer unit. The user interface aeeording to the present

invention is specifically adapted to computer units comprising a touch sensitive area 1, which is divided into a menu area

2 and a display area 3. It should be understood that there are

several different kinds ofknown toueh sensitive displays and

that the present invention to does not depend on what kind of

touch sensitive display that is used in relation to the inventive

user interface.

The eomputer unit is adapted to run several applications

simultaneously and to present an active applieation on top of

4

any other application on the display area 3. It should be

understoodthat by simultaneously it is meant any technology

that will make it appear to a user ofthe eomputer unit that

applications are run simultaneously and that the present

invention does not depend on how this is realised, whether it

is through time-sharing ofone proeessor, parallel use ofseveral processors, or any other technique.

Aeeording to the present invention the menu area 2 is

adapted to present a representation of a first 21, a second 22

and a third 23 predefined funetion.

The first function 21 is a general application dependent

funetion, the seeond funetion 22 is a keyboard function, and

the third function 23 is a task and file manager.

FIG. 2 shows that any one ofthese three funetions 21, 22,

23 ean be aetivated when the toueh sensitive area 1 detects a

movement of an object 4 with its starting point A within the

representation of a function on the menu area 2 and with a

direction B from the menu area 2 to the display area 3.

FIG. 3 shows that ifthe first function 21 is activated, then

the display area 3 is adapted to display icons 211, 212, 213,

214, 215, 216 representing serviees or functions depending

on the eurrent aetive application. One of the ieons, in the

figure exemplified by ieon 211, always represents a "help"-

service, regardless of application. Any key that, because of

lack ofspace on the display area, or because the key should be

hidden from the aetive applieation, or because of any other

reason is not shown on the display area of an active application, can be represented by one ofthe ieons 212, 213, 214,

215, 216 that is shown when the first function 21 is activated.

Iffor instance the active application handles a picture, then

the ieons that are shown when the first function is activated

can be services such as "save to disk", "send as SMS", or

“delete" and they ean be settings sueh as "'resolution",

“colour", or "brightness".

If no application is currently active on the eomputer unit,

then the ieons 211, 212, 213, 214, 215, 216 are adapted to

represent serviees or settings ofthe operations system ofthe

eomputer unit, such as background picture, eloek, alarm 215,

users 213, help 211, ete.

FIG. 4 showsthat selection ofa preferred serviee orsetting

is done by tapping G, D on corresponding icon 213.

FIG. 5 shows that if the seeond function 22 is activated,

then the display area 3 is adapted to display a keyboard 221

and a text field 222.

Two different seenarios can be at hand when this funetion

key is activated. A first scenario can be that a text passage in

the active application is highlighted as the seeond function is

activated. Ifthis is the case then the highlighted text passage

is displayed in the text field 222 for editing through the

keyboard 221.

The highlighted text passage is replaeed by the edited text

passage when the seeond function 21 is deactivated.

A second scenario can be that no text passage in the aetive

applieation is highlighted. Ifthis is the ease then the text field

222 is available for inputting and editing oftext through the

keyboard 221.

In the ease ofthe latterscenario, the first function 21 can be

activated, or the seeond function 22 can be closed. If the

seeond function 22 is closed then a choice ofsaving or deleting the inputted text is given, where the ehoiee ofsaving the

inputted text results in an activation ofthe first function 21.

As the first function 21 is activated with the second function 22 as currently active application the first function 21 will

present serviees or settings available for the inputted text,

sueh as saving the inputted text for later use, using the inputted text as telephone number in a telephone applieation, or

5

10

15

20

25

30

35

40

45

50

55

60

65

Case 6:20-cv-00505-ADA Document 27-9 Filed 11/05/20 Page 10 of 12
US 8,812,993 B2

5

sending the inputted text as message in eommnnieations

application, snch as e-mail, SMS, or fax.

FIG. 6 shows that ifthe third function 23 is activated, then

the display area 3 is adaptedto display a list 231 with a library

of available applieations and files on the eomputer unit.

A seleetion ofan application will start the applieation, and

a selection of a s file will open the file in an application

intended for the file The name ofa selected file can be edited

by activation of the seeond funetion 22 as the file is highlighted.

FIG. 7 shows that a selection of an application or a file is

done by moving E the objeet 4 so that the representation of

desired applieation or file is highlighted, removing F the

objeet 4 from the toueh sensitive area 1, and then tapping G,

H on the toueh sensitive area 1.

An application or file is highlighted by plaeing some kind

ofmarking 232 onthe representation ofthe applieation orfile.

This marking can be done in different ways, for example by

putting a frame aroundthe representation ofthe application or

file, as shown in the figure, or by inverting the representation

ofthe applieation or file.

It should be understood that all lists in the eomputer unit,

such as a list of contact information in an address book, a list

of e-mail messages in a mailbox, or a telephone fog, ean be

managed in the above deseribed manner.

The list 231 ean be adapted to present only files or only

applications. In this case, the top area of the list 231 ean

present a field 233 through which the content ofthe list 231

ean be altered. Ifthe list only presents files, then the field 233

ean display a representation ofa taskmanager and a selection

ofthe field 233 will eause the list 231 to after to present only

applications, and if the list 231 only presents applieations,

then the field 233 displays a representation of a file manager

and a selection ofthe field 233 will eause the list 231 to after

and present only files.

FIG. 8 shows that navigation in the list is performed by

moving the objeet 4 ina direction I towardsthe top 2310 ofthe

list 231 or towards J the bottom 231ろ of le list 231. This

movement I, J ofthe objeet 4 will eause the marking 232 to

move K, L in the same direetion. The speed ofthe movement

K, L of the marking 232 is lower than the speed of the

movement I, J ofthe objeet 4.

FIG. 9 shows that ifthe number ofapplications and/or files

in the list 231 exeeedsthe number ofapplications and/or files

that can be presented on the display area 3, and ifthe objeet 4

is moved to the top orbottomposition ofthe display area, then

lifted, replaeed on the display area, and then again moved to

the top or bottom ofthe display area, then the eontent ofthe

display area will be replaeed one whole page, meaning that if

the objeet 4 is positioned N at the bottom 3b ofthe display

area 3, then lifted, replaeed on the display area 3, and then

againmoved Mto the bottom 3b ofthe display area 3, then the

eontent 31 of the display area 3 will be replaeed P by the

following applications and/or files 32 in the list 231. In the

same way, but not shown in the figure, ifthe objeet is positioned at the top ofthe display area, then lifted, replaeed on

the display area 3, and then again moved to the top of the

display area, the eontent ofthe display area will be replaeed

by the preeeding applieations and/or files in the list.

FIG. 10 shows that ifthe objeet 4 is removed Q from a first

position 33 on the display area 3 and then replaeed R, S on a

second position 34 on the display area 3, then the navigation

can be continued T from the seeond position 34.

FIG. 11 shows that moving U the objeet 4 from the left of

the display area 3 to the right ofthe display area 3 moves the

aetive application, function, service or setting on one step 

forwards. FIG. 12 shows that, in a similar manner, the aetive

6

applieation, function, service or setting is elosed or backed

one step by moving V the objeet 4 fromthe right ofthe display

area 3 to the left ofthe display area 3.

As shown in FIG. 1, the menu area 2 is positioned at the

bottom ofthe toueh sensitive area 1. fhe representation ofthe

first ftmction 21 is positioned at the left side ofthe menu area

2, the representation ofthe seeond function 22 is positioned at

the middle ofthe menu area 2, and the representation ofthe

third ftmction 23 is positioned at the right side ofthe menu

area 2.

As shown in FIG. 13, the present invention relates to a user

interface for a hand held mobile unit that preferably can be

manageable with one hand. Hence the present invention

teaches that the user interface is adapted to a touch sensitive

area 1 with a size that is inthe order of2-3 inches, meaning the

diagonal distance W between two comers ofthe toueh sensitive area 1.

The user interface is adapted to be operated by one hand,

where the objeet 4 ean be a finger, such as the thumb shown in

the figures, ofa user ofthe eomputer unit. It should be understood though that the present invention might also be used

with another object, sueh as a pen or other pointing device.

According to one preferred embodiment of the present

invention the eomputer unit is covered with an enclosure 5,

which is provided with an opening 51 for the display area 3,

andwhere the representations ofthemenu area 2 is printed on

top ofthe enelosure 5. It should be understood that the opening 51 might be a transparent part ofthe enelosure 5 orthat it

might be an open aperture depending on among other things

technical considerations pertaining to the toueh sensitive area

1.

This makes it possible to allow the enelosure 5 to be removable and exchangeable.

FIG. 14 shows a computer readable medium, in the figure

sehematieally shown as a solid-statememory 61. A computer

program product is stored within the computer readable

medium. This eomputer program product comprises eomputer readable eode 62, whieh, when read by a computer 6,

will make it possible for the eomputer 6 to present a user

interface aeeording to the present invention.

The present invention also teaches that the computer program product is adapted to function as a shell upon an operations system.

It will be understood that the invention is not restricted to

the aforedeseribedand illustratedexemplifying embodiments

thereof, and that these embodiments ean be modified within

the seope ofthe inventive eoneept illustrated in the aeeompanying claims.

The invention claimed is:

1. A non-transitory computer readable medium storing

instructions, which, when executed by a processor ofan electronic device having a touch-sensitive display screen, eause

the processorto enable a userinterface ofthe device, the user

interface comprising at least two states, namely, (a) a tappresent state, wherein a plurality oftap-activatable icons for a

respective plurality of pre-designated system fimetions are

present, eaeh system funetion being aetivated in response to a

tap on its respective icon, and (b) a tap-ahsent state, wherein

tap-aetivatable icons are absent but an otherwise-activatable

graphic is present in a strip along at least one edge of the

display screen for transitioning the user interface from the

tap-absent state to the tap-present state in response to a multistep user gesture eomprising (i) an objecttouching the display

screen within the strip, and (ii) the object gliding on the

display screen away from and out ofthe strip.

2. The eomputerreadable medium ofclaim 1, wherein any 

state transition elicited by a user gesture that begins at a

5

10

15

20

25

30

35

40

45

50

55

60

65

Case 6:20-cv-00505-ADA Document 27-9 Filed 11/05/20 Page 11 of 12
US 8,812,993 B2

7 8

location at which the otherwise-activatahle graphic is provided, transitions to the tap-present state.

3. The eomputer readable medium of claim 1, wherein the

tap-present state does not display the tap-aetivatahle icons

within a window frame. 5

4. The eomputer readable medium of claim 1, wherein the

plurality ofpre-designated system funetions eomprises a help

funetion.

5. The eomputer readable medium of claim 1, wherein the

plurality of pre-designated system funetions comprises a lo

clock funetion.

6. The computer readable medium of claim 1, wherein the

plurality of pre-designated system funetions comprises an

alarm function.

7. The computer readable medium of claim 1, wherein the 15

strip is less than a thumb's width wide within the display

screen.

8. The eomputer readable medium of claim 1, wherein the

multi-step user gesture comprises (i) the objeet touehing the

otherwise-aetivatable graphic, and (ii) the objeet gliding on 20

the display screen away from and out ofthe strip.

Case 6:20-cv-00505-ADA Document 27-9 Filed 11/05/20 Page 12 of 12