Exhibit 16 

Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 1 of 49
UNITED STATES PATENT AND TRADEMARK OFFICE

____________

BEFORE THE PATENT TRIAL AND APPEAL BOARD

____________

SAMSUNG ELECTRONICS CO. LTD., SAMSUNG ELECTRONICS

AMERICA, INC., AND APPLE, INC.,

Petitioner,

v.

NEONODE SMARTPHONE LLC,

Patent Owner.

____________

Case IPR2021-00144

Patent 8,095,879

____________

EXHIBIT 2007

SECOND DECLARATION OF CRAIG ROSENBERG, PH.D.

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 2 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 2 of 49
i

TABLE OF CONTENTS

Page

I. INTRODUCTION.........................................................................................1

II. QUALIFICATIONS .....................................................................................2

III. APPLICABLE LEGAL STANDARDS.......................................................9

A. Priority Date of the Patent.....................................................................9

B. Level of Ordinary Skill in the Art.........................................................9

C. My Understanding of Legal Standards...............................................10

IV. OPINIONS...................................................................................................12

A. Objective Evidence Of Non-Obviousness. .........................................12

B. Claim 1 (And All Dependents): “Gliding ... Away”..........................19

C. Claim 1 (And All Dependents): “Wherein The Representation Of 

The Function Is Not Relocated Or Duplicated”..................................27

1. Disclosure By Hirayama-307 Alone........................................ 28

2. Obviousness In View Of Hirayama-307 Alone....................... 36

3. Obviousness By Hirayama-307 In View Of Ren .................... 40

V. CONCLUSION............................................................................................45

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 3 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 3 of 49
1

I. INTRODUCTION

I, Craig Rosenberg, declare as follows: 

1. I have been retained on behalf of Neonode Smartphone LLC

(“Neonode” and/or “Patent Owner”) for the above-captioned inter partes review to 

provide my expert opinions and expert knowledge. I understand that this 

proceeding involves U.S. Patent No. 8,095,879 (“the ’879 patent”). I understand 

that the ’879 patent is currently assigned to Neonode. 

2. I understand that the present Petition for inter partes review challenges

claims 1-6, 12, 14-17 (“the challenged claims” or “claims”) of the ’879 patent and 

was filed by Petitioners Samsung Electronics Co. Ltd., Samsung Electronics 

America, Inc., and Apple Inc. (“Petitioner”).

3. I understand that while the Petition originally presented Grounds 1A1E, 2A-2D and 3, Patent Owner and Petitioners have since agreed to limit the 

Petition to Grounds 2A-2D only and, therefore, my opinions in this declaration are 

limited to these Grounds.

4. I have been asked to provide my independent review, analysis, 

insights, and opinions regarding technical aspects of the ’879 patent and the 

Petition challenging the patentability of its claims. In particular, I have been asked 

to provide my analysis, insights, and opinions regarding the state of the art at the 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 4 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 4 of 49
2

time of the alleged invention and how a person of ordinary skill in the art would 

have understood the ’443 patent disclosure at that time. 

5. In preparing this declaration, I have reviewed all of the references cited 

herein and in the Petition. In particular, I have reviewed and am familiar with the 

’879 patent and its prosecution history, and the references cited against it, 

discussed further below. 

6. In this declaration, I set forth the independent opinions that I have 

reached and the basis for those opinions in view of the information currently 

available to me. Such opinions are based, at least in part, on my experience for the 

past three decades with image and video processing, including encoding, decoding, 

and transmission. I reserve the right to supplement or revise my opinions should 

additional documents or other information be provided to me.

7. I am being compensated at an hourly rate of $450/hour for my work on 

this case. My compensation is not dependent upon my opinions, my testimony, or 

the outcome of this case.

II. QUALIFICATIONS

8. All of my opinions stated in this declaration are based on my own 

personal knowledge and professional judgment. In forming my opinions I have 

relied on my knowledge and experience in human factors, user interface design, user 

interaction design, human-computer interaction, and software engineering. 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 5 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 5 of 49
3

9. My qualifications to testify about the ’879 patent and the relevant 

technology are set forth in my curriculum vitae (“CV”), which I have included as 

Ex. 2002. In addition, a brief summary of my qualifications is included below: 

10. I hold a Bachelor of Science in Industrial Engineering, a Master of 

Science in Human Factors, and a Ph.D. in Human Factors from the University of 

Washington School of Engineering. For 30 years, I have worked in the areas of 

human factors, user interface design, software development, software architecture, 

systems engineering, and modeling and simulation across a wide variety of 

application areas, including aerospace, communications, entertainment, and 

healthcare. 

11. I graduated from the University of Washington in 1988 with a B.S. in 

Industrial Engineering. After graduation, I continued my studies at the University of 

Washington. In 1990, I obtained an M.S. in Human Factors. In 1994, I graduated 

with a Ph.D. in Human Factors. In the course of my doctoral studies, I worked as an 

Associate Assistant Human Factors Professor at the University of Washington 

Industrial Engineering Department. My duties included teaching, writing research 

proposals, designing and conducting funded human factors experiments for the 

National Science Foundation, as well as hiring and supervising students. While 

studying at the University of Washington, I also worked as a human factors 

researcher and designed and performed advanced human factors experiments 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 6 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 6 of 49
4

relating to virtual environments and interface design, stereoscopic displays, and 

advanced visualization research, which was funded by the National Science 

Foundation. My duties included user interface design, systems design, software 

development, graphics programming, experimental design, as well as hardware and 

software interfacing. 

12. I have published twenty-one research papers in professional journals 

and proceedings in the areas of user interface design, computer graphics, and the 

design of spatial, stereographic, and auditory displays. I also authored a book chapter 

on augmented reality displays in the book “Virtual Environments and Advanced 

Interface Design” (Oxford University Press, 1995). In addition, I created one of the 

first virtual spatial musical instruments called the MIDIBIRD that utilized the MIDI 

protocol, two six-dimensional spatial trackers, a music synthesizer, and a computer 

graphics workstation to create an advanced and novel musical instrument. 

13. For the past 21 years, I have served as a consultant for Global Technica, 

Sunny Day Software, Stanley Associates, Techrizon, CDI Corporation, and the Barr 

Group. In this capacity, I have provided advanced engineering services for many 

companies. 

14. I consulted for the Boeing Company for over 16 years as a senior human 

factors engineer, user interface designer, and software architect for a wide range of 

advanced commercial and military programs. Many of the projects that I have been 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 7 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 7 of 49
5

involved with include advanced software development, user interface design, agentbased software, and modeling and simulations in the areas of missile defense, 

homeland security, battle command management, computer aided design, 

networking and communications, air traffic control, location-based services, and 

Unmanned Aerial Vehicle (“UAV”) command and control. Additionally, I was the 

lead system architect developing advanced air traffic controller workstations and air 

traffic control analysis applications, toolsets, and trade study simulations for Boeing 

Air Traffic Management. 

15. I was also the architect of the Boeing Human Agent Model. The Boeing 

Human Agent Model is an advanced model for the simulation of human sensory, 

cognitive, and motor performance as applied to the roles of air traffic controllers, 

pilots, and UAV operators. In another project, I was the lead human factors engineer 

and user interface designer for Boeing’s main vector and raster computer aided 

drafting and editing system that produces the maintenance manuals, shop floor 

illustrations, and service bulletins for aircraft produced by the Boeing Commercial 

Aircraft Company. Additional responsibilities in my time as a consultant include 

system engineering, requirements analysis, functional specification, use case 

development, user stories, application prototyping, modeling and simulation, objectoriented software architecture, graphical user interface analysis and design, as well 

as UML, C++, C#, and Java software development. 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 8 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 8 of 49
6

16. In 1995 and 1996, I was hired as the lead human factors engineer and 

user interface designer for the first two-way pager produced by AT&T. Prior to this 

technology, people could receive pages but had no way to respond utilizing their 

pager. This new technology allowed users to use a small handheld device to receive 

and send canned or custom text messages, access and update an address book, and 

access and update a personal calendar. This high-profile project involved designing 

the entire feature set, user interface/user interaction design and specification, as well 

as all graphical design and graphical design standards. 

17. From 1999–2001, I was the lead human factors engineer and user 

interface designer for a company called Eyematic Interfaces that was responsible for 

all user interface design and development activities associated with real-time mobile 

handheld 3D facial tracking, animation, avatar creation and editing software for a 

product for Mattel. My work involved user interface design, human factors analysis, 

requirements gathering and analysis, and functional specifications. 

18. In 2001, I was the lead user interface designer for a company called 

Ahaza that was building IPv6 routers. I designed the user interfaces for the 

configuration and control of these advanced network hardware devices. My 

responsibilities included requirements analysis, functional specification, user 

interface design, user experience design, and human factors analysis.

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 9 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 9 of 49
7

19. In 2006-07, I was the lead user interface designer for a company called 

ObjectSpeed that developed a portable handheld telephone for use in homes and 

businesses that had many of the same capabilities that we take for granted in mobile 

cellular phones. This portable multifunction device supported voice, email, chat, 

video conferencing, internet radio, streaming media, Microsoft Outlook integration, 

photo taking and sharing, etc. The ObjectSpeed device was specifically designed and 

developed as a portable handheld device. 

20. I am the founder, inventor, user interface designer, and software 

architect of WhereWuz. WhereWuz is a company that produces advanced mobile 

software running on GPS-enabled smartphones and handheld devices. WhereWuz 

allows users to record exactly where they have been and query this data in unique 

ways for subsequent retrieval based on time or location. WhereWuz was specifically 

designed and developed to run on small handheld devices. 

21. I am the co-founder of a medical technology company called Healium. 

Healium developed advanced wearable and handheld user interface technology to 

allow physicians to more effectively interact with electronic medical records. 

22. I am the co-founder of a medical technology company called 

StratoScientific. StratoScientific is developing an innovative case for a smartphone 

that turns a standard handheld smartphone into a full featured digital stethoscope that 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 10 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 10 of 49
8

incorporates visualization and machine learning that can be utilized for telemedicine 

and automated diagnosis.

23. In 2012-13, I designed and developed a large software project for 

Disney World called xVR that allowed the operational employees of Disney World 

to utilize a handheld device to view the current and historical status of all of the 

guests of Disney World within multiple attractions as well as within one of their 

restaurants. The application could run in a real-time/live mode where it would 

display data collected from sensors that showed the location and status of all guests 

within the attraction; the application could also be run in a fast-time/simulated mode. 

The application was developed on a laptop computer and was specifically designed 

to run on a variety of devices, including laptops, PCs, smartphones, and tablets. 

24. I have received several awards for my engineering work relating to 

interface design, computer graphics, and the design of spatial, stereographic, and 

auditory displays, including a $10,000 scholarship from the I/ITSEC for advancing 

the field of interactive computer graphics for flight simulation and a Link Foundation 

award for furthering the field of flight simulation and virtual interface design. I have 

also created graphics for several popular book covers as well as animations for a 

movie produced by MIRAMAR. 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 11 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 11 of 49
9

III. APPLICABLE LEGAL STANDARDS

25. When considering the ’879 patent and stating my opinions, I rely on 

the following legal standards as described to me by the attorneys for Neonode.

A. Priority Date of the Patent

26. I understand that the analysis of alleged obviousness of the Patent 

should be performed from the perspective of a POSITA as of the priority date of 

the Patent. The Patent was filed on December 10, 2002. My opinions in this 

matter are from the perspective of a POSITA as of that date; however, my opinions 

do not change if the priority date is slightly changed. 

B. Level of Ordinary Skill in the Art 

27. I understand that various factors should be considered when 

determining the person of ordinary skill in the art in connection with a particular 

patent. I understand that these include, without limitation: (a) the educational level 

of the inventors and that of practitioners and other inventors in the art (e.g., 

degrees, subjects, etc.); (b) the type of problems encountered in the art; (c) prior art 

solutions to such problems; (d) the speed at which innovations are made in the art; 

and (e) the sophistication of the invention.

28. Dr. Bederson opines that a “POSA for the ’879 patent would have had 

at least a bachelor’s degree in computer science, computer engineering, or the 

equivalent education and at least two years of experience in user-interface design 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 12 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 12 of 49
10

and development. Additional years of experience could substitute for formal 

education, and vice versa.” For the purpose of this declaration, I will apply the same 

definition of the level of skill of a POSITA. 

29. Based on my experience, education, and training, I met the definition of 

a POSITA in December of 2002, the time of filing of the application that issued as 

the ’879 Patent. I also had greater knowledge and experience than a POSITA. I 

worked with POSAs in 2002, and I am able to render opinions from the perspective 

of a POSITA based on my knowledge and experience. My opinions concerning the 

’879 Patent claims and the prior art are from the perspective of a POSITA, as set 

forth above. 

30. As further discussed below, my opinions as stated in this declaration 

are valid even if the Board adopts a slightly different level of ordinary skill in the 

art. 

C. My Understanding of Legal Standards

31. I understand that a patent claim is unpatentable if the claimed 

invention would have been obvious to a person of ordinary skill in the art at the 

time of the purported invention. 

32. I understand that an obviousness analysis involves comparing a claim 

to the prior art to determine whether the claimed invention would have been 

obvious to a person of ordinary skill in the art at the time of the invention in view 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 13 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 13 of 49
11

of the prior art and in light of the general knowledge in the art as a whole. I also 

understand that obviousness is ultimately a legal conclusion based on underlying 

facts of four general types, all of which must be considered: (1) the scope and 

content of the prior art; (2) the level of ordinary skill in the art; (3) the differences 

between the claimed invention and the prior art; and (4) any objective indicia of 

non-obviousness.

33. I also understand that obviousness may be established under certain 

circumstances by combining or modifying the teachings of the prior art. Specific 

teachings, suggestions, or motivations to combine any first prior art reference with 

a second prior art reference can be explicit or implicit, but must have existed 

before the date of purported invention. I understand that prior art references 

themselves may be one source of a specific teaching or suggestion to combine 

features of the prior art, but that such suggestions or motivations to combine art 

may come from the knowledge that a person of ordinary skill in the art would have 

had. 

34. I understand that a reference may be relied upon for all that it teaches, 

including uses beyond its primary purpose, but also including teachings that lead 

away from the invention. I understand that a reference may be said to teach away 

when a person of ordinary skill, upon reading the reference, would be discouraged 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 14 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 14 of 49
12

from following the path set out in the reference, although the mere disclosure of 

alternative designs does not teach away.

35. I further understand that whether there is a reasonable expectation of 

success from combining references in a particular way is also relevant to the 

analysis. 

36. I understand that it is improper to use hindsight to combine references 

or elements of references to reconstruct the invention using the claims as a guide. 

My analysis of the prior art is made from the perspective of a person of ordinary 

skill in the art at the time of the invention. 

37. I am not offering any legal opinions in this declaration nor am I 

qualified to do so. I only consider such legal standards in framing my opinions and 

conclusions as well as placing assertions made by Petitioner in the Petition into the 

proper context. Additionally, from a subject matter perspective, I understand that 

the petitioner always has the burden of persuasion regarding a challenge of 

patentability of an invention under an inter partes review. 

IV. OPINIONS

A. Objective Evidence Of Non-Obviousness.

38. I understand that Neonode’s N1 phone was introduced in Spring 2002 

and commercially sold starting in 2004. I further understand that Neonode’s N2 

phone was sold starting in 2007. Ex. 2022, ¶ 6. 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 15 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 15 of 49
13

39. I have reviewed Neonode’s promotional material, which highlight the 

phone’s swipe-based user interface. Neonode specifically touted its “specially 

designed interface” that allows “you to easily access the different applications with 

simple sweeping gestures ... on the screen.” Ex. 2008 [N2 Advertisement Video] 

(00:27-00:35); see also id., (00:45-00:51) (“And you can easily access all of the 

Neonode N2’s content using the seven available sweeps.”). As Neonode explained, 

“there is nothing else you need other than your intuition.” Id., (01:25-01:27). 

40. From my review of Neonode’s promotional video and other materials, 

the “swipe” gesture of Neonode’s user interfaces in the N1 and N2 phones is covered 

by claim 1. The claimed inventions concern a user interface for a mobile handheld 

computer unit that includes a touch sensitive area that includes a representation of a 

function wherein the representation consists of only one option for activating the 

function wherein an object (e.g., a finger) touches the touch sensitive area where the 

representation is provided and then the “object,” the finger in our example, “glid[es]

along the touch sensitive area away from the touched location, wherein the 

representation of the function is not relocated or duplicated during the gliding.” I 

have also reviewed the Shain Declaration (Ex. 2023), and his testimony further 

confirms that the N1 and N2 devices practiced the claim 1 limitations: 

Both the Neonode N1 and N2 presented three icons in a strip along the 

lower edge of the display immediately following unlocking of the 

phone. One of the icons represented the Start Menu, one represented 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 16 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 16 of 49
14

the Keyboard Menu, and the third represented the Tools Menu. Each of 

the icons consisted of only one option for activating the associated 

function. Each of the icons were activatable by a gesture in which a 

thumb or finger touches the icon, and swipes up toward the center of 

the screen before lifting off of the screen. None of the icons were 

relocated or duplicated during the swiping gesture.

41. The Applicant also equated the “gliding ... away” motion with 

“swiping.” Ex. 1003 [Prosecution-History] 269 (“a finger touches a touch-sensitive 

screen at a location where an icon for a function is displayed, and then 

rubs/swipes/glides along the touch screen away from the location without lifting the 

finger.”). The Applicant also specifically referenced and provided a link to its 

promotional video for a commercial embodiment, the Neonode N2 phone, and asked 

the Examiner to “view the demonstration video ... prior to reviewing Applicant’s

arguments ....” Ex. 2035 [2008-03-14 Office-Action-Response] 15-16; Ex. 2008 

[N2-Advertisement-Video]. As the screen shots below from the video show, the 

“gliding ... away” gesture is similar to what many of today’s systems refer to as a 

“swipe” gesture and is distinct from a drag-and-drop operation. Specifically, the 

thumb is placed on a representation of a function (an arrow) and through a swiping 

motion, the menu screen opens:

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 17 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 17 of 49
15

See Ex. 2008 [N2-Advertisement-Video ] (screenshots from 00:26-00:27). 

42. Such gliding corresponds to what is shown, for instance, in Figure 2 

which shows a thumb gliding along the touchscreen:

43. I further have reviewed material showing praise for Neonode’s swiping 

user interface. For example, Pen Computing Magazine described Neonode N1 

phone’s swipe as “simple and brilliant” and different from the “dreaded gestures” of 

the pen computing devices (like Petitioner’s Hirayama-307):

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 18 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 18 of 49
16

Swipe, swipe, swipe 

You see, instead of the usual menus and pulldowns, most operations are 

performed by sweeps of your finger—usually your thumb—across the 

surface of the Neonode’s display. [...] If this sounds like the dreaded 

“gestures” that never really caught on in pen computing, it’s not. The 

swipes are much simpler, there are only a few, and they are consistently 

used throughout all applications. The idea here is to let you hold a 

phone in the palm of your hand and operate it entirely with your thumb. 

No need to push buttons, view tiny menus, pull out a tiny stylus, or use 

scroll wheels, rockers or other such vexing miniature controls. [...]

Neonode’s swiping interface is [] simple and brilliant. 

Ex. 2012, 2-3; id., 5 (“What’s the bottomline? The Neonode phone is quite 

obviously unique, ... The user interface is compelling and it’s easy to see how just 

a bit more development could provide almost total consistency and thus a user 

experience simpler than pretty much anything else that comes to mind. The speed 

is simply amazing. That’s the way a phone should operate.”).

44. I have also reviewed material showing praise of Neonode’s swipe-based 

user interface by various technology observers. Exs. 2014 (“I’ve been playing with 

my N1m on and off, and I’m very impressed! It’s definitely a best kept secret device 

– Neonode’s touch-based user interface with gesture recognition ... is extremely 

intuitive ...”); 2016 (“[The N2] has the most advanced touchscreen available, and 

has no buttons ... ‘Neonode N2 is designed for advanced simplicity. You do 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 19 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 19 of 49
17

everything on-screen, simply and conveniently, with just one finger, Infibeam says. 

‘The combination of an optical touch screen and specifically designed user interface 

makes access to all features and content of your Neonode N2 both quick and easy.’”); 

2015, 2017; Ex. 2021 [Hunting the iPhone killer; Swedish Neonode generates buzz 

for device] (“the N2 from Neonode Inc. – is the strongest contender for the title of 

‘iPhone killer,’ ... ‘They’ve come out with a kick-ass device’... the [N1’s] screen 

reacts to the intuitive passage of a finger over the screen to initiate basic phone, Web 

browser and multimedia functions.”).

45. I have also reviewed testimony that Senior management at Samsung’s 

mobile telecom division were extremely impressed by Neonode’s N1, and in early 

2005 began discussions with Neonode about licensing the N1’s gesture-based user 

interface and touch screen technology. Ex. 2026, ¶ 9. Ki-Tai Lee (K. T. Lee), head 

of Samsung’s mobile telecom division, presciently told Neonode that he believed 

Neonode’s intuitive user interface was “the future of mobile phones.” Id. Neonode 

had many hours of meetings with Samsung, including a meeting in London, attended 

by Marcus Bäcklund, Thomas Ericsson, and Per Bystedt. Id. Mr. Lee told 

Samsung’s negotiators—in Neonode’s presence—that “we need this,” referring to 

the Neonode’s N1 gesture-based user interface and the license for the user interface. 

Id. I further understand that Samsung subsequently signed a licensing agreement 

with Neonode in 2005, and the licensing agreement covered, among other things, 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 20 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 20 of 49
18

the application that ultimately issued as the ’879 patent. Ex. 2026, ¶ 10; Ex. 2024, 

¶¶ 13-14.

46. I have further reviewed evidence demonstrating the belief among the 

technology observers that when Apple introduced the first iPhone in 2007, its 

swiping gestures resembled that of Neonode. For example, Pen Computing 

Magazine wrote:

Listening to Apple’s claims of all the patents covering the iPhone’s user 

interface one might assume the iPhone broke completely new ground 

and went where no phone had ever gone before.

That is not entirely so. Neonode, a small Swedish company ... 

announced the Neonode N1 back in 2002. ... It did not use a stylus 

either. Instead, it used a swipe and tap system on a novel touch screen 

that used a grid of infrared beams to sense finger movement. 

... And if the iPhone’s swipes and taps seem futuristic, they are not. 

Neonode has been using them since the first N1 came out. In fact, the 

company’s Neno user interface is based entirely on swipes and taps. 

Ex. 2013, 1. The author followed, 

[I]t must be vexing to see Apple essentially claim ownership of 

concepts the Neonode phone has been using for at least five years. 

Ex. 2013, 9. 

47. I have also reviewed a Ph.D. dissertation and a Master’s thesis that 

described Neonode as “the first smartphone to use a touchscreen as primary input 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 21 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 21 of 49
19

and to support touch gestures for several functions,” (Ex. 2018, 9), and “The 

Neonode N1 was the first commercially available mobile device to make extensive 

use of swipe gestures appropriate for one-handed use, including a browser that 

scrolled content vertically with swipes,” (Ex. 2020, 8).

48. I have also reviewed testimony about commercial sales of Neonode 

phones, explaining that Neonode sold tens of thousands of its N1 and N2 phones to 

various operators around the world, including Mexico, Belgium and India. Ex. 2022, 

¶ 6; Ex. 2025; Ex. 2028; Ex. 2026, ¶ 11; Ex. 2024, ¶¶ 8-10. I understand that, as a 

small startup company without the backing of any major carrier, and with limited 

manufacturing experience, Neonode phones were priced up to $1,000, which is 

many times more expensive than the typical luxury phones of its time. Id. 

49. The above evidence, showing praise by industry observers, competitors 

and users, further support my conclusions below that the claims were not obvious at 

the time of the invention.

B. Claim 1 (And All Dependents): “Gliding ... Away”

50. The claims require “activating [a] function” via a “multi-step operation 

comprising (i) an object touching the touch sensitive area at a location where the 

representation is provided and then (ii) the object gliding along the touch sensitive 

area away from the touched location ...” Ex. 1001 [’879] cl. 1. 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 22 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 22 of 49
20

51. The pending claims during prosecution originally recited “moving in a 

direction from a starting point that is the representation [of a function] ... to said 

display area.” Ex. 1003 [Prosecution History] 326. However, “in accordance with 

the conclusions of [an examiner] interview” and in order to “properly claim the 

present invention,” the claims were amended to recite the specific gesture of “gliding

... away from the location [of the representation of a function].” Id., 343. Thus, the 

claim was altered from any “moving in a direction from a starting point” to a specific 

gesture, “gliding ... away” to “properly claim the present invention.” 

52. The Applicant also equated the “gliding ... away” motion with 

“swiping.” Ex. 1003 [Prosecution History] 269 (“a finger touches a touch-sensitive 

screen at a location where an icon for a function is displayed, and then 

rubs/swipes/glides along the touch screen away from the location without lifting the 

finger.”). The Applicant also specifically referenced and provided a link to its 

promotional video for a commercial embodiment, the Neonode N2 phone, and asked 

the Examiner to “view the demonstration video ... prior to reviewing Applicant’s

arguments ....” Ex. 2035 [2008-03-4 Office-Action-Response] 15-16; Ex. 2008 [N2 

Advertisement Video]. As the screen shots below from the video show, the “gliding 

... away” gesture is similar to what many today’s systems refer to as a “swipe” 

gesture and is distinct from a drag-and-drop operation. Specifically, the thumb is 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 23 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 23 of 49
21

placed on a representation of a function (an arrow) and through a swipe motion, the 

menu screen opens:

See Ex. 2008 [N2 Advertisement Video] (screenshots from 00:26-00:27).

53. The specification similarly demonstrates a swipe motion. See Ex. 1001 

[’879] Fig. 2, 4:7-11.

54. During prosecution, the Applicant also made it clear that the claimed 

“gliding ... away” is distinct from “drag-and-drop”:

Hoshino does not teach gliding a finger away from an icon. Instead, 

Hoshino teaches a drag-and-drop operation for moving an icon.

Ex. 1003 [Prosecution-History] 171. 

55. In accord, the Applicant distinguished Hoshino’s “conventional” “dragand-drop” “operation” from the “novel” “touch-and-glide” operation of the 

“[c]laimed invention”:

Id., 170.

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 24 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 24 of 49
22

56. I understand that when an Applicant makes unequivocal statement about 

what is not included in the scope of the claim, courts have found such statements to 

be prosecution disclaimers, which means that the claim should be interpreted to 

exclude the disavowed aspect. I further understand that regardless of whether or not 

the applicant’s statements during prosecution raise to the level of prosecution 

disclaimer, they are relevant in informing the POSITA regarding the meaning and 

scope of the claims.

57. Based on my overall review of the claim language, the specification and 

the prosecution history, it is my opinion that a POSITA would have understood that 

“gliding ... away” does not encompass a drag-and-drop operation.

58. The Petition relies on Hirayama-307 alone for the disclosure of the 

“gliding ... away” limitation. Pet., 58-59. Hirayama-307 was already analyzed by 

the Examiner, who explained that Hirayama-307 is “pertinent to applicant’s 

disclosure,” as it “teaches a method of activating functions.” Ex. 2009 [2006-03-23 

Non-Final Rejection] 15. Yet, the Examiner never relied upon Hirayama-307 as a 

basis of rejection.

59. Hirayama-307’s operation is a “conventional” drag-and-drop 

“operation” referenced by the Applicant during prosecution. Specifically, when the 

user wishes to use Hirayama-307’s dialing application, he/she moves the stylus to 

the application icon 41. Ex. 1006 [Hirayama-307] 2:1-4; 5:30-32. The user then 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 25 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 25 of 49
23

drags the icon outside of the hatched area, into the non-hatched area. Id., 2:5-8; 

5:39-53. As the icon 41 is dragged outside of the hatched area, it is “enlarged as a 

window 43.” Id., 2:8-13; 5:59-66. The enlarged window 43 is then placed (i.e., 

dropped) at the location within the non-hatched area where the user lifts the stylus. 

Id., 2:8-13; 5:59-66. 1 I have visually demonstrated Hirayama’307’s drag-and-drop 

process in the figure below that I have created based on Hirayama-307’s written 

description of its method:

60. That Hirayama-307 changes the size of icon 41 into an enlarged window 

43 does not change the nature of its operation as a drag-and-drop operation. Many 

1 While Hirayama-307 states that the “enlarg[ing]” of icon 41 into window 43 can occur either 

“if the pen coordinate is considerably shifted from the reference moving amount,” or “if the pen 

coordinate is outside of the predetermined designated area (e.g., the hatched area in this 

embodiment)” id., 5:59-63, a POSITA would understand that only the latter alternative is 

feasible in the context of Hirayama-307’s disclosed user interface. This is because the icon 

41/window 43 follows the stylus so that the enlarged window 43 can be located at the position 

identified by the user, id., 2:8-13; 5:59-66, and, the enlarged window cannot be placed within the 

hatched area of Hirayama-307. 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 26 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 26 of 49
24

drag-and-drop operations may result in a change in the graphical user display of the 

item being dragged as a result of drag-and-drop. For example, a file may be dragged 

across the screen into an open application window, which results in the file opening. 

As another example, when a file is dragged from the desktop into the trash bin of an 

operating system, the icon may change its appearance inside the trash bin. As 

another example, the prosecution Hoshino (see infra) also opened a new window as 

a result of the drag-and-drop gesture. 

61. It should also be noted that while Petitioner disputes whether Hirayama307 provides feedback to the user by visually showing icon 41 as being dragged 

during the entirety of the dragging process, Pet., 60-62, this is irrelevant to the fact 

that Hirayama-307’s gesture is a drag-and-drop operation. From the perspective of 

the user, some form of Hirayama-307’s dialing application is logically dragged (and 

behaves as if it is being logically dragged) with the movement of the stylus, and is 

dropped at the location where the stylus leaves the screen. It is true that, as I will 

explain in paragraphs 81-82, it was (and is) preferable in most instances to have an 

icon be visually shown as moving or being duplicated during “drag-and-drop” as the 

user moves the stylus/finger in order to provide feedback to the user. However, if a 

GUI, for any reason, does not provide interim feedback to the user by visually 

showing the icon actually moving with the stylus/finger, that does not change the 

nature of the operation as a drag-and-drop operation.

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 27 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 27 of 49
25

62. Hirayama-307’s drag-and-drop operation is also functionally identical 

to the drag-and-drop operation disclosed by the prosecution Hoshino reference to 

activate an icon, which the Applicant during prosecution explained “does not teach 

gliding a finger away from an icon,” but “[i]nstead, Hoshino teaches a drag-anddrop operation for moving an icon.” Ex. 1003 [Prosecution History] 171. 

Specifically, as shown in the schematic annotation of Hoshino Fig. 19 below, item 

23 in Hoshino is music content which the user may wish to, for example, play or 

purchase. Ex. 2010 [Hoshino] ¶ [0110]. The user activates the file for playback or 

purchase by dragging it into the corresponding box that states “purchase” or “listen 

for trial.” Ex. 2010 [Hoshino] ¶¶ [0111]-[0114]. 

Hoshino Fig. 19 (partial)

63. As is apparent from the above, Hoshino’s user activates a music icon by 

dragging the icon to a designated area, resulting in a larger window opening. 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 28 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 28 of 49
26

64. This is basically the same as Hirayama-307’s operation. Hirayama307’s user similarly opens an icon 41 window by dragging the icon into the nonhatched area at the location where the user wishes the enlarged window to be placed. 

65. The distinction between “gliding ... away” and a drag-and-drop gesture 

is material, even though they may have overlapping movements. In the field of 

human computer interaction, even small differences between gestures can have 

substantial consequences. Notably, and as discussed in greater detail in paragraphs

38-49 Neonode’s N1 and N2 phones were widely praised for their intuitive gliding 

feature. See also, e.g., Exs. 2012, 2013 (praising the swiping feature, calling it 

“simple and brilliant”). It is most unlikely that Neonode’s phones would have 

received such praise if they replaced their seamless gliding functionality with a 

cumbersome drag-and-drop operation as shown in Hirayama-307. 

66. That small differences matter is also evident in Petitioner’s Ren 

reference. Two of the gestures analyzed by Ren are “Slide Touch” and “Slide Off.” 

Ex. 1004 [Ren] 389-390. Even though both gestures include dragging a stylus on 

the screen to select an item, they are different in that in a “Slide Touch,” the stylus 

lands outside of the icon, is dragged to the icon, and then removed from the screen. 

Id. In contrast, in a “Slide Off,” the stylus lands on the icon, and is dragged outside 

of the icon, and is then removed from the screen. Id. Therefore, the two gestures 

can be identical in the stylus movement, but different only in where the stylus lands 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 29 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 29 of 49
27

on, and is removed, from the screen. Despite this seemingly small difference, Ren 

was able to objectively and conclusively determine that “[t]he results showed that 

the Slide Touch strategy was the best of the six, in terms of selection time (Figure 

5), error rate (Figure 6), and subject preference (Figure 9).” Id., 402. 

67. Similarly, even though both the claimed “gliding ... away” gesture and 

a drag-and-drop operation may include movement of the stylus/finger on the screen, 

which may even happen to start and end at similar positions on the screen, they are 

fundamentally different with pronounced differences for the user. In a drag-anddrop operation, the user generally perceives some form of an object/function as 

behaving as if it is being dragged by the movement of the stylus/pen. Sometimes an 

operating system provides visual feedback by actually showing the object moving 

on the screen together with the stylus/pen. 

C. Claim 1 (And All Dependents): “Wherein The Representation Of 

The Function Is Not Relocated Or Duplicated”

68. The claims require that “the representation of the function is not 

relocated or duplicated during the gliding” of the object (e.g., finger). Petitioner 

presents two alternative theories for why this limitation is obvious. First, the Petition 

relies on Hirayama-307 alone to argue that this limitation was obvious. Pet., 60. 

Alternatively, the Petition argues that a POSITA would have been motivated to 

import Ren’s “Slide-Off” selection technique into Hirayama-307 instead of 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 30 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 30 of 49
28

Hirayama-307’s drag-and-drop operation. As I will explain below, both theories are 

incorrect.

1. Disclosure By Hirayama-307 Alone.

69. As shown in the annotated figure blow, the Petition (at 53, 60) maps 

Hirayama-307’s icon 41 (green) for a telephone application to the claimed 

“representation of a function,” and maps (Pet., 58-60) the user’s drag-and-drop 

operation (red arrow) to enlarge icon 41 into the window 43, and position the 

window 43 on the screen, to the claimed “gliding ... away” limitation: 

70. The Petition then argues that “[a] POSA would have recognized 

Hirayama307 does not describe or show icon 41 (‘representation’) is dragged or 

otherwise relocated or duplicated during the movement of pen 3.” Pet., 60. The 

Petition is incorrect.

71. I will now explain the operation of Hirayama-307, and why the selected 

icon (e.g., icon 41) is “relocated or duplicated” at least for a portion of the drag-andCase 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 31 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 31 of 49
29

drop operation. For the convenience of the reader, my discussion is with reference 

to the schematics below, which I have prepared based on Hirayama-307’s written 

description of its system:2

2 The movement of the stylus in the figures is initially shown to the left, before the stylus is 

brought down into the non-hatched area, in order to more easily show the duplication/relocation 

of icon 41 while the pen is in the hatched area.

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 32 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 32 of 49
30

72. In order to utilize a particular application, the user has to drag the icon 

for that application from its location in the hatched area and drop it inside the nonhatched area at a location where the user wishes the enlarged window of the 

application to be located. For example, when the user wishes to use Hirayama-307’s

dialing application (represented by icon 41), the user moves the stylus to the 

application icon 41. Ex. 1006 [Hirayama-307] 2:1-4; 5:30-32. The user then must 

drag icon 41 into the non-hatched area of the screen. Id., 2:5-8; 5:39-53. As the 

icon 41 is dragged outside of the hatched area, “a circuit converts the icon into a 

window.” Id., 2:8-13; 5:59-66 (“it is enlarged as a window 43”). The enlarged 

window 43 is then placed (i.e., dropped) at the location within the non-hatched area 

where the user lifts the stylus. Id., 2:8-13; 5:59-66.3

73. Hirayama-307 expressly discloses that as icon-41 is being dragged, and 

prior to its conversion into window 43, it moves with the movement of the pen 3. 

Specifically, as highlighted portion of Hirayama-307, 2:1-13 shows, the pen first 

touches the “coordinate position in which a desired icon [e.g., icon 41] is displayed” 

(yellow), then as the pen moves, “the icon [41] display coordinate position” moves 

3 While Hirayama-307 provides both disclosures that the icon can be converted into a window 

either upon lift off of the pen (Hirayama-307, 2:8-13) or upon the pen leaving the hatched screen 

area (id., 5:58-66), that distinction is not relevant here. For the purposes of this discussion, as 

discussed further below, the relevant fact is that Hirayama-307 is clear that icon 41 moves with 

the pen prior to being converted into a window.

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 33 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 33 of 49
31

with the pen (green), and then icon 41 is “convert[ed]” “into a window” (e.g., 

window 43) in the non-hatched display area (blue).

74. This relocation or duplication is schematically shown in the modified 

version of Hirayama’s figure which depicts the green icon at the point of the red 

arrow is the “moving” location of the original icon 41 as it is being dragged:

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 34 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 34 of 49
32

75. Other disclosures in Hirayama-307 confirm the above. For example, 

Hirayama-307 also discusses a reverse operation, where an open window is draggedand-dropped into its application icon position in order to close the window. Id., 

6:22-7:6; Fig. 4B. In discussing the reverse drag-and-drop operation for closing an 

open window, Hirayama-307 explains that the open window is dragged and dropped 

“to the predetermined vacant position,” referring to the position of the application 

icon corresponding to the window. Id., 7:3-6; Fig. 4B (step S10). This disclosure 

confirms that application icon 41 was “relocated” when it was opened by the dragand-drop operation as its location is now “vacant.” The location would not be 

“vacant” if the icon had not moved.

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 35 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 35 of 49
33

76. Furthermore, in the same reverse drag-and-drop to close an open 

window, Hirayama-307 similarly explains that “the window display coordinate 

position is moved in accordance with the movement of the position coordinate of the 

point of the pen.” Ex. 1006 [Hirayama-307] 2:24-27. This is precisely the reverse

of the procedure to create a window 43, where the application icon 41 moves with 

the movement of the pen prior to its conversion into a window, id., 2:5-8, 

demonstrating that Hirayama-307 takes the same approach to both operations.

77. It would also be common sense for the POSITA that Hirayama-307 

would “relocate” icon 41 and leave its location vacant while its window 43 is open. 

Specifically, in the context of the small application icons of Hirayama-307, it may 

be challenging for the user to determine which specific one of the numerous 

application icons 40 corresponds to the currently open window, so that the open 

window can be dragged to the position of its corresponding application icon in order 

to close the window. Thus, Hirayama-307 appears to solve this problem by having 

the application icon position “vacant” while the application has an open window in 

order to let the user know that the vacant position corresponds to the currently open 

window.

78. As yet another supporting disclosure, Hirayama-307 routinely states that 

the application window 43 is created by “enlarging” the dragged application icon 

41. See, e.g., id., 5:39-40 (“icon is enlarged in the form of window”); 5:64-66 (“the 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 36 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 36 of 49
34

processing display form of the icon 41 designated is enlarged as a window 43”); 

7:14-15 (“the icon position to be enlarged”); 2:10-11 (“a circuit converts the icon 

into a window”). This disclosure further confirms that icon 41 is moved with the 

stylus/pen during the drag-and-drop process, and it is “enlarged” (or “converted”) 

into a window when the proper trigger happens. If the icon 41 did not move with 

the movement of the pen to then be “enlarged” or “converted” into window 43, 

Hirayama-307 would have referred to the process as “creating” or “opening” a 

window.

79. Furthermore, it is notable that there is no disclosure in the text of 

Hirayama-307 that contradicts its disclosures that the application icon is moved with 

the pen prior to being enlarged into the window. For example, Ex. 1006 [Hirayama307] 5:16-67 explains the flow diagram of logical determinations of its drag-anddrop operation, but is silent on whether icon 41 is duplicated or moved during the 

dragging process within the hatched area.

80. As I explain below, Petitioners’ reliance on Hirayama-307’s Fig. 3A and 

its related text for the proposition that icon 41 is not relocated or duplicated during 

the drag-and-drop operation is misplaced. Pet., 60-61. A POSITA would understand 

that Fig. 3A represents the state of the device before icon 41 is being dragged. In 

connection with Figs. 3A and 3B, Hirayama-307 explains that “[i]f the pen 

coordinate is considerably shifted from the reference moving amount, or if the pen 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 37 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 37 of 49
35

coordinate is outside of the predetermined designated area (e.g., the hatched area in 

this embodiment),” then icon 41 is “enlarged as a window 43 as shown in Fig. 3B.” 

Ex. 1006 [Hirayama-307] 5:59-66. In Fig. 3A, however, the tip of the pen is both 

outside of the hatched area, and has considerably shifted as it is well into the active 

screen area, but there is no enlarged window:

Id., Fig. 3A. Thus, a POSITA would understand that if Fig. 3A was intended to 

show the state of the screen during a drag-and-drop operation, then it should have 

also shown an enlarged window 43. Instead, Fig. 3A only shows “a cross-shaped 

position designating cursor 42,” which is shown “as the point of the pen 3 approaches 

the panel surface of the display portion 1”—i.e., before the pen has selected the icon 

41 to initiate the process of drag-and-drop. Id., 4:65-68. 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 38 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 38 of 49
36

81. Furthermore, the written description corresponding to Fig. 3A is clear 

that the figure is intended to show the state of the device at power-on, explaining 

that “when the power switch 10 shown in Fig. 1 is depressed, icon groups 40 which 

make various processing possible are displayed on the display portion 1 as shown 

in FIG. 3A.” Ex. 1006 [Hirayama-307] 4:58-61. Thus, even though Fig. 3A shows 

a pointed arrow from icon 41 to the location of the point of the pen, a POSITA would 

understand that the arrow is likely intended to show how the device is going to be 

operated on power-on, and not a depiction of the screen during a drag-and-drop 

operation.

2. Obviousness In View Of Hirayama-307 Alone.

82. As I will explain below, a POSITA would not have been motivated to 

implement a drag-and-drop operation (such as Hirayama-307’s) without relocating 

or duplicating the icon being dragged during the drag process. 

83. In graphical user interfaces, it is important to provide visual feedback to 

the user during an operation. This helps, for example, inform the user that the 

operation is in fact being successfully performed. It also helps design the graphical 

user interface as close as possible to real life experiences outside of the virtual world, 

and give the user a real-life “feel.” This is an important concept in GUI design. In 

the context of a dragging operation, this feedback mechanism was, and continues to 

be, generally provided by visually showing the icon being moved or duplicated 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 39 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 39 of 49
37

across the screen during the drag operation. This would help the user receive 

feedback that, as the user drags an icon with the mouse/stylus/finger, the icon is in 

fact being dragged and the drag-and-drop operation is successfully in progress. This 

would also help the user get a real life “feel” for the drag-and-drop operation by 

visually seeing an icon being dragged.

84. Further supporting the above, the visual presentation of a “dragging” 

icon was also common in both the Microsoft Windows and Macintosh MacOS 

environments. For example, the graphic below demonstrates a file icon being 

dragged into a blank Microsoft Word active window in a MacOS Monterey 

operating system, so that the file can be opened. As can be seen, while the file icon 

is being dragged, and before it is dropped into the active MS Word window to be 

opened, the file icon is duplicated:

85. Thus, there simply is no reason for a POSITA to implement Hirayama307’s drag-and-drop process, but avoid the industry-standard method of providing 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 40 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 40 of 49
38

user feedback by “relocating or duplicating” the icon during the drag-and-drop 

operation.

86. In relying on Hirayama-307’s cursor, Dr. Bederson appears to suggest 

that the cursor itself is the only feedback provided to the user during the drag-anddrop operation in Hirayama-307, and, therefore, there is no need to relocate or 

duplicate the icon 41 as it is being dragged. Ex. 1002 [Bederson-Decl.] ¶¶ 158-159. 

That is not correct. Hirayama-307’s text only discloses that the cursor appears on 

the screen “as the point of the pen 3 approaches the panel surface of the display,” 

Ex. 1006 [Hirayama-307] 4:65-68—i.e., before the pen has started the drag-anddrop operation. In other words, the cursor is used to assist the user in moving the 

pen to the location of the icon, just as a mouse cursor on a desktop display moves on 

the screen to assist the user in locating the mouse pointer. Hirayama-307 does not 

state that the cross-shaped pointer continues to appear on the screen during the dragand-drop operation instead of the image of the icon 41 actually being dragged. 

Rather, as noted above, Hirayama-307 states that icon 41 itself moves with the pen 

during the dragging process. Id., 2:5-8. 

87. Furthermore, a POSITA would understand that Hirayama-307’s cursor 

is insufficient to provide feedback to the user during the drag-and-drop operation, 

and that there is no motivation to eliminate the typical “relocation or duplication” of 

the icon during a drag-and-drop operation in reliance on the existence of the cursor. 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 41 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 41 of 49
39

This is because, even if Dr. Bederson was correct that the cursor appears on the 

screen during the drag-and-drop operation, the cross-shaped pointer also appears on 

the screen prior to the pen/stylus initiating the drag-and-drop operation. Ex. 1006 

[Hirayama-307] 4:65-68. In other words, the appearance of the cross-shaped cursor 

does not denote to the user anything about the drag-and-drop operation, but simply 

that the pen is communicating with the screen, and the location of the tip of the pen. 

Thus, if all the user sees on the screen is a pen with a cross-shaped cursor after the 

user has initiated a drag-and-drop operation, the user would not know if the dragand-drop operation is being successfully performed as the user drags the pen. All 

the user can learn from the position of the cursor is that the pen itself is moving 

across the screen. This is shown schematically below:

88. This is why, again, a cursor moving alone is typically insufficient to 

provide feedback for a drag-and-drop operation.

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 42 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 42 of 49
40

3. Obviousness By Hirayama-307 In View Of Ren

89. Petition also argues that “it would have been obvious to combine the 

teaching of Ren with Hirayama307” by “implement[ing] Ren’s selection 

techniques” in Hirayama-307’s device. Pet., 62. Petitioner proposes that a POSITA 

would have been motivated to utilize a specific variant of one of six categories of 

gestures disclosed by Ren instead of Hirayama-307’s drag-and-drop operation. Pet., 

30-31, 62. As a preliminary matter, a POSITA would not have been motivated to 

implement a drag-and-drop operation in Hirayama-307 without “relocating or 

duplicating” as a form of feedback to the user, for the reasons already explained in 

the previous section. Furthermore, as explained here, a POSITA would not have 

been motivated to do so also in view of Ren. 

90. Ren discloses six categories of selection strategies: Direct On, Slide 

Touch, Direct Off, Slide Off, Space On, and Space Touch. Ex. 1004 [Ren] 389-391. 

Under each of these general selection strategies, Ren discloses one or more variants. 

Id. The Petition relies on a specific variant of the “Slide Off” category, referred to 

as “a à c à b à a.” Ex. 1002 [Bederson-Decl.] ¶¶ 160-161. Ren’s six selection 

categories (blue), and the specific gesture relied upon by Petitioner (red), are shown 

below in the annotated Fig. 3 from Ren:

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 43 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 43 of 49
41

91. The Petition proposes the following reason to substitute Hirayama307’s drag-and-drop operation with a variant of Ren’s Slide-Off strategy:

For example, Ren and Hirayama307 both are directed to solutions to 

the same problem, namely target selection techniques in pen-based 

tablet systems. As another example, a POSA would have recognized 

Ren as disclosing a small number of selection techniques that would 

have been obvious to try and implement with pen-based GUI 

interaction systems.

Pet., 62. Petitioner is incorrect.

92. Hirayama-307 sets out to create a pen-based operation that can replace 

two mouse clicks required in mouse-operated systems to enlarge a desired window 

and place it at a desired location on the desktop. Ex. 1006 [Hirayama-307] 1:14-35. 

Thus, as previously explained, Hirayama-307 discloses a drag-and-drop operation 

where the pen touches a desired icon, drags the icon from the hatched area to the 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 44 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 44 of 49
42

non-hatched area of the screen, and drops the enlarged icon at the location where the 

user desires to place it on the screen. Id., 5:16-6:21. 

93. There is no apparent deficiency in Hirayama-307 that a combination 

with Ren would remedy and Petition cites none. In fact, as explained below, the 

Petition’s proposed combination only deteriorates Hirayama-307. Specifically, 

Petitioner proposes that a POSITA would have combined Hirayama-307 with Ren 

so that Hirayama-307’s drag-and-drop would not “relocate or duplicate” icon-41 

during the dragging process. Pet., 62. However, as explained, the user feedback of 

relocating or duplicating an icon during the drag of a drag-and-drop operation was, 

and continues to be, significant as also implemented by main-stream systems such 

as MS Windows and MacOS and eliminating user feedback would worsen the user’s 

experience.

94. As I will explain now, Petitioner’s “obvious-to-try” rationale is 

similarly incorrect. First, the assumptions that there are only six (or a similarly 

small) number of possible selection gestures such that a POSITA would set out to 

try all possible gestures is incorrect. I note that Ren states that “[t]heoretically, an 

infinite range of selection strategies exists.” Ex. 1004 [Ren] 6. Nor has Petitioner 

provided any reason as to why a POSITA would be motivated to try all possible 

gestures so that I can analyze and respond.

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 45 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 45 of 49
43

95. Furthermore, even in the context of selection techniques disclosed by 

Ren, a POSITA would not have been motivated to specifically utilize the “a à c à

b à a” variant of the “Slide-Off” strategy as Ren expressly states that, both 

objectively and subjectively, another strategy, “Slide Touch,” is superior.

96. Ren performed two sets of experiments “to determine the best individual 

strategy and the best strategy group.” Ex. 1004 [Ren] 392-93. At the conclusion of 

the two experiments, Ren unambiguously determined that the “Slide Touch” 

strategy—not the “Slide Off” strategy relied upon by Petitioner—is the “single best 

strategy”: 

Ex. 1004 [Ren] 412.

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 46 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 46 of 49
44

97. Ex. 1004 [Ren] 412. Ren’s overall conclusions are also supported by 

the underlying results from its experiments. For example, as shown in the 

reproduction of Ren’s Figs. 5 and 11, with respect to “Mean Selection Time,” 

Petitioner’s relied upon “Slide Off” strategy (red) came in third in the first (left) 

experiment and fifth in the second (right) experiment:

98. Similarly, as shown in the reproduction of Ren’s Figs. 6 and 12, with 

respect to “Mean Error Rate,” Petitioner’s relied upon “Slide Off” strategy (red) 

came in third in both the first (left) and second (right) experiments:

99. Finally, in addition to Ren finding objective superiority of the “Slide 

Touch” strategy to the exclusion of Petitioner’s relied upon “Slide Off,” Ren also 

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 47 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 47 of 49
45

found similar subjective preference by the users. In experiment 1, the users had a 

clear preference for “Slide Touch” when object sizes were relatively small. Id., 401-

402; Fig. 9. When the object sizes were relatively larger, “Slide Touch” and “Slide 

Off” were statistically tied. Id. “Slide Touch” was the overall preferred method 

under the first experiment. Id., 402-403 (“... the Slide Touch strategy was the most 

preferred ....”). Similarly in the second experiment, “the Slide Touch strategy was 

the most preferred.” Id., 409. 

100. Another reference relied upon by Petitioner (Allard) supports Ren’s

conclusion. Allard also uses the strategy referred by Ren as “Slide Touch.” 

Specifically, in Allard, the button that is last touched by the user is the button that is 

selected when the user lifts the stylus off the screen, whether the stylus originally 

landed outside of the button on the screen or on the button itself. Ex. 1010 [Allard] 

5:39-44. In fact, Allard explains that using the Slide-Touch strategy is preferred 

because “a user can easily recover from touching an unintended button by leaving a 

finger on the screen and sliding to another button or a non-button area.” Id., 5:45-

54. 

V. CONCLUSION

101. For the foregoing reasons, based on my expertise and experience and

the record of this case that I have reviewed, it is my opinion that the Challenged 

Claims are not shown to be disclosed or obvious.

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 48 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 48 of 49
46

102. I understand that my opinions discussed above support a legal 

conclusion that the challenged claims are nonobvious.

In signing this declaration, I recognize that the declaration will be filed as 

evidence in a contested case before the Patent Trial and Appeal Board of the 

United States Patent and Trademark Office. I also recognize that I may be subject 

to cross-examination in the case and that cross-examination will take place within 

the United States. If cross-examination is required, I will appear for crossexamination within the United States during the time allotted.

I hereby declare that all statements made herein of my own knowledge are 

true and all statements made herein on information and belief were and are 

believed by me to be true, and that all statements herein were and are made with 

the knowledge that willful false statements and the like so made are punishable by 

fine or imprisonment, or both, under section 1001 of Title 18 of the United States 

Code and that any such willful false statements may jeopardize the validity of the 

application or any patents issued thereon. 

 Respectfully submitted,

Dated: March 25, 2022 

____________________________

Craig Rosenberg, Ph.D.

Case 6:20-cv-00507-ADA Document 71-28 Filed 04/14/23 Page 49 of 49 Case 6:20-cv-00507-ADA Document 81-18 Filed 05/19/23 Page 49 of 49