Office Action 4: Final Rejection (2024-03-05)
Rejection types: 103 - Obviousness

CLAIMS:
AMENDMENTS TO THE CLAIMS

This listing of the claims will replace all prior versions and listing of claims in the application:

1\. (Currently Amended) A computer system that is in communication with a display generation component and an optical sensor, the computer system comprising:a first hardware input mechanism;
a second hardware input mechanism;
one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected;
while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture; and
in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is a first type of gesture, displaying, via the display generation component, an indication that the second user interface object is selected;
in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected; \[\[and\]\] in accordance with a determination that the hand gesture is a third type of gesture that is different from the first type of gesture and the second type of gesture, performing an operation corresponding to selection of the first user interface object; and

in accordance with a determination that the hand gesture is a fourth type of gesture that is different from the first type of gesture, the second type of gesture, and the third type of gesture, displaying, via the display generation component, a menu, wherein:a first control and a second control are available via the menu;
selection of the first control causes the computer system to perform an operation corresponding to a press of the first hardware input mechanism; and selection of the second control causes the computer system to perform an operation corresponding to a rotation of the second hardware input mechanism
.

2\. (Original) The computer system of claim 1, wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor.

3\. (Currently Amended) The computer system of claim \[\[1\]\] 4, wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture.

4\. (Currently Amended) The computer system of claim 1, wherein before detecting the hand gesture, the computer system is in a first operation mode, wherein performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode, wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture, and wherein the one or more programs further include instructions for:while the computer system is in the second operation mode, detecting, via at least the optical sensor, a second hand gesture; andin response to detecting, via at least the optical sensor, the second hand gesture:in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation;
in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation;
in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation; and in accordance with a determination that the second hand gesture is the fourth type of gesture, ending the automatic scrolling operation
.

5\. (Canceled)

6\. (Canceled)

7\. (Canceled)

8\. (Currently Amended) The computer system of claim \[\[1\]\] 4, wherein the automatic scrolling operation scrolls a sequence of a plurality of interface objects in a second direction, and wherein the one or more programs further include instructions for:while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects; and
in response to detecting the end of the sequence of the plurality of user interface objects,
performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction.

9\. (Canceled)

10\. (Currently Amended) The computer system of claim \[\[1\]\] 4, wherein the one or more programs further include instructions for:in accordance with a determination that the second hand gesture is the first type of gesture, displaying a notification that indicates a state of the automatic scrolling operation.

11\. (Currently Amended) The computer system of claim \[\[1\]\] 4, wherein the third type of gesture is a type of gesture that is multiple instances of the fourth type of gesture.

12\. (Previously Presented) The computer system of claim 1, wherein the fourth type of gesture does not include multiple instances of the first type of gesture and the third type of gesture does not include multiple instances of the second type of gesture.

13\. (Previously Presented) The computer system of claim 1, wherein the one or more programs further include instructions for:detecting, via at least the optical sensor, a third hand gesture; and

in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that a notification was received within a threshold period of time and in accordance with a determination that the third hand gesture is the fourth type of gesture, performing an action related to the notification.

14\. (Currently Amended) The computer system of claim \[\[4\]\] 1, wherein the menu includes one or more selectable options, wherein the one or more selectable options includes a first selectable user interface object for changing operations that one or more hand gestures can cause to the computer system to perform, wherein selection of the first selectable user interface object causes the computer system to display a plurality of settings, and wherein each setting controls an operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system.

15\. (Currently Amended) The computer system of claim \[\[4\]\] 1, wherein:the menu includes one or more selectable options;
the one or more selectable options includes a second selectable option for transitioning the computer system to a fourth operation mode and a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode;
selection of the second selectable option causes the computer system to transition to the fourth operation mode; and
selection of the third selectable option causes the computer system to transition to the fifth operation mode.

16\. (Currently Amended) The computer system of claim \[\[4\]\] 1, wherein the menu includes one or more selectable options, wherein the one or more selectable options include a fourth selectable option for displaying one or more additional selectable options, and wherein selection of the fourth selectable option causes the computer system to display the one or more additional selectable options that were not previously displayed before selection of the fourth selectable option.

17\. (Currently Amended) The computer system of claim \[\[4\]\] 1, wherein the menu is displayed:in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location; and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location.

18\. (Currently Amended) The computer system of claim \[\[4\]\] 1, wherein the one or more programs further include instructions for:while displaying the menu , detecting, via at least the optical sensor, a third hand gesture; and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that the third hand gesture is the fourth type of gesture, ceasing to display the menu .

19\. (Original) The computer system of claim 1, wherein the one or more programs further include instructions for:after displaying the user interface that includes the first user interface object, detecting, via at least the optical sensor, a fourth hand gesture; and
in response to detecting, via at least the optical sensor, the fourth hand gesture:
in accordance with a determination that the fourth hand gesture is a fifth type of gesture that is different from the first type of gesture and the second type of gesture, transitioning the computer system from an inactive state to an active state.

20\. (Previously Presented) The computer system of claim 1, wherein the one or more programs further include instructions for:while displaying, via the display generation component, the indication that the second user interface object is selected, detecting, via at least the optical sensor, a fifth hand gesture; and
in response to detecting, via at least the optical sensor, the fifth hand gesture:

in accordance with a determination that the fifth hand gesture is the first type of gesture, displaying, via the display generation component, an indication that a fourth user interface object is selected; and
in accordance a determination that the fifth hand gesture is the second type of gesture, displaying, via the display generation component, the indication that the first user interface object is selected.

21\. (Previously Presented) The computer system of claim 1, wherein the one or more programs further include instructions for:while displaying the user interface that includes the first user interface object, detecting a request to transition the computer system from the first operation mode to a fourth operation mode;
in response to detecting the request to transition the computer system from the first operation mode to the fourth operation mode, transitioning the computer system from the first operation mode to the fourth operation mode;
while the computer system is in the fourth operation mode and while displaying the user interface that includes the first user interface object, the second user interface object, and the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a sixth hand gesture; and
in response detecting, via least the optical sensor, the sixth hand gesture:
in accordance with a determination that the sixth hand gesture is the first type of gesture, continuing to display the indication that the first user interface object is selected; and
in accordance with a determination that the sixth hand gesture is the second type of gesture, continuing to display the indication that the first user interface object is selected.

22\. (Currently Amended) A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system that includes a first hardware input mechanism and a second hardware input mechanism and that is in communication with a display generation component and an optical sensor, the one or more programs including instructions for:

displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected;
while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture; and
in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is a first type of gesture,
displaying, via the display generation component, an indication that the second user interface object is selected;in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected; \[\[and\]\] in accordance with a determination that the hand gesture is a third type of gesture that is different from the first type of gesture and the second type of gesture, performing an operation corresponding to selection of the first user interface object; and
in accordance with a determination that the hand gesture is a fourth type of gesture that is different from the first type of gesture, the second type of gesture, and the third type of gesture, displaying, via the display generation component, a menu, wherein:a first control and a second control are available via the menu;selection of the first control causes the computer system to perform an operation corresponding to a press of the first hardware input mechanism; and selection of the second control causes the computer system to perform an operation corresponding to a rotation of the second hardware input mechanismSVG {EMBED HERE} 0.1430.477

.

23\. (Currently Amended) A method, comprising:at a computer system that includes a first hardware input mechanism and a second hardware input mechanism and that is in communication with a display generation component and an optical sensor:displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected;
while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture; and
in response to detecting, via at least the optical sensor, the hand gesture:

in accordance with a determination that the hand gesture is a first type of gesture, displaying, via the display generation component, an indication that the second user interface object is selected;
in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected; \[\[and\]\] in accordance with a determination that the hand gesture is a third type of gesture that is different from the first type of gesture and the second type of gesture, performing an operation corresponding to selection of the first user interface object; and in accordance with a determination that the hand gesture is a fourth type of gesture that is different from the first type of gesture, the second type of gesture, and the third type of gesture, displaying, via the display generation component, a menu, wherein:a first control and a second control are available via the menu;
selection of the first control causes the computer system to perform an operation corresponding to a press of the first hardware input mechanism; and selection of the second control causes the computer system to perform an operation corresponding to a rotation of the second hardware input mechanism

.

24\. (Previously Presented) The non-transitory computer-readable storage medium of claim 22, wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor.

25\. (Currently Amended) The non-transitory computer-readable storage medium of claim \[\[22\]\]26 wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture.

26\. (Currently Amended) The non-transitory computer-readable storage medium of claim 22, wherein before detecting the hand gesture, the computer system is in a first operation mode, wherein performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode, wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture, and wherein the one or more programs further include instructions for:while the computer system is in the second operation mode, detecting, via at least the optical sensor, a second hand gesture; and
in response to detecting, via at least the optical sensor, the second hand gesture:in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation;
in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation:

in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation; and
in accordance with a determination that the second hand gesture is the fourth type of gesture, ending the automatic scrolling operation
.

27\. (Canceled)

28\. (Currently Amended) The non-transitory computer-readable storage medium of claim \[\[22\]\]26, wherein the automatic scrolling operation scrolls a sequence of a plurality of interface objects in a second direction, and wherein the one or more programs further include instructions for:while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects; and
in response to detecting the end of the sequence of the plurality of user interface objects,
performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction.

29\. (Canceled)

30\. (Currently Amended) The non-transitory computer-readable storage medium of claim \[\[22\]\]26, wherein the one or more programs further include instructions for:in accordance with a determination that the second hand gesture is the first type of gesture, displaying a notification that indicates a state of the automatic scrolling operation.

31\. (Currently Amended) The non-transitory computer-readable storage medium of claim \[\[22\]\]26, wherein the third type of gesture is a type of gesture that is multiple instances of the fourth type of gesture.

32\. (Previously Presented) The non-transitory computer-readable storage medium of claim 22, wherein the fourth type of gesture does not include multiple instances of the first type of gesture and the third type of gesture does not include multiple instances of the second type of gesture.

33\. (Previously Presented) The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs further include instructions for:detecting, via at least the optical sensor, a third hand gesture; and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that a notification was received within a threshold period of time and in accordance with a determination that the third hand gesture is the fourth type of gesture, performing an action related to the notification.

34\. (Currently Amended) The non-transitory computer-readable storage medium of claim \[\[26\]\] 22, wherein the menu includes one or more selectable options, wherein the one or more selectable options includes a first selectable user interface object for changing operations that one or more hand gestures can cause to the computer system to perform, wherein selection of the first selectable user interface object causes the computer system to display a plurality of settings, and wherein each setting controls an operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system.

35\. (Currently Amended) The non-transitory computer-readable storage medium of claim \[\[26\]\] 22, wherein:the menu includes one or more selectable options;
the one or more selectable options includes a second selectable option for transitioning the computer system to a fourth operation mode and a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode;
selection of the second selectable option causes the computer system to transition to the fourth operation mode; and
selection of the third selectable option causes the computer system to transition to the fifth operation mode.

36\. (Currently Amended) The non-transitory computer-readable storage medium of claim \[\[26\]\] 22, wherein the menu includes one or more selectable options, wherein the one or more selectable options include a fourth selectable option for displaying one or more additional selectable options, and wherein selection of the fourth selectable option causes the computer system to display the one or more additional selectable options that were not previously displayed before selection of the fourth selectable option.

37\. (Currently Amended) The non-transitory computer-readable storage medium of claim \[\[26\]\] 22, wherein the menu is displayed:in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location; and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location.

38\. (Currently Amended) The non-transitory computer-readable storage medium of claim \[\[26\]\] 22, wherein the one or more programs further include instructions for:while displaying the menu , detecting, via at least the optical sensor, a third hand gesture; and

in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that the third hand gesture is the fourth type of gesture, ceasing to display the menu .

39\. (Previously Presented) The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs further include instructions for:after displaying the user interface that includes the first user interface object, detecting, via at least the optical sensor, a fourth hand gesture; and
in response to detecting, via at least the optical sensor, the fourth hand gesture:
in accordance with a determination that the fourth hand gesture is a fifth type of gesture that is different from the first type of gesture and the second type of gesture, transitioning the computer system from an inactive state to an active state.

40\. (Previously Presented) The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs further include instructions for:while displaying, via the display generation component, the indication that the second user interface object is selected, detecting, via at least the optical sensor, a fifth hand gesture; and
in response to detecting, via at least the optical sensor, the fifth hand gesture:
in accordance with a determination that the fifth hand gesture is the first type of gesture, displaying, via the display generation component, an indication that a fourth user interface object is selected; and
in accordance a determination that the fifth hand gesture is the second type of gesture, displaying, via the display generation component, the indication that the first user interface object is selected.

41\. (Previously Presented) The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs further include instructions for:while displaying the user interface that includes the first user interface object, detecting a request to transition the computer system from the first operation mode to a fourth operation mode;

in response to detecting the request to transition the computer system from the first operation mode to the fourth operation mode, transitioning the computer system from the first operation mode to the fourth operation mode;
while the computer system is in the fourth operation mode and while displaying the user interface that includes the first user interface object, the second user interface object, and the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a sixth hand gesture; and
in response detecting, via least the optical sensor, the sixth hand gesture:
in accordance with a determination that the sixth hand gesture is the first type of gesture, continuing to display the indication that the first user interface object is selected; and
in accordance with a determination that the sixth hand gesture is the second type of gesture, continuing to display the indication that the first user interface object is selected.

42\. (Previously Presented) The method of claim 23, wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor.

43\. (Currently Amended) The method of claim \[\[23\]\] 44 wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture.

44\. (Currently Amended) The method of claim 23, before detecting the hand gesture, the computer system is in a first operation mode, wherein performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode, wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture, and wherein the method further comprises :while the computer system is in the second operation mode, detecting, via at least the optical sensor, a second hand gesture; and
in response to detecting, via at least the optical sensor, the second hand gesture:

in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation;
in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation;
in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation; and
in accordance with a determination that the second hand gesture is the fourth type of gesture, ending the automatic scrolling operation
.

45\. (Canceled)

46\. (Currently Amended) The method of claim \[\[23\]\] 44, wherein the automatic scrolling operation scrolls a sequence of a plurality of interface objects in a second direction, and wherein the method further comprises:while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects; and
in response to detecting the end of the sequence of the plurality of user interface objects,
performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction.

47\. (Canceled)

48\. (Currently Amended) The method of claim \[\[23\]\] 44, further comprising:in accordance with a determination that the second hand gesture is the first type of gesture, displaying a notification that indicates a state of the automatic scrolling operation.

49\. (Currently Amended) The method of claim \[\[23\]\] 44 wherein the third type of gesture is a type of gesture that is multiple instances of the fourth type of gesture.

50\. (Previously Presented) The method of claim 23, wherein the fourth type of gesture does not include multiple instances of the first type of gesture and the third type of gesture does not include multiple instances of the second type of gesture.

51\. (Previously Presented) The method of claim 23, further comprising:detecting, via at least the optical sensor, a third hand gesture; and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that a notification was received within a threshold period of time and in accordance with a determination that the third hand gesture is the fourth type of gesture, performing an action related to the notification.

52\. (Currently Amended) The method of claim \[\[44\]\] 23,wherein the menu includes one or more selectable options, wherein the one or more selectable options includes a first selectable user interface object for changing operations that one or more hand gestures can cause to the computer system to perform, wherein selection of the first selectable user interface object causes the computer system to display a plurality of settings, and wherein each setting controls an operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system.

53\. (Currently Amended) The method of claim \[\[44\]\] 23 wherein:the menu includes one or more selectable options;
the one or more selectable options includes a second selectable option for transitioning the computer system to a fourth operation mode and a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode;

selection of the second selectable option causes the computer system to transition to the fourth operation mode; and
selection of the third selectable option causes the computer system to transition to the fifth operation mode.

54\. (Currently Amended) The method of claim \[\[44\]\] 23,wherein the menu includes one or more selectable options, wherein the one or more selectable options include a fourth selectable option for displaying one or more additional selectable options, and wherein selection of the fourth selectable option causes the computer system to display the one or more additional selectable options that were not previously displayed before selection of the fourth selectable option.

55\. (Currently Amended) The method of claim \[\[44\]\] 23, wherein the menu is displayed:in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location; and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location.

56\. (Currently Amended) The method of claim \[\[44\]\] 23, further comprising:while displaying the menu , detecting, via at least the optical sensor, a third hand gesture; and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that the third hand gesture is the fourth type of gesture, ceasing to display the menu .

57\. (Previously Presented) The method of claim 23, further comprising:after displaying the user interface that includes the first user interface object, detecting, via at least the optical sensor, a fourth hand gesture; and
in response to detecting, via at least the optical sensor, the fourth hand gesture:

in accordance with a determination that the fourth hand gesture is a fifth type of gesture that is different from the first type of gesture and the second type of gesture, transitioning the computer system from an inactive state to an active state.

58\. (Previously Presented) The method of claim 23, further comprising:while displaying, via the display generation component, the indication that the second user interface object is selected, detecting, via at least the optical sensor, a fifth hand gesture; and
in response to detecting, via at least the optical sensor, the fifth hand gesture:
in accordance with a determination that the fifth hand gesture is the first type of gesture, displaying, via the display generation component, an indication that a fourth user interface object is selected; and
in accordance a determination that the fifth hand gesture is the second type of gesture, displaying, via the display generation component, the indication that the first user interface object is selected.

59\. (Previously Presented) The method of claim 23, further comprising:while displaying the user interface that includes the first user interface object, detecting a request to transition the computer system from the first operation mode to a fourth operation mode;
in response to detecting the request to transition the computer system from the first operation mode to the fourth operation mode, transitioning the computer system from the first operation mode to the fourth operation mode;
while the computer system is in the fourth operation mode and while displaying the user interface that includes the first user interface object, the second user interface object, and the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a sixth hand gesture; and
in response detecting, via least the optical sensor, the sixth hand gesture:
in accordance with a determination that the sixth hand gesture is the first type of gesture, continuing to display the indication that the first user interface object is selected; and

in accordance with a determination that the sixth hand gesture is the second type of gesture, continuing to display the indication that the first user interface object is selected.

OFFICE ACTION:
DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the Amendment filed on January 30, 2024.  Claims 1, 3, 4, 8, 10, 11, 14-18, 22, 23, 25, 26, 28, 30, 31, 34-38, 43, 44, 46, 48, 49, and 52-56 are amended.  Claims 5, 6, 7, 9, 27, 29, 45, and 47 are cancelled.  Therefore, Claims 1-4 8, 10-26, 28, 30-44, 46, and 48-59 are pending in the case.  Claims 1, 22, and 23 are the independent claims.
This action is final.
Applicant’s Response
In the Amendment filed on January 30, 2024, Applicant amended the claims and provided arguments in response to rejection of the claims under 35 USC 103 in the previous office action.
Response to Argument/Amendment
Applicant’s amendments to the claims in response to the rejection of the claims under 35 USC 103 in the previous office action are acknowledged, and Applicant’s associated arguments have been fully considered.
Regarding the independent claims, Applicant argues that the cited references (Osborn, Li, Kin, and Pantel) fail to teach or suggest displaying a menu that includes controls to perform respective operations that correspond to pressing a hardware input device and rotating a hardware input device, as recited in the amended claims.  Applicant’s arguments are persuasive, and the rejection is withdrawn.
Regarding claims 3, 25, and 43, Applicant argues that Osborn, Li, Kin, and Pantel do not teach that the gesture that reverses autoscroll direction is multiple instances of the gesture that pauses autoscroll.  The claims recite “wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture.”  As cited in the previous office action, Osborn does appear to teach this limitation as recited with respect to at least the first operation mode recited in claim 4 (i.e. receiving a single instance of a gesture in order to display an indication that a second object is selected and receiving multiple instances of the gesture in order to display an indication that a third object is selected).  However, Examiner agrees that the cited references do not appear to teach this limitation with respect to the second operation mode recited in claim 4 (i.e. as argued by Applicant, where the second type of gesture, used to reverse autoscroll direction, is multiple instances of the first type of gesture, used to pause autoscroll).  Given that the claims recite first and second types of gestures (i.e. corresponding to classes/categories of gestures, and not necessarily to any specific gesture within the type/class/category), and further recite that these are used for different purposes in different modes, it is not entirely clear that the claims, under the broadest reasonable interpretation, require the elements specifically argued by applicant (i.e. that the specific gesture (as opposed to type of gesture) that reverses autoscroll must be multiple instances of the specific gesture that pauses autoscroll, where these specific gestures have this same one-to-multiple relationship across both recited operation modes).  However, this argument is moot in view of the new grounds of rejection provided below.
Regarding claims 11, 31, and 49, Applicant argues that Osborn, Li, Kin, Pantel, and Kim do not teach that the gesture that ends autoscrolling is multiple instances of the gesture that changes the speed of autoscrolling.  First, the claims actually recite “wherein the third type of gesture is a type of gesture that is multiple instances of the fourth type of gesture.”  Examiner respectfully submits that this appears to require the opposite of what is argued by Applicant, i.e. appears to require that the gesture that changes the speed of autoscrolling (i.e. the third gesture in the second mode as recited in claim 4) is multiple instances of the gesture that ends autoscrolling (i.e. the fourth gesture in the second mode as recited in claim 4).  Assuming that this understanding is correct, Applicant’s argument cannot be persuasive, as it is directed to limitations not present in the claims.  In addition, similar to claims  3, 25, and 43, given that the claims recite third and fourth types of gestures (i.e. corresponding to classes/categories of gestures, and not necessarily to any specific gesture within the type/class/category), and further recite that these are used for different purposes in different modes, it is not entirely clear that the claims, under the broadest reasonable interpretation, require the elements specifically argued by applicant (i.e. that the specific gesture (as opposed to type of gesture) that ends autoscrolling must be multiple instances of the specific gesture that changes the speed of autoscrolling, as argued by Applicant, or the reverse situation discussed above in which the specific gesture that changes the speed of autoscrolling must be multiple instances of the specific gesture that ends autoscrolling, where these specific gestures have this same one-to-multiple relationship across both recited operation modes).  However, this argument is moot in view of the new grounds of rejection provided below.
As noted above, Applicant’s arguments are persuasive, and the rejection is withdrawn.  However, new grounds of rejection are provided below.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims under pre-AIA  35 U.S.C. 103(a), the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were made absent any evidence to the contrary.  Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and invention dates of each claim that was not commonly owned at the time a later invention was made in order for the examiner to consider the applicability of pre-AIA  35 U.S.C. 103(c) and potential pre-AIA  35 U.S.C. 102€, (f) or (g) prior art under pre-AIA  35 U.S.C. 103(a).
Claims 1, 12, 14, 15, 16, 19, 20, 21, 22, 23, 32, 34, 35, 36, 39, 40, 41, 50, 52, 53, 54, 57, 58, and 59 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn et al. (US 20230072423 A1) in view of Fleizach et al. (US 21020306748 A1).
With respect to claims 1, 22, and 23, Osborn teaches a computer system that is in communication with a display generation component and an optical sensor, the computer system comprising: a first hardware input mechanism (e.g. paragraph 0386, vibrotactile system having touch-sensitive pads, pressure sensors, on/off button, etc.; paragraph 1605, text entry control signal via physical or electrical button, switch, touchscreen, etc.); a second hardware input mechanism (e.g. paragraph 0278, system including IMUs such as accelerometer, gyroscope, magnetometer, etc.; paragraph 0352, sensors including accelerometers, gyroscopes, magnetometers, etc.; paragraph 0386, vibrotactile system having motion/position sensors); one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method; a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system that is in communication with a display generation component and an optical sensor, the one or more programs including instructions for performing the method (e.g. paragraph 0257, supplementing sensor data with images of user captured by camera; paragraph 0286, auxiliary devices/sensors including imaging devices/cameras, heart rate monitors, etc.; paragraph 0288, Figs. 9A-B, wearable system with sensors such as EMG sensors, sensors 810; computer processor such as microcontroller 934 receiving inputs from sensors; communicating with dongle portion 920 and outputting processing output to dongle portion; paragraph 0289, dongle portion 920 receiving signals from wearable portion 910, and providing them to host computer for further processing, display, effecting control of objects, etc.; paragraph 0292, describing Fig. 1, system including processors 112 programmed to communicate with sensors 110 and executing algorithms; paragraph 0301, system including display controller configured to display visual representations; computer processors implementing trained inference models that receive sensor signals as input and provide information to generate control signals for system as output; paragraph 0302, user interface; paragraph 0303, various types of devices in which processors may be included; paragraph 0304, multiple processors; paragraph 0306, system including processors, camera, and display; paragraph 310, camera information/cameras; paragraph 0311, camera mounted on user; paragraph 0344, embodiments implemented in software executed on processors; paragraph 0345, non-transitory computer-readable storage medium/memory encoded with computer program/instructions executed on processor to perform described functions; paragraph 0352, sensor including light emitter/detector; paragraph 0385, Fig. 13, vibrotactile system 1300 may be standalone system, or may be configured with interaction with various types of other devices, and includes processor 1360; paragraph 0610, cameras; paragraph 0729, photoplethysmography/PPG sensors detecting vascular changes/changes in blood volume; paragraph 1384, Fig. 35B, wearable device 35102 capable of controlling computer devices via neuromuscular signals of users, including set of sensors 35104 that detect/sense neuromuscular signals traversing body of user; processing device 35106 communicatively coupled to sensors, and memory 35108 including models and computer-executable instructions that when executed by processor cause it to identify force generated, generate commands for HCI system, etc.; i.e. the device includes at least one processor and memory storing instructions for execution by the processor, and is further in communication with at least one display/display controller/generation component, and at least one optical sensor, such as a camera (which may be mounted on the user)); and the method, comprising:
displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected (e.g. paragraph 1460, describing radial menu 351000 in Figs. 35O-V, where the menu may include visual indicators including a selection indicator 351020; paragraph 1461, selection indicator 351020 may indicate a currently selected action indicator; paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual, auditory, or haptic feedback; visual feedback representing change of a characteristic of visual element within radial menu in response to selection of the visual indicator, such as position change, orientation change, color change, size change, transparency change, fill change, emphasis change, shadow change, animation change, font change, line type change, line width change, combinations, etc.; paragraph 1473, describing Figs. 36B-C, radial menu 351110 including a set of action indicators including indicators 351111 and 351113 displayed in GUI along arc, where as shown in Fig. 36B includes an indication that indicator 35111 is selected; demonstrating/confirming selection of visual indicator using visual feedback; paragraph 1475, describing Figs. 36D-E, displaying sequential menu 351120 including a sequence of visual indicators depicted as A through F, where as shown in Fig. 36 D includes in indication that indicator D/351123 is selected; i.e. both of the menus/interfaces shown in the cited figures include at least first, second, and third UI objects along with an indication that an object of these is selected, such as a visual feedback including a shading/coloration of a selected indicator; paragraphs 1385-1387, determining based on neuromuscular signals state of body part of user and generating input commands accounting for state of user’s body part; sensors constitute any type of form of sensor capable of detecting/sensing neuromuscular signals in user’s body including EMG sensors, MMG sensors, SMG sensors; sensors may also include auxiliary sensors such as IMU, position tracking, imaging (camera), and radiation based sensors, microphones, heart rate monitors, etc.; paragraph 1394, wearable device 35102 coupled with computing system/display; user placing body in state/condition to control/modify presentation/performance of computing system, generating neuromuscular signals representative/indicative/suggestive of state/action);
while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture (e.g. paragraph 0310, camera information used to improve interpretation of neuromuscular signals; camera used to record image of user, may be within particular range/wavelengths of light, images produced by visible/non-visible light, including showing heat distribution in user’s body, providing indication of blood flow within user and used to infer condition of user such as forces being exerted by fingers; paragraph 0311, camera mounted on user; paragraph 0312, camera data in addition to neuromuscular activity signals used to reconstruct positions, geometries, and forces being applied by user’s arm/hand; paragraph 0323, auxiliary signals from auxiliary devices such as camera supplementing neuromuscular signals; paragraph 0610, cameras used to improve interpretation of signals; paragraph 0611, camera data used to provide real time determination of musculoskeletal representations of gestures; paragraph 0729, PPG sensors detecting vascular changes; paragraph 1271, camera/imaging data; paragraph 1387, sensors including auxiliary sensors such as cameras; paragraph 1395, user pose/gesture generating neuromuscular signals as they traverse down arm toward hand; sensors 35104 detecting/sensing neuromuscular signals and generating data representative of the signals; paragraph 1396-1398, sensor data undergoing processing, analyzing data to determine state of body parts of user, producing/outputting classification indicating state of body part; paragraph 1398, gestures; paragraph 1400, causing computing system to perform actions mapped to state of body parts; paragraph 1401, describing various actions mapped to different states of user’s body parts; paragraph 1444, capturing gestures with camera; photographic data of hand gestures; paragraph 1473, body state mapped to selection of next indicator in a given sequence; identification of body state; paragraph 1474, body states including wrist extension, flexion, rotation, radial deviation, ulnar deviation, etc.; paragraph 1475, body states mapped to selection of visual indicators in sequence; identification of body states; i.e. detecting a body state of the user, such as a hand gesture, including by using an optical sensor such as a camera,  PPG, etc., possibly in combination with other sensors),
in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is a first type of gesture, displaying, via the display generation component, an indication that the second user interface object is selected (e.g. paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual feedback; paragraph 1473, Figs. 36B-C, first body state mapped to selection of the next indicator in a given sequence; in response to identification of first body state, deselecting indicator 35111 and selecting indicator 351113; demonstrating/confirming selection of visual indicator using visual feedback; paragraph 1475, Figs. 36D-E, first body state mapped to selection of next visual indicator in the sequence; i.e. in response to detecting a first type of gesture, as indicated by a first body state, the indication that a first UI object is selected may be changed to an indication that a second UI object is selected, such as by moving the indicator of the selection to the second UI object, where the second UI object is a next UI object in a sequence with respect to the first UI object); and
in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected (e.g. paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual feedback; paragraph 1473-1474, Figs. 36B-C, first body state mapped to selection of next indicator in given sequence; demonstrating/confirming selection of visual indicator using visual feedback; first and second body states may represent counterparts of one another; 1475, Figs. 36D-E, second body state mapped to selection of previous visual indicator in the sequence; in response to identification of second body state, deselecting subsequent indicator 351123 and selecting prior indicator 351121; i.e. in response to detecting a second/different type of gesture, as indicated by a second body state, the indication that a first UI object is selected may be changed to an indication that a third UI object is selected, such as by moving the indicator of the selection to the third UI object, where the third UI object is a previous UI object in a sequence with respect to the first UI object; in addition, because these gestures cause selection to move to a next/previous object in sequence, receipt of multiple of either of these gestures may also cause a different/third UI object to be selected; for example, where multiple instances/two instances of the first type of gesture are received, the object selected as indicated may be first moved from the first UI object to the second/next UI object in the sequence, and then moved again from the second/next UI object to a third/subsequent UI object in the sequence); and
in accordance with a determination that the hand gesture is a third type of gesture that is different from the first type of gesture and the second type of gesture, performing an operation corresponding to selection of the first user interface object (e.g. paragraph 1281, fist squeeze used for discrete event such as to engage or disengage a particular feature within a given application or a continuous event, such as to maintain activation of a particular feature within a given application; paragraph 1283, selecting from presented options by performing hand gesture such as clenching the fist; paragraph 1468, Fig. 35Q, illustrating finger pinch pose 351006, which may be mapped to command/instruction to click visual indicator in the radial menu, analogous to mouse button click or keypress; alternatively combination of finger pinch pose and certain degree of force mapped to click command; paragraph 1469, Fig. 35 U, clicking/engaging with or activating function of the currently selected action indicator; clicking action indicator in response to identified click gesture; upon performing the click action, causing visual indicator to appear to depress and/or release, similar to a physical button; i.e. clicking the currently selected action indicator causes the corresponding function to be activated); and
in accordance with a determination that the hand gesture is a fourth type of gesture that is different from the first type of gesture, the second type of gesture, and the third type of gesture, displaying, via the display generation component, a menu (e.g. paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user (as compared to other movements/poses causing item selection/clicking); paragraph 1485, displaying interface 351300 of Fig. 36G, which may be a popup box/dialog that appears in response to user input; i.e. where a some suitable user input, such as a fist clenching gesture, may be used to open various menus, such as a menu including selectable options).
Osborn does not explicitly disclose wherein:
a first control and a second control are available via the menu;
selection of the first control causes the computer system to perform an operation corresponding to a press of the first hardware input mechanism; and
selection of the second control causes the computer system to perform an operation corresponding to rotation of the second hardware input mechanism.
However, Fleizach teaches wherein:
a first control and a second control are available via the menu (e.g. paragraph 0232-0233, Fig. 5B, menu 508 displayed in response to predefined input, and includes plurality of icons 510 including virtual touches selection 510-1, virtual gestures 510-2, virtual device 510-3, and home button 510-4; paragraph 0248, Fig. 5Q, menu of virtual device operations representing plurality of icons representing types of virtual device operations);
selection of the first control causes the computer system to perform an operation corresponding to a press of the first hardware input mechanism (e.g. paragraph 0234-0247, Figs. 5C-O, virtual touches icon 510-1 selected, number of contacts selected, performing gestures on device based on selected number of contacts; virtual gestures icon 510-2 selected, respective gesture is selected, corresponding gesture performed on device; paragraph 0247, Fig. 5P-, virtual device icon 510-3 selected; paragraph 0248, Fig. 5Q, virtual device icons for muting/unmuting 526-1, device lock 526-4, volume up 526-5, volume down 526-6; virtual device operations including operations usually performed via physical controls on the device, including muting and unmuting the ringer and sound effects on the device, which is normally performed via a mute switch; locking the device, which is normally initiated via a lock/unlock switch; increase volume, which is normally performed via a volume up button; decrease volume, which is normally performed via a volume down button; paragraph 0251-0252, Figs. 5T-U, home button icon 510-4 selected, and home screen displayed in response; i.e. selection of a touch-related, home screen, or other device control menu option causes performance of a device function which corresponds to a hardware input mechanism press, such as pressing a volume button, mute or lock switch, home screen button, etc.); and
selection of the second control causes the computer system to perform an operation corresponding to rotation of the second hardware input mechanism (e.g. paragraph 0248, Fig. 5Q, virtual device rotation icon 526-2 selected; virtual device operations including operations usually performed via physical controls on the device, including shaking device to perform predefined operation, rotating device to rotate display orientation, which are typically detected via accelerometer in the device; paragraph 0249-0250, Fig. 5R-S, icons 530 representing types of virtual device orientations, or rotations of device orientation; user interface rotated 90 degrees clockwise in response to selection of rotate-left orientation icon 530-2, without actual physical rotation of the device; i.e. selection of a rotation-related menu option causes performance of a device function which corresponds to a hardware input mechanism rotation (such as rotation of a device accelerometer through space), such as a 90 degree (or greater rotation of the device)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Fleizach in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) to incorporate the teachings of Fleizach (directed to providing control of a touch-based user interface absent physical touch capabilities) to include the capability to display, via the menu (i.e. of Osborn), a first control which, when selected, causes the device to perform operations corresponding to pressing of a hardware input mechanism (i.e. such as pressing of a physical button or touching of a touchscreen or touch-sensitive surface, to perform an associated function, such as performing touch screen selections and/or gestures, or a function associated with a physical button, such as a home screen function, screen lock function, volume/mute control functions, etc.) and a second control which, when selected, causes the device to perform operations corresponding to rotation of a hardware input mechanism (i.e. such as rotation of one or more accelerometers of the device in space, to perform an associated function, such as altering a display orientation), as taught by Fleizach.  One of ordinary skill would have been motivated to perform such a modification in order to provide electronic devices with touch-based user interfaces with methods and interfaces that make these devices accessible to users without fine motor skills in their hands, as described in Fleizach (paragraph 0063).
With respect to claims 12, 32, and 50, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the fourth type of gesture does not include multiple instance of the first type of gesture and the third type of gesture does not include multiple instances of the second type of gesture (e.g. paragraph 0620, finger/hand gesture combined with force, user clenching first, etc., to open menu/dialog; paragraph 1468, finger pinch pose used to click selected item; paragraph 1381, first pose to display menu; paragraph 1474, first and second body states may be counterparts of one another; body states including wrist extension, flexion, rotation, radial deviation, ulnar deviation, etc.; i.e. where a fist clenching or other user input used for opening a menu, analogous to a fourth type of gesture does not include multiple instances of a first body state such as wrist extension, flexion, rotation, deviation, etc., analogous to a first type of gesture, and where a finger pinch pose for clicking a selected item, analogous to a third type of gesture, does not include multiple instances of a second body state such as wrist extension, flexion, rotation, deviation, etc., counterpart to a first body state, analogous to a second type of gesture ).
With respect to claims 14, 34, and 52, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the menu includes one or more selectable options, wherein the one or more selectable options includes a first selectable user interface object for changing operations that one or more hand gestures can cause to the computer system to perform, wherein selection of the first selectable user interface object causes the computer system to display a plurality of settings, and wherein each setting controls an operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system (e.g. paragraph 1483, Fig. 36S, multi-state user interface configurable/programmable by user; user able to specify mappings between body states and responsive actions; such as mapping body part sates of ring/middle/pinky/index finger pinches, first pose, and open hand pose to scroll up/down, activate/deactivate link, increment/decrement selector actions; paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H; paragraph 1486, Fig. 36I, user modifying mappings between body states and actions; configuring web navigation settings by selecting setting options shown in drop-down menu of web browser; user clicking and selecting “scroll:options” to configure type of pose that initiates scrolling control or feature; paragraph 1487, Fig. 36J, link activate setting used to highlight links included in web page; user clicking on “linksActivate:options” to configure type of pose and predetermined pose duration that initiates the activation and rendering of link displayed in webpage; paragraph 1488, Fig. 36K, user clicking on “click:options” to configure type of pose and predetermined duration that initiates a click action in an application; i.e. selection of the “Control Options” dropdown object of Fig. 36G causes the plurality of settings to be shown in Fig. 36H, where these settings allow control of which gestures/poses are mapped to which operations).
With respect to claims 15, 35, and 53, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein:
the menu includes one or more selectable options (e.g. paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H);
the one or more selectable options includes a second selectable option for transitioning the computer system to a fourth operation mode and a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode (e.g. paragraph 1485, Fig. 36G, multi-state user interface facilitates enabling and disabling body state control using “enabled” button; drop down menu control from which user is able to select certain display settings and map body states to responsive actions; see also “reset settings” button in Fig. 36G;  i.e. where an “enabled” button can be selected to transition the system to a state/mode in which the body state controls are enabled, and to a state/mode in which body state controls are disabled, analogous to at least a second selectable option for transitioning the computer system to a fourth operation mode (i.e. a body state controls disabled mode); further where the user selects the drop down menu and modifies the mappings of body states and responsive actions, this causes the system to be transitioned from a first mode (i.e. a mode in which a current set of body state/action mappings are utilized) to a different/fifth mode (i.e. a mode in which a modified/custom set of body state/action mappings are utilized), analogous to a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode; in addition, where body state/action mappings are already customized/modified, and the user selects the option to “reset settings” this would cause the system to be transitioned from a first mode (i.e. a mode in which a current set of modified/customized body state/action mappings are utilized) to a different/fifth mode (i.e. a mode in which a default/reset set of body state/action mappings are utilized), analogous to a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode);
selection of the second selectable option causes the computer system to transition to the fourth operation mode (e.g. paragraph 1485, Fig. 36G, enabling/disabling body state control in using “enabled” button, as discussed above); and
selection of the third selectable option causes the computer system to transition to the fifth operation mode (e.g. paragraph 1485, Fig. 36G, user selecting drop down menu control to map body states to responsive actions; user selecting to “reset settings”, as discussed above).
With respect to claims 16, 36, and 54, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the menu includes one or more selectable options, wherein the one or more selectable options include a fourth selectable option for displaying one or more additional selectable options, and wherein selection of the fourth selectable option causes the computer system to display the one or more additional selectable options that were not previously displayed before selection of the fourth selectable option (e.g. paragraph 1483, Fig. 36S, multi-state user interface configurable/programmable by user; user able to specify mappings between body states and responsive actions; such as mapping body part sates of ring/middle/pinky/index finger pinches, first pose, and open hand pose to scroll up/down, activate/deactivate link, increment/decrement selector actions; paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H; paragraph 1486, Fig. 36I, user modifying mappings between body states and actions; configuring web navigation settings by selecting setting options shown in drop-down menu of web browser; user clicking and selecting “scroll:options” to configure type of pose that initiates scrolling control or feature; paragraph 1487, Fig. 36J, link activate setting used to highlight links included in web page; user clicking on “linksActivate:options” to configure type of pose and predetermined pose duration that initiates the activation and rendering of link displayed in webpage; paragraph 1488, Fig. 36K, user clicking on “click:options” to configure type of pose and predetermined duration that initiates a click action in an application; i.e. selection of the “Control Options” dropdown object of Fig. 36G, analogous to a fourth selectable option, causes additional selectable options, not previously displayed, to be displayed, as shown in Fig. 36H).
With respect to claims 19, 39, and 57, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
after displaying the user interface that includes the first user interface object, detecting, via at least the optical sensor, a fourth hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, fist clenching; paragraph 1463, Fig. 35O, fist pose; i.e. while/after the menu including the first user interface object is being/has been previously displayed, or at some later time, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the fourth hand gesture:
in accordance with a determination that the fourth hand gesture is a fifth type of gesture that is different from the first type of gesture and the second type of gesture, transitioning the computer system from an inactive state to an active state (e.g. paragraph 0509, mapping neuromuscular signals to control signals to control device based on type of gesture; paragraph 0618, determining the gesture being performed; paragraph 0620, fist clenching by user results in wake signal being sent to the system; paragraph 0703, first muscular activation state detected used to determine that a wake up mode is to be controlled, and second muscular activation state used to second activation state used to control initialization operation of the wake up mode; paragraph 0706, wake up/initialization mode; paragraph 1380, mappings causing device to perform action in response to detection of specific body part states; paragraph 1463, Fig. 35O, fist pose mapped to command to display, enter, and activate radial menu; i.e. when the system determines, based on a mapping body part states/gestures to actions, that the user is performing a fist pose hand gesture or other suitable gesture, the system may perform a corresponding action of activating a menu, or waking the system from an inactive/shut down state, analogous to transitioning the system from an inactive state to an active state).
With respect to claims 20, 40, and 58, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying, via the display generation component, the indication that the second user interface object is selected, detecting, via at least the optical sensor, a fifth hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraphs 1473-1475, first body state mapped to selection of next indicator in given sequence; first and second body states may represent counterparts; first body state mapped to selection of next visual indicator in sequence, second body state mapped to selection of previous visual indicator in sequence; i.e. while the interface/menu including the first, second, and third objects is being displayed, including after the user has performed a gesture causing a second user interface object to be selected and while the second user interface object is displayed with an indication that it is selected, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the fifth hand gesture:
in accordance with a determination that the fifth hand gesture is the first type of gesture, displaying, via the display generation component, an indication that a fourth first user interface object is selected (e.g. paragraphs 1473-1475, first body state mapped to selection of next indicator in given sequence; in response to identification of first body state, deselecting indicator 351111 and selecting indicator 351113, providing visual feedback; Figs. 36B-E; i.e., where the user has already performed the first type of gesture once, causing a second user interface object to be selected as shown in Fig. 36C, the user may perform another instance of the first type of gesture, which is mapped to selection of a next indicator, such that after this second instance of the first type of gesture is performed, a next/subsequent indicator (with respect to currently selected indicator 351113, such as the next indicator below it) will be selected, and indicated as selected via visual feedback, analogous to displaying an indication that a fourth UI object is selected in accordance with a determination that the fifth hand gesture is the first type of gesture (Examiner notes that a similar rationale would be applicable in the interfaces of Figs. 36D-E, such that if a user performs a first instance of the first type of gesture to cause item “E” to be selected as the next in sequence, and then performs a second instance of the first type of gesture, this will cause the item “F” to be selected as the next item in the sequence, and indicated as such)); and
in accordance a determination that the fifth hand gesture is the second type of gesture, displaying, via the display generation component, the indication that the first user interface object is selected (paragraphs 1473-1475, first and second body states may represent counterparts; second body state mapped to selection of previous visual indicator in sequence; i.e., where the user has already performed the first type of gesture, causing a second user interface object to be selected as shown in Fig. 36C, the user may perform the second type of gesture, which is mapped to selection of a previous indicator, such that after this second type of gesture is performed, a previous indicator (with respect to currently selected indicator 351113, such as the previous indicator above it) will be selected, and indicated as selected via visual feedback (i.e. the selection will be reverted back to the state shown in Fig. 36B), analogous to displaying an indication that the first UI object is selected in accordance with a determination that the fifth hand gesture is the second type of gesture (Examiner notes that a similar rationale would be applicable in the interfaces of Figs. 36D-E, such that if a user performs a first instance of the first type of gesture to cause item “E” to be selected as the next in sequence, and then performs the second type of gesture, this will cause the item “D” to be selected as the previous item in the sequence, and indicated as such, similar to what is shown in Fig. 36D)).
With respect to claims 21, 41, and 59, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying the user interface that includes the first user interface object, detecting a request to transition the computer system from the first operation mode to a fourth operation mode (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 1470, Fig. 35 R, open hand pose, mapped to command to deactivate the menu; i.e. while the menu is being displayed, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected, including a gesture to request transitioning from a first operation mode (such as one in which a menu is activated) to a fourth operation mode (such as one in which the menu is deactivated/disabled));
in response to detecting the request to transition the computer system from the first operation mode to the fourth operation mode, transitioning the computer system from the first operation mode to the fourth operation mode (e.g. paragraph 1461, disabled radial menu; paragraph 1471, deactivating radial menu; i.e. in response to detecting the user’s open hand gesture, the radial menu is disabled/deactivated, transitioning from an operation mode in which the radial menu is activated (analogous to a first operation mode) to an operation mode in which the radial menu is deactivated (analogous to a fourth operation mode));
while the computer system is in the fourth operation mode and while displaying the user interface that includes the first user interface object, the second user interface object, and the third user interface object, and the indication that the first user interface object is selected (e.g. paragraph 1471, state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; i.e. even after the radial menu is deactivated, all user interface objects, and selection indicator (indicating which object is selected) may continue to be displayed on the display), detecting, via at least the optical sensor, a sixth hand gesture (e.g. continuously receiving neuromuscular/musculoskeletal signals and continuously performing corresponding gesture recognition as cited above); and
in response detecting, via least the optical sensor, the sixth hand gesture:
in accordance with a determination that the sixth hand gesture is the first type of gesture, continuing to display the indication that the first user interface object is selected (e.g. paragraph 1411, user rotating arm without making fist pose has no effect on the radial menu; paragraph 1461, disable radial menu is a mode in which the user is not able to interact with the radial menu; paragraph 1470, deactivated radial menu; paragraph 1471 state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; radial menu will not recognize/respond to subsequent selection (wrist gesture) or click command until receiving a subsequent activation command); i.e. after the radial menu is deactivated, all user interface objects including the selection indicator are continuously displayed and, while in the deactivated state, the user’s hand gestures, such as a first type of gesture to select a next object, will have no effect, and will not be recognized/responded to, such that the menu will continue to be displayed including the selection indicator (indicating a current action indicator as selected) in an unchanged state); and
in accordance with a determination that the sixth hand gesture is the second type of gesture, continuing to display the indication that the first user interface object is selected (e.g. paragraph 1411, user rotating arm without making fist pose has not effect on the radial menu; paragraph 1461, disable radial menu is a mode in which the user is not able to interact with the radial menu; paragraph 1470, deactivated radial menu; paragraph 1471 state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; radial menu will not recognize/respond to subsequent selection (wrist gesture) or click command until receiving a subsequent activation command); i.e. after the radial menu is deactivated, all user interface objects including the selection indicator are continuously displayed and, while in the deactivated state, the user’s hand gestures, such as a second type of gesture to select a previous object, will have no effect, and will not be recognized/responded to, such that the menu will continue to be displayed including the selection indicator (indicating a current action indicator as selected) in an unchanged state).
Claims 13, 18, 33, 38, 51, and 56 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Kim et al. (US 20160029899 A1).
With respect to claims 18, 38, and 56, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying the menu, detecting, via at least the optical sensor, a third hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 1470, Fig. 35 R, open hand pose; i.e. while a menu is being displayed, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that the third hand gesture is a type of gesture, ceasing to display the menu (e.g. paragraph 0509, mapping neuromuscular signals to control signals to control device based on type of gesture; paragraph 0618, determining the gesture being performed; paragraph 1381, open hand pose may cause the wearable device to direct the computing device to close the menu; paragraph 1380, mappings causing device to perform action in response to detection of specific body part states; paragraph 1470, open hand pose mapped to command to hide, exit, or deactivate menu; following identification of open hand pose, some or all of menu may be obscured or hidden from view within the GUI; i.e. when the system determines, based on a mapping body part states/gestures to actions, that the user is performing an open hand gesture or other suitable gesture, the system may perform a corresponding action of ceasing to display a currently displayed menu).
Osborn does not explicitly disclose that the type of gesture is a fourth type of gesture.  However, Kim teaches that the type of gesture is a fourth type of gesture (e.g. paragraph 0256, Figs. 17A-B, while message check screen outputted through display, if command for switching to lock screen is received, stopping output of message check screen and outputting/switching to lock screen; command may include first input (which is shown as a clenched fist gesture in Figs. 17A-B); i.e. where a user performing a fist gesture to initiate/switch to a lock screen display would cause the system to cease displaying all currently displayed content, including menus).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Fleizach, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) and Fleizach (directed to providing control of a touch-based user interface absent physical touch capabilities), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability, while displaying user interface content (i.e. such as a GUI including currently displayed menus, as taught by Osborn) to receive a hand gesture to transition to a lock screen state/mode, such as in response to a clenched fist hand gesture, and in response, cease displaying currently displayed content (such as the menu of Osborn) and display a lock screen (i.e. as taught by Kim).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claims 13, 33, and 51, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
detecting, via at least the optical sensor, a third hand gesture; and in response to detecting, via at least the optical sensor, the third hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected):
in accordance with a determination that the third gesture is the fourth type of gesture, performing an action (e.g. paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user); and
in accordance with a determination that a notification was received and in accordance with a determination that the third hand gesture is a type of gesture, performing an action related to the notification (e.g. paragraph 1649 and corresponding table, using swipe to engage/ignore notification; paragraphs 1651-1653, flick gesture may be used as dismissal mechanism for notifications; notifying user that next meeting is coming up, flick dismissing notification outright, index pinch tap engaging timer function, rolling wrist to select timer duration, etc.).
Osborn does not explicitly disclose that the determinations (for performing the action related to the notification) are that the notification was received within a threshold period of time and that the third hand gesture is the fourth type of gesture.  However, Kim teaches in accordance with a determination that a notification was received within a threshold period of time and in accordance with a determination that the third hand gesture is the fourth type of gesture, performing an action related to the notification (e.g. paragraph 0014, determining level of folding fingers of the user as being in various states (i.e. gestures) based on measured blood flow rate; paragraph 0186, Fig. 5A, user determined in sleep state, generated notifications controlled to not be outputted to user; when user wakes up, user can be informed of saved notifications; paragraph 0187, after detecting user has got out of sleep state, if prescribed command (such as making a fist twice) is received, outputting weather information through the display; paragraph 0190, if user wakes from sleep state, outputting notification screen 501 for cumulative notifications (notification list of plurality of notifications) to user, if prescribed command is received, such as input of making a fist twice, as shown in Figs. 5A-C; i.e. where a notification is received within a time period during which the user is sleeping (analogous to a threshold period of time) and then an input of making a fist gesture is received, an action related to the notification is performed, such as displaying the notification; paragraph 0227, Fig. 11A, call signal currently received, displaying call screen 1101 indicating call signal is currently being received; paragraph 0229, Figs. 11A-C, when call signal currently received, if first input maintained over prescribed time, making response to call signal, and outputting call connected screen 1103; i.e. as shown in Figs. 11A-C, if, during the time period that a call signal is being currently received and a corresponding notification screen is displayed (e.g., during the time period beginning when the call signal is first received and ending when the signal is no longer received, analogous to a threshold period of time within which a notification is received), if a user makes an input such as holding a fist for a time period, the call is answered, analogous to performing an action (answering the call) related to the notification (the screen indicating the call is being received)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Fleizach, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) and Fleizach (directed to providing control of a touch-based user interface absent physical touch capabilities), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to detect the fourth type of gesture (i.e. of Osborn, such as a fist gesture for causing a menu to be displayed) and, based on determining that a received gesture is the fourth type of gesture/fist gesture and further determining that a notification has been received within a threshold time period (such as during a time period in which the user was sleeping, or during a time period in which a call signal is currently received and a corresponding notification is displayed), performing a related action, such as displaying the notification, or answering a corresponding phone call, etc., as taught by Kim.  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
Claims 2, 24, and 42 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Chung et al. (US 20220326779 A1).
With respect to claims 2, 24, and 42, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed.  Although Osborn teaches that auxiliary sensors which may contribute to gesture detection may include heart rate sensors, PPG sensors, and cameras used to detect changes in blood flow within the user (e.g. paragraph 0281, sensors including cardio sensor to measure user’s pulse, heart rate; paragraph 0286, auxiliary device/sensor including heart-rate monitor; paragraph 0310, nonvisible light used to capture image of heat distribution, providing indication of blood flow, used to infer condition of user such as forces being exerted by certain fingers; paragraph 0729, PPG sensors detecting vascular changes), Osborn does not explicitly disclose wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor.
However, Chung teaches wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor (e.g. paragraph 0093, sensor module sensing information for movement of electronic device, and may include photoplethysmogram (PPG) sensor; paragraph 0098, recognizing OCO and shake gestures, gestures of turning hand, moving hand with palm open, lifting arm, moving thumb, moving up/down hand, etc.; gesture may be recognized based on user’s biometric signal (e.g. a signal indicating relaxation/contraction of blood vessels or a heartbeat signal); paragraph 0104, gesture application invoking sensor API to activate sensors, such as PPG sensor; gesture application identifying information for gestures; paragraph 0136, OCO gesture, PPG sensor detecting change due to relaxation and contraction of blood vessels when user opens and closes her hand; ).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Fleizach, and Chung in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), and Fleizach (directed to providing control of a touch-based user interface absent physical touch capabilities), to incorporate the teachings of Chung (directed to recognizing gestures) to include the capability to detect the hand gesture based on heart rate data detected via the optical sensor, such as by detecting a variety of gestures using a PPG sensor, based on a user biometric signal indicating relaxation/contraction of blood vessels or heartbeat (as taught by Chung).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability to recognize a gesture when there is a gesture recognition request from a specific application, eliminating the need for sensors to stay on and the gesture recognition process to keep on running, reducing power consumption and resource waste, and in order to make it possible to immediately recognize relatively small gestures, as described in Chung (paragraphs 0014-0015).
Claims 17, 37, and 55 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Saurabh et al. (US 20180181261 A1).
With respect to claims 17, 37, and 55, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed.  Osborn does not explicitly disclose wherein the menu is displayed:
in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location; and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location.
However, Saurabh teaches wherein the menu is displayed:
in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location (e.g. abstract, positioning popup interface within display frame in relation to corresponding target element; relative positioning of popup interface to associated target element dynamically selected and determined based on location of corresponding target element within quadrants/subquandrants of display frame; paragraph 0001, presenting additional information/menu in the form of a popup interface; paragraph 0004, rendering popup interface at position determined to be best fit position with regard to target element’s position; paragraph 0005, determining dynamically made where to display the popup interface by determining quadrant the target element is actually displayed within; paragraph 0027, pop interface may refer to a context menu; paragraph 0028, target element linked to popup interface; target element may be word or icon, and popup interface may be a selection menu related to the word or icon; paragraph 0077, Fig. 8, target element 802 residing within subquadrant 5/quadrant 1, resulting popup interface 804 displayed at bottom position relative to target element; i.e. where a respective user interface element (target element) is displayed at a position within a first quadrant (analogous to a first location), the corresponding popup menu is also displayed at a position within the same quadrant/location); and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location (e.g. abstract, positioning popup interface within display frame in relation to corresponding target element; relative positioning of popup interface to associated target element dynamically selected and determined based on location of corresponding target element within quadrants/subquandrants of display frame; location of popup interface is dynamically adjustable as opposed to being positioned based on a predetermined location; paragraph 0004, rendering popup interface at position determined to be best fit position with regard to target element’s position, event when that target element changes position or otherwise moves; paragraph 0005, determining dynamically made where to display the popup interface by determining quadrant the target element is actually displayed within; as the target element moves within the user interface (due to scrolling etc.) the relative position of the popup interface is dynamically adjusted so that the popup interface is continuously displayed in a position identified as best fit relative to target element; paragraph 0083, Fig. 10, target element 1002 displayed spanning quadrant1 and quadrant2, resulting popup interface 1004 displayed in bottom position relative to target element 1002; i.e., where a respective user interface element is displayed at a different location such as a location spanning multiple quadrants (analogous to not being displayed at a first location), the corresponding popup menu may also be moved/repositioned on the display so that it is also displayed at a second location which is different from the first location).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Fleizach, and Saurabh in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects, including displaying associated menus) and Fleizach (directed to providing control of a touch-based user interface absent physical touch capabilities), to incorporate the teachings of Saurabh (directed to a positioning mechanism for bubble as a custom tooltip, such as positioning popup interfaces/menus in relation to a corresponding target element) to include the capability to dynamically display the menu (i.e. of Osborn) in a position relative to the position of a corresponding UI object such that when the UI object is displayed at a first location, the menu is also displayed at the first location and when the UI object is not displayed at the first location, such as because it is moved to a different location, the menu is displayed at a second location different from the first location, such as at the different location that the UI object has been moved to (as taught by Saurabh).  One of ordinary skill would have been motivated to perform such a modification in order to solve disadvantages found in existing popup interfaces by providing the capability to dynamically render a popup interface at a position that is determined to be a best fit position with regard to a target elements position, event when that target element changes position or otherwise moves in some manner, as described in Saurabh (paragraphs 0002-0004).
Claims 4, 26, and 44 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Li (US 20160328108 A1) , further in view of Kin (US 10261595 B1), further in view of Pantel (US 20140320434 A1).
With respect to claims 4, 26, and 44, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches
wherein before detecting the hand gesture, the computer system is in a first operation mode (e.g. paragraph 1463, Fig. 35O, fist pose mapped to command to display, enter, and activate radial menu; i.e. where in the first state, at the time the radial menu is displayed, the user has provided a previous input causing the radial menu to be activated, and therefore in an operation mode in which the menu is activated, analogous to a first operation mode),
wherein performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode (e.g. paragraph 1468, Fig. 35Q, illustrating finger pinch pose 351006, which may be mapped to command/instruction to click visual indicator in the radial menu, analogous to mouse button click or keypress; alternatively combination of finger pinch pose and certain degree of force mapped to click command; paragraph 1469, Fig. 35 U, clicking/engaging with or activating function of the currently selected action indicator; clicking action indicator in response to identified click gesture; upon performing the click action, causing visual indicator to appear to depress and/or release, similar to a physical button; i.e. clicking the currently selected action indicator causes the corresponding function to be activated, and therefore causes the system to transition to an operation mode in which the corresponding function has been activated, analogous to a second operation mode)
detecting, via at least the optical sensor, a second hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that the second hand gesture is one type of gesture, toggling a first automatic scrolling operation between an active state and an inactive state (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling up a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll up a GUI; paragraph 1484, ring finger pinch mapped to scroll up so that when user makes ring finger pinch pose, the page/browser displayed scrolls up); and
in accordance with a determination that the second hand gesture is another type of gesture, reversing a first direction of the first automatic scrolling operation  (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling down a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll down a GUI; paragraph 1484, pinky finger pinch mapped to scroll down so that when user makes pinky finger pinch pose, the page/browser displayed scrolls down).
Although Osborn further teaches that user hand gestures may be mapped to different commands in different states/modes, and further teaches that a wrist roll gesture may be mapped to a scroll command (e.g. paragraph 0648, switching between modes; paragraph 0666-0667, selection of mode, such as from list or using gesture; paragraph 1380, transitioning between different mappings of body part states and responsive actions; paragraph 1408-1410, activating first mapping and/or deactivating second mapping between possible body part states and responsive actions; user switching between mappings; transition/switch between mappings; paragraph 1486, user modifying mappings between body states and actions, user configuring/mapping scroll controls to different poses; table accompanying paragraph 1649, indicating that an interaction with a list feature may include a “scroll through” interaction with a “wrist roll” control scheme), Osborn does not explicitly disclose:
wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture,
while the computer system is in the second operation mode, detecting the second hand gesture;
in response to detecting the second hand gesture:
in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation;
in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation; and
in accordance with a determination that the second hand gesture is a fourth type of gesture different from the first type of gesture, the second type of gesture, and the third type of gesture, ending the automatic scrolling operation.
However, Li teaches:
wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture (e.g. paragraph 0037-0038, utilizing auto-scroll method such that contents are presented in multiple steps/views within given time period; user starting content show with auto scroll where contents are presented and scrolled automatically and continuously; user selecting scroll options, etc.; paragraph 0046, auto scroll replaces contents on display at given rate continuously and without user involvement),
while the computer system is in the second operation mode, detecting the second hand gesture (e.g. paragraph 0046, describing user inputs/gestures provided while content is automatically scrolled, such as a finger hover/touch input gesture to pause scrolling; paragraph 0047, describing other types of gesture which may be used as command input, including user holding hand upright with palm facing a horizontal direction, user hand movements to make selections, etc.; paragraph 0053, describing user methods for stopping a scroll process temporarily, such as selecting on screen buttons, touch inputs, and use of gestures via gesture recognition system, such as by moving hand or fingertip in a given direction, where the user’s hand may be in any position, vertical, horizontal, tilted, using both a fist or spread hand for hand gestures, and using one or multiple fingertips for finger gestures);
in response to detecting the second hand gesture:
in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation (e.g. paragraph 0046-0047, user inputs to pause scrolling, including hand gestures; paragraph 0053, user input to stop scroll process temporarily, including use of hand gestures, including hand gestures in a given direction, using a tilted hand, etc.; i.e. the user may provide a hand gesture to pause scrolling, including a directional/tilted hand gesture, analogous to pausing an automatic scrolling operation in accordance with a determination that a hand gesture is a first type of hand gesture);
in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation (e.g. paragraph 0044-0045, scroll direction may be reversed using gesture, such as swipe on touch screen; via buttons, etc.; paragraph 0047, describing use of other gestures as command input, including directional movements to select an option; paragraph 0053, user turning page forward or backward during scroll session, including use of hand gestures, including hand gestures in a given direction, using a tilted hand, etc.; i.e. the user may provide a hand gesture to reverse scrolling direction, or to turn pages in any direction (including an opposite direction) during automatic scrolling, including a directional/tilted hand gesture, analogous to reversing a direction of scrolling in accordance with a determination that a hand gesture is a second type of gesture); and
in accordance with a determination that the second hand gesture is the fourth type of gesture, ending the automatic scrolling operation (e.g. paragraph 0043, selection of button to close content window and end scroll session; selection of no button to stop scroll process; scroll session controlled by device movement, etc.; paragraph 0047, describing various user gestures which may be used as command input, including hand movements to select a no icon/option; paragraph 0053, describing various hand gestures which may be used as command inputs, including hand gestures with a fist; paragraph 0054, option to quit auto scroll prepared; tapping/clicking/selecting quit button; i.e. the user may provide a different hand gesture, such as a closed fist or other hand gesture, in order to provide a command to close/exit the auto scroll operation mode).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Li in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), to incorporate the teachings of Li (directed to presenting contents using auto scroll) to include the capability to place the system in second mode in response to a third type of user input (i.e. such as a mode of executing a corresponding functionality following a user’s selection activating a displayed item, as taught by Osborn, and a mode of initiating automatic and continuous content scrolling functionality following a user’s selection of a corresponding user interface component/option, as taught by Li), where the second mode includes an automatic scrolling function and, while in this automatic scrolling mode, receive an input/gesture to pause the automatic scrolling (including a directional/tilted hand gesture, analogous to a first type of gesture, as taught by Li, similar to the directional wrist rotation gesture as taught by Osborn), receive an input/gesture to reverse the direction of the automatic scrolling (including a directional/tilted hand gesture in a direction corresponding to a reverse direction, as taught by Li, similar to the directional wrist rotation gesture as taught by Osborn), and receive an input/gesture to exit/quit the auto scroll operation/mode (including a gesture using a hand in a fist and/or selecting a corresponding option, as taught by Li, similar to a fist gesture or other gesture as taught by Osborn).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability for the user to relax and watch contents effortlessly, as described in Li (paragraph 0037).
Although Li further teaches user inputs/gestures for changing a speed of the automatic scrolling operation (e.g. paragraphs 0037-0038, user changing scroll rate as needed/setting scroll rate, including using an adjustable button/column; paragraph 0054, user option to adjust scroll rate; paragraph 0071, user selecting option, including scroll rate, to adjust), Osborn and Li do not explicitly disclose in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation.  However, Kin teaches in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation (e.g. col. 14 lines 3-20, detecting pinch scroll gesture, if detecting pinch scroll gesture with thumb and two fingers, speeding up rate of scroll action compared to action with just one finger; after detecting pinch scroll gesture, detecting change in pose of user’s hand while in pinched position, and increasing the rate of scrolling).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, and Kin in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) and Li (directed to presenting contents using auto scroll), to incorporate the teachings of Kin (directed to high resolution tracking and response to hand gestures through three dimensions) to include the capability, while performing automatic scrolling (as taught by Li, incorporated into the system of Osborn), to detect a pinch gesture, or a change in an existing pinch gesture (as taught by Kin, analogous to a third type of gesture as taught by Osborn, such as for selecting a displayed option), and in response, change the speed/rate of the scrolling operation (as taught by Kin, such as by pinching with an increased number of fingers, or changing a hand pose while pinching).  One of ordinary skill would have been motivated to perform such a modification in order to overcome limitations of conventional hand tracking systems, which are unable to accurately track positions of user fingers and thumbs and precise movements of user digits, causing poor representation of the user’s hands, by providing the capability to detect various gestures performed by the user’s hands and cause a change in display instructions in response to the detected gesture as described in Kin (col. 1 lines 20-43).
Assuming arguendo that Osborn, Li, and Kin do not explicitly disclose in accordance with a determination that the second hand gesture is the first type of hand gesture, pausing the automatic scrolling operation, Pantel teaches in accordance with a determination that the second hand gesture is the first type of hand gesture, pausing the automatic scrolling operation (e.g. paragraph 0204, Fig. 17, illustrating forearm-spin gesture, in which smartwatch is rotated about the x axis by twisting the forearm; paragraph 0220, forearm spin gesture can be executed in two directions, and direction can be assigned to different functions/tasks, including, at paragraph 0222, locking of the scroll capability of a page/card/tab; i.e. the user may temporarily lock/pause a scrolling operation/capability using wrist/forearm rotation in an assigned direction).  In addition, assuming arguendo that Osborn, Li, and Kin do not explicitly disclose in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation, Pantel teaches in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation (e.g. paragraph 0175, user performing drag away gesture, such as performing conventional drag gesture to scroll through a page, where user can also change direction of drag movement/page scrolling; paragraph 0204, Fig. 17, forearm spin gesture is variant of drag along/away gesture, where smart watch is rotated around axis by twisting the forearm while index finger is held in place; paragraph 0209, in forearm spin gesture, user touches screen with finger and starts to twist forearm while maintaining finger steady, resulting in dragging of the finger in manner comparable to drag along/away gestures; paragraph 0214-0215, triggering drag completed/flick based on direction/distance/speed; paragraph 0220, forearm spin gesture executed in two directions, direction of gesture assigned to different functions; i.e. the user may also utilize the forearm spin gesture in a given direction to execute functions associated with drag inputs, including causing scrolling to be performed in a given direction, such as a reversed/changed direction).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, and Pantel in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Li (directed to presenting contents using auto scroll), and Kin (directed to high resolution tracking and response to hand gestures through three dimensions), to incorporate the teachings of Pantel (directed to gesture control) to include the capability, while performing automatic scrolling (as taught by Li, incorporated into the system of Osborn), to detect a forearm spin gesture (as taught by Pantel, similar to a wrist rotation gesture as taught by Osborn) in a given direction and either lock/pause a scrolling operation (as taught by Pantel, similar to pausing scrolling according to a hand gesture as taught by Li), or reverse the direction of scrolling (as taught by Pantel, similar to controlling scrolling direction as taught by Osborn and Li).  One of ordinary skill would have been motivated to perform such a modification in order to provide novel gestures in addition to familiar touchscreen gestures, allowing access of several features or functions of an operating system or application using only a single button, icon, or control element, or global system functions using novel gestures at any location regardless of displayed content as described in Pantel (paragraph 0014).
Claims 3, 25, and 43 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Li , further in view of Kin, further in view of Pantel, further in view of Friedman (US 20110090407 A1).
With respect to claims 3, 25, and 43, Osborn in view of Fleizach, further in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture (e.g. paragraph 1473-1474, Figs. 36B-C, first body state mapped to selection of next indicator in given sequence; in response to identification of first body state, deselecting indicator 35111 and selecting indicator 351113; first and second body states may represent counterparts of one another; 1475, Figs. 36D-E, first body state mapped to selection of next visual indicator in the sequence; i.e. because first body state/gesture causes selection to move to a next object in sequence, receipt of multiple instances of this gesture may also cause a different/third UI object to be selected; for example, where multiple instances/two instances of the first type of gesture are received, the object selected as indicated may be first moved from the first UI object to the second/next UI object in the sequence, and then moved again from the second/next UI object to a third/subsequent UI object in the sequence).
Assuming arguendo that Osborn, Fleizach, Li, Kin, and Pantel do not explicitly disclose wherein the second type of gesture that is multiple instances of the first type of gesture (i.e. with respect to the second mode, where this gesture is used to reverse a direction of scrolling), Friedman teaches wherein the second type of gesture that is multiple instances of the first type of gesture (e.g. paragraphs 0046-0047, describing Fig. 5, showing table 500 with gesture motions and respective commands, including rows 504 and 506, with gesture commands having two short down movements, or two short up movements, to scroll down or up a given number of rows, and rows 510 and 512, used to scroll time forwards, or reverse/scroll time backwards within menu (i.e. scroll left or right in a timeline menu); i.e. where a gesture to scroll in a given direction, such as in a reverse direction, may include multiple instances of a given gesture, such as doing the gesture twice with two short movements in order to scroll in a direction, including a direction opposite of a current direction).
To clarify, as cited above, Osborn teaches that in a first mode, a first type of gesture, such as wrist extension/flexion, rotation, radial/ulnar deviation, etc. may be used to change selection to a second user interface object, and Li and Pantel both teach that in a second mode, this same first type of gesture (i.e. a directional/tilted hand gesture) may be used to pause an automatic scrolling operation.  Osborn and Kim both teach that in the first mode, a second type of gesture, which is multiple instances of the first type of gesture, may be used in order to change a selection to a third user interface object, and Li and Pantel both teach that in the second mode, a second type of gesture (which is at least one instance of the first type of gesture) may be used to reverse a direction of the automatic scrolling operation.  Therefore, the only thing missing from this combination is that the second type of gesture (which may include at least one instance of the first type of gesture as taught by Li and Pantel), performed in the second mode for reversing the direction of the scrolling operation, is performed using multiple instances of the gesture.  However, as discussed and cited above, Friedman teaches that the second type of gesture, which can be used to set/reverse a given scrolling direction, can be performed using multiple instances of the gesture.
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Friedman in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions) and Pantel (directed to gesture control), to incorporate the teachings of Friedman (directed to gesture controls) to include the capability, while performing automatic scrolling in a given direction (as taught by Li, incorporated into the system of Osborn), to detect multiple instances of a given gesture (i.e. such as the forearm spin gesture as taught by Pantel, similar to a wrist rotation gesture as taught by Osborn) in a given direction and reverse the direction of scrolling (as taught by Friedman, similar to controlling scrolling direction as taught by Osborn, Li, and Pantel; i.e. utilizing multiple instances of a gesture, as taught by Friedman, in order to change scrolling direction, such as multiple instances of the wrist tilt/rotation gesture as taught by at least Osborn and Pantel).  One of ordinary skill would have been motivated to perform such a modification in order to provide a simplified yet flexible user interface as described in Friedman (paragraph 0018).
Claims 10, 30, and 48, are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Li, further in view of Kin, further in view of Pantel, further in view of Kim et al. (US 20160029899 A1).
With respect to claims 10, 30, and 48, Osborn in view of Fleizach, further in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
in accordance with a determination that the second hand gesture is one type of gesture, displaying a notification that indicates a state of the automatic scrolling operation (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling up a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll up a GUI; paragraph 1484, ring finger pinch mapped to scroll up so that when user makes ring finger pinch pose, the page/browser displayed scrolls up; paragraph 1459, providing visual feedback of both identified pose and identified force; when action includes scrolling GUI, displaying cursor in connection with scrolling, which may be presented as a horizontal line with a bar extending above or below the line, depending on the scrolling direction; paragraph 1480, providing current state/setting of interface, current body state, and current action corresponding to current body state; paragraph 1481, graphical elements displaying current interface state, current body state, and responsive action; i.e. based on determining that the user has provided a gesture for scrolling, the system may initiate the scroll operation and may also provide a notification/indicator regarding the state of the operation, such as a cursor indicating a scrolling direction and magnitude, etc.).
Kim teaches that the one type of gesture is the first type of gesture (e.g. paragraphs 0251-254, Figs. 16A-C, outputting notification screen of received text message, receiving message check command such as folding input and outputting message check screen 1601 for displaying details of received text message; scroll action on message check screen performed by rotation of wrist; wrist rotated in first direction, scrolling to move outputted message check screen; direction for rotating wrist includes scroll direction; if scroll in bottom direction desired, rotating wrist in bottom direction; i.e. the user may provide a wrist rotation input in a first or second direction in order to perform a scroll operation in a designated direction, as shown between Fig. 16B and Fig. 16C).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Fleizach, Li, Kin, Pantel, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Fleizach (directed to providing control of a touch-based user interface absent physical touch capabilities), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to receive a hand gesture to initiate automatic scrolling in a direction, and a hand gesture to reverse the scrolling direction (i.e. such as using different fingers to perform pinch gestures to scroll up/down as taught by Kim), where the hand gesture may be a wrist rotation gesture in different directions corresponding to the desired scroll direction (i.e. as taught by Kim, where Osborn also teaches the use of wrist rotation for at least scrolling a wrist, and further teaches that the wrist rotation may be performed in two different directions, and further teaches, based on detecting the scroll gesture, outputting a notification/indicator of the scrolling operation state).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
Claims 11, 31, and 49 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Li, further in view of Kin, further in view of Pantel, further in view of Kim et al. (US 20160029899 A1), further in view of Coenen et al. (US 20190191208 A1).
With respect to claims 11, 31, and 49, Osborn in view of Fleizach, further in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the third type of gesture is a type of gesture that is an instance of the fourth type of gesture (e.g. paragraph 1281, fist squeeze used for discrete event such as to engage or disengage a particular feature within a given application or a continuous event, such as to maintain activation of a particular feature within a given application; paragraph 1283, selecting from presented options by performing hand gesture such as clenching the fist).
Osborn does not teach that the third type of gesture is multiple instances of the fourth type of gesture.  However, Kim teaches the third type of gesture is multiple instances of the fourth type of gesture (e.g. paragraph 0187, user command of making fist twice to cause output of weather information; i.e. as shown in Fig. 5A, where a UI element associated with a weather information check is currently displayed, the user may perform a gesture which is multiple instances of a fist clenching gesture in order to select/display content corresponding to the UI element).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to receive a hand gesture to select a displayed item on the screen(i.e. such as a fist clench gesture as taught by Osborn) as multiple instances of the hand gesture (i.e. such as receiving multiple fist clench gestures as taught by Kim).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
Assuming arguendo that Osborn, Fleizach,Li, Kin, Pantel, and Kim do not explicitly disclose that the third type of gesture is multiple instances of the fourth type of gesture, Coenen teaches the third type of gesture is multiple instances of the fourth type of gesture (e.g. abstract, paragraph 0011, 0082, scrolling speed increased by performing multiple gestures in quick succession).
To clarify, as cited above, Osborn teaches that in a first mode, a third type of gesture, such as finger pinching or fist squeezing, may be used to select a currently indicated object, and Kin teaches that in a second mode, this same third type of gesture (i.e. pinching with a given number of fingers, including all fingers) may be used to change a scrolling speed.  Osborn additionally teaches that in the first mode, a fourth type of gesture, which may be the same as the third type of gesture (i.e. making a fist), may be used in order to display a menu, and Li teaches that in the second mode, the fourth type of gesture, which may be the same as the third type of gesture (i.e. a gesture using a fist) may be used to end/exit the automatic scrolling operation.  Kim additionally teaches that, in the first mode, the third type of gesture may be multiple instances of the fourth type of gesture (i.e. multiple instances of fist clenching/squeezing to select an object for display).  Therefore, the only thing missing from this combination is that the third type of gesture (which may include at least one instance of the fourth type of gesture as taught by Osborn, Li, and Kin), performed in the second mode for changing speed of the scrolling operation, is performed using multiple instances of the gesture.  However, as discussed and cited above, Coenen teaches that the third type of gesture, which can be used to change speed of the scrolling operation, can be performed using multiple instances of the gesture.
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Coenen in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions) and Pantel (directed to gesture control), to incorporate the teachings of Coenen (directed to gestures for scrolling in a user interface) to include the capability, while performing automatic scrolling (as taught by Li, incorporated into the system of Osborn), to detect multiple instances of a given gesture (i.e. such as the fist clenching gesture taught by Osborn, Li, and Kim, and the similar gesture), and in response change the speed of scrolling (as taught by Coenen, i.e. utilizing multiple instances of a gesture, as taught by Coenen, in order to change scrolling speed, such as multiple instances of the fist gesture as taught by at least Osborn, Li, and Kim (where Kim also teaches use of multiple instances of the fist gesture in a different mode)).  One of ordinary skill would have been motivated to perform such a modification in order to provide gestures which are more intuitive to the user than controlling scrolling based on speed or acceleration of gestures as described in Coenen (paragraph 0011).
Claims 8, 28, and 46 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Li, further in view of Kin, further in view of Pantel, further in view of Platzer et al. (US 20130239046 A1).
With respect to claims 8, 28, and 46, Osborn in view of Fleizach, further in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn and Li further teach wherein:
the automatic scrolling operation scrolls a sequence of a plurality of interface objects in a second direction (e.g. Osborn paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling down a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll down a GUI; paragraph 1484, pinky finger pinch mapped to scroll down so that when user makes pinky finger pinch pose, the page/browser displayed scrolls down; Li paragraph 0038, user selecting scroll options, including scroll direction; paragraph 0044-0045, scroll direction may be reversed using gesture, such as swipe on touch screen; via buttons, etc.).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Fleizach, Kin, Pantel, and Li in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Fleizach (directed to providing control of a touch-based user interface absent physical touch capabilities), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Li (directed to presenting contents using auto scroll) to include the capability to perform the automatic scrolling operation in a second direction (i.e. in a specified direction, as taught by Li).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability for the user to relax and watch contents effortlessly, as described in Li (paragraph 0037).
Osborn does not explicitly disclose wherein the one or more programs further includes instructions for:
while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects; and
in response to detecting the end of the sequence of the plurality of user interface objects, performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction.
However, Platzer teaches wherein the one or more programs further includes instructions for:
while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects (e.g. paragraph 0077, Figs. 5A-C, scrolling list of items to terminus of the list; paragraph 0078, terminus is edge of scrolled region; while list is being scrolled; terminus is virtual boundary associated with displayed objects; displayed objects reach or intersect with the terminus 514; i.e. the system detects the end of a list of displayed UI objects, as indicated by a terminus at the edge of the scrolled list); and
in response to detecting the end of the sequence of the plurality of user interface objects, performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction (e.g. paragraph 0077, Figs. 5A-C, reaching terminus of list, smoothly bouncing displayed items at end of list, reversing direction; paragraph 0078, as illustrated in Fig. 5B, when objects reach or intersect with the terminus 514, the movement corresponding to scrolling may stop and, as illustrated in Fig. 5C, subsequently reverse direction; i.e. scrolling list reaches and bounces at its terminus; i.e. in response to reaching/detecting the terminus while scrolling in a second direction (such as opposite of a first direction), the system automatically performs a “bounce” by scrolling the list of displayed objects in the opposite direction (which may be a third direction, which may be the same as a first direction)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Platzer in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects, such as scrolling menus/lists of UI objects, webpages, etc.), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Platzer (directed to scrolling operations) to include the capability to, while scrolling user interface content, including a plurality of objects, in a given direction (i.e. as taught by Osborn, Kim, and Platzer, such as in a second direction which is opposite of a first scroll direction, or some other direction, where at least Jose teaches a scrolling direction may be set as left-right, right-left, up-down, or down-up), detect a terminus indicating the end of the plurality of objects and, in response automatically scroll the content including the plurality of objects in a reverse direction (as taught by Platzer, where this reverse direction may be a third direction which is different from the second direction, but potentially the same as a first direction).  One of ordinary skill would have been motivated to perform such a modification in order to provide operations desired by users, such as scrolling operations on a device, including on device having limited display size, user interface, etc. which limits the ease of use of the device, as described in Platzer (paragraphs 0006, 0008).
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain,” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting in re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (GCPA 1968)). Further, a reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill the art, including nonpreferred embodiments. Merck & Co, v. Biocraft Laboratories, 874 F.2d 804, 10 USPQ2d 1843 (Fed. Cir.), cert, denied, 493 U.S. 975 (1989). See also Upsher-Smith Labs. v. Pamlab, LLC, 412 F,3d 1319, 1323, 75 USPQ2d 1213, 1215 (Fed. Cir, 2005): Celeritas Technologies Ltd. v. Rockwell International Corp., 150 F.3d 1354, 1361, 47 USPQ2d 1516, 1522-23 (Fed. Cir. 1998).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Yoon et al. (US 20150061842 A1) teaches various gestures for controlling a mobile device such as a wrist worn device (Figs. 7-33).
Offenberg et al (US 20150248235 A1) teaches various gestures for controlling a mobile device such as a wrist worn device (Figs. 27-92), as well as for selecting among displayed items with corresponding visual indications, scrolling, etc. (Figs. 102A-D).
Hazra et al. (US 20170123487 A1) teaches various gestures for controlling a mobile device including a wrist worn device (Figs. 5, 8-15).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action.  Accordingly, THIS ACTION IS MADE FINAL.  See MPEP § 706.07(a).  Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action.  In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action.  In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.

APPLICANT REBUTTAL: REMARKS
Claims 1-4, 8, 10-26, 28, 30-44, 46, and 48-59 were previously pending. By virtue of this
amendment, claims 1, 3, 4, 8, 10, 11, 14-18, 22, 23, 25, 26, 28, 30, 31, 34-38, 43, 44, 46, 48, 49, and 52-56 have been amended. The amendments to independent claims 1, 22, and 23 are supported by at least paragraph [0205] and FIG. 14 of the Application as-filed. The amendments to the dependent claims are for providing consistency across the claims due to the amendment to claims 1, 22, and 23 and are fully supported by the Application as-filed. Thus, no new matter has been added. Claims 1-4, 8, 10-26, 28, 30-44, 46, and 48-59 remain pending.
Applicant respectfully requests reconsideration of this application based on the foregoing amendments and the following remarks.
Examiner Interview
Applicant thanks Examiner Stanley for the telephonic interview on January 16, 2024 with Applicant's representatives Horace Zhang and Christopher Eide. During the interview, the attendees discussed the 103 rejection of claim 1 over Osborn, Li, Kin, and Pantel. In particular, Applicant's representatives proposed amending the independent claims as indicated above. The Examiner indicated that the amendment likely overcomes the present 103 rejection of the independent claims, but that further search and consideration would be required to determine allowability.
The attendees further discussed the 103 rejection of claim 3 over Osborn, Li, Kin, and Pantel and the 103 rejection of claim 11 over Osborn, Li, Kin, Pantel, and Kim. In particular, Applicant's representatives explained that the respective cited references fail to teach or suggest claims 3 and 11. The Examiner indicated that he understood Applicant's explanation and that further
consideration of claims 3 and 11 would be required. (Applicant Initiated Interview Summary dated January 23, 2024.)
Claim Re
j
ections 35 USC 103
Independent Claims
Claims 1, 3, 4, 12, 14-16, 19-23, 25-27, 32, 34-36, 39-41, 43-45, 50, 52-54, 57, and 58 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn et al. (US 20230072423 Al) in view of Li (US 20160328108 Al), further in view of Kin (US 10261595 81), further in view of Pantel (US 20140320434 Al).
Applicant has amended independent claim 1 (and similarly independent claims 22 and 23) to recite, inter alia:
in accordance with a determination that the hand gesture is a fourth type of gesture that is different from the first type of gesture, the second type of gesture, and the third type of gesture, displaying, via the display generation component, a menu, wherein: a first control and a second control are available via the menu; selection of the first control causes the computer system to perform an operation corresponding to a press of the first hardware input mechanism; and selection of the second control causes the computer system to perform an operation corresponding to a rotation of the second hardware input mechanism.
Amended claims 1, 22, and 23 (and all claims depending therefrom) are allowable over the cited references for at least the reasons discussed below.
As an initial matter, the Examiner agreed during the interview that the amendment would overcome the present 103 rejection. For this reason alone, Applicant respectfully requests that the 103 rejection of claims 1, 22, and 23 (and all claims depending therefrom) be withdrawn and the claims be allowed.
Further, the cited references fail to teach or suggest the amended limitations of claims 1, 22, and 23. Specifically, while Osborn teaches displaying a menu in response to detecting a hand
gesture, Osborn fails to teach or suggest that the menu includes controls to perform respective operations that correspond to pressing a hardware input device and rotating a hardware input device. Li, Kin, and Pantel fail to cure the deficiencies of Osborn.
In view of the above, Applicant respectfully requests that the 103 rejection of claims 1, 22,
and 23 (and all claims depending therefrom) be withdrawn and the claims be allowed.
Claims 3
,
,
,
,
,
and 49
Claims 3 and 11 depend from claim 4. Claims 3, 4, and 11 are reproduced below.
4. The computer system of claim 1, wherein before detecting the hand gesture, the computer system is in a first operation mode, wherein performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode, wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture, and wherein the one or more programs further include instructions for:
while the computer system is in the second operation mode, detecting, via at least the optical sensor, a second hand gesture; and
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation; in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation; in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation; and in accordance with a determination that the second hand gesture is the fourth type of gesture, ending the automatic scrolling operation.
3. The computer system of claim 4, wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture.
11. The computer system of claim 4, wherein the third type of gesture is a type of gesture that is multiple instances of the fourth type of gesture.
Accordingly, claim 3 specifies that the computer system detects the "first type of gesture" to pause autoscroll and detects multiple instances of the "first type of gesture" to reverse the autoscroll direction.
Similarly, claim 11 specifies that computer system detects the "third type of gesture" to
change the speed of autoscrolling and detects multiple instances of the "third type of gesture" to end autoscrolling.
Claims 3, 11, 25, 31, 43, and 49 are allowable for at least the reasons discussed below.
Initially, claims 3, 11, 25, 31, 43, and 49 depend variously from independent claims 1, 22, and 23, which are allowable for at least the reasons discussed above. Thus, claims 3, 11, 25, 31, 43, and 49 are allowable for at least the reason that they each depend from a respective allowable independent claim. Claims 3, 11, 25, 31, 43, and 49 are allowable for at least the further reasons discussed below.
Claims 3, 25, and 43 stand rejected under 103 over Osborn, Li, Kin, and Pantel. The cited portions of Osborn teach that a gesture navigates by one unit (e.g., moves forward by one). This, according to the Examiner, suggests that multiple instances of the gesture navigate by multiple units.
However, none of Osborn, Li, Kin, or Pantel teach that that the gesture that reverses autoscroll direction is multiple instances ofthe gesture that pauses autoscroll.
Claims 11, 31, and 49 stand rejected under 103 over Osborn, Li, Kin, Pantel, and Kim.
Specifically, the Office Action relies on Osborn and Kim for allegedly teaching claim 11. The cited portions of Osborn teach using a fist squeeze gesture to engage (e.g., select) an element or to maintain activation of an element. The cited portions of Kim teach using a double fist squeeze gesture to display information.
However, none of Osborn, Li, Kin, Pantel, or Kim teach or suggest that the gesture that ends autoscrolling is multiple instances ofthe gesture that changes the speed of autoscrolling.
Applicant thus respectfully requests the 103 rejection of claims 3, 11, 25, 31, 43, and 49 be withdrawn and the claims be allowed.
Remainin
g
Claims
Claims 10, 11, 13, 18, 30, 31, 33, 38, 48, 49, 51, and 56 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Li, further in view of Kin, further in view of Pantel, further in view of Kim et al. (US 20160029899 Al ).
Claims 10, 11, 13, 18, 30, 31, 33, 38, 48, 49, 51, and 56 depend variously from independent claims 1, 22, and 23, which are allowable for at least the reasons discussed above. Thus, claims 10, 11, 13, 18, 30, 31, 33, 38, 48, 49, 51, and 56 are allowable for at least the reason that they each depend from a respective allowable independent claim. The addition of Kim fails to cure the deficiencies of Osborn, Li, Kin, and Pantel described above, nor is it alleged to. Applicant thus respectfully requests the 103 rejection of claims 10, 11, 13, 18, 30, 31, 33, 38, 48, 49, 51, and 56 be withdrawn and the claims be allowed.
Claims 2, 24, and 42 are rejected under 35 U.S.C.103 as being unpatentable over Osborn in view of Li, further in view of Kin, further in view of Pantel, further in view of Chung et al. (US 20220326779 Al).
Claims 2, 24, and 42 depend variously from independent claims 1, 22, and 23, which are allowable for at least the reasons discussed above. Thus, claims 2, 24, and 42 are allowable for at least the reason that they each depend from a respective allowable independent claim. The addition of Chung fails to cure the deficiencies of Osborn, Li, Kin, and Pantel described above, nor is it alleged to. Applicant thus respectfully requests the 103 rejection of claims 2, 24, and 42 be withdrawn and the claims be allowed.
Claims 8, 28, and 46 are rejected under 35 U.S.C.103 as being unpatentable over Osborn in view of Li, further in view of Kin, further in view of Pantel, further in view of Platzer et al. (US 20130239046 Al).
Claims 8, 28, and 46 depend variously from independent claims 1, 22, and 23, which are allowable for at least the reasons discussed above. Thus, claims 8, 28, and 46 are allowable for at least the reason that they each depend from a respective allowable independent claim. The addition of Platzer fails to cure the deficiencies of Osborn, Li, Kin, and Pantel described above, nor is it alleged to. Applicant thus respectfully requests the 103 rejection of claims 8, 28, and 46 be withdrawn and the claims be allowed.
Claims 17, 37, and 55 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Li, further in view of Kin, further in view of Pantel, further in view of Saurabh et al. (US 20180181261 Al).
Claims 17, 37, and 55 depend variously from independent claims 1, 22, and 23, which are allowable for at least the reasons discussed above. Thus, claims 17, 37, and 55 are allowable for at least the reason that they each depend from a respective allowable independent claim. The addition of Saurabh fails to cure the deficiencies of Osborn, Li, Kin, and Pantel described above, nor is it alleged to. Applicant thus respectfully requests the 103 rejection of claims 17, 37, and 55 be withdrawn and the claims be allowed.
CONCLUSION

NEXT OFFICE ACTION: DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the Amendment filed on April 30, 2024 and Request for Continued Examination filed on May 20, 2024.  Claims 1, 11, 22, 23, 31, 44, and 49 are amended.  Claims 5, 6, 7, 9, 27, 29, 45, and 47 are cancelled.  Therefore, Claims 1-4, 8, 10-26, 28, 30-44, 46, and 48-59 are pending in the case.  Claims 1, 22, and 23 are the independent claims.
This action is non-final.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection.  Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114.  Applicant's submission filed on April 30, 2024 has been entered.
Applicant’s Response
In the Amendment filed on April 30, 2024, Applicant amended the claims and provided arguments in response to rejection of the claims under 35 USC 103 in the previous office action.
Response to Argument/Amendment
Applicant’s amendments to the claims in response to the rejection of the claims under 35 USC 103 in the previous office action are acknowledged, and Applicant’s associated arguments have been fully considered.
Regarding the independent claims, Applicant argues that the cited references (Osborn and Fleziach) fail to teach or suggest a second hardware input mechanism that is physically rotatable by a user of the computer system and that is physically depressible by the user of the computer system, and selection of the second control causes the computer system to perform an operation corresponding to a physical rotation, relative to the computer system, of the second hardware input mechanism, as recited in the amended claims.  Applicant’s arguments are persuasive, and the rejection is withdrawn.  However, new grounds of rejection are provided below.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims under pre-AIA  35 U.S.C. 103(a), the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were made absent any evidence to the contrary.  Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and invention dates of each claim that was not commonly owned at the time a later invention was made in order for the examiner to consider the applicability of pre-AIA  35 U.S.C. 103(c) and potential pre-AIA  35 U.S.C. 102€, (f) or (g) prior art under pre-AIA  35 U.S.C. 103(a).
Claims 1, 12, 14, 15, 16, 19, 20, 21, 22, 23, 32, 34, 35, 36, 39, 40, 41, 50, 52, 53, 54, 57, 58, and 59 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn et al. (US 20230072423 A1) in view of Chae et al. (US 21020306748 A1), further in view of Undasan (US 20060010402 A1).
With respect to claims 1, 22, and 23, Osborn teaches a computer system that is in communication with a display generation component and an optical sensor, the computer system comprising: a first hardware input mechanism (e.g. paragraph 0386, vibrotactile system having touch-sensitive pads, pressure sensors, on/off button, etc.; paragraph 1605, text entry control signal via physical or electrical button, switch, touchscreen, etc.); a second hardware input mechanism (e.g. paragraph 0278, system including IMUs such as accelerometer, gyroscope, magnetometer, etc.; paragraph 0352, sensors including accelerometers, gyroscopes, magnetometers, etc.; paragraph 0386, vibrotactile system having motion/position sensors); one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method; a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system that is in communication with a display generation component and an optical sensor, the one or more programs including instructions for performing the method (e.g. paragraph 0257, supplementing sensor data with images of user captured by camera; paragraph 0286, auxiliary devices/sensors including imaging devices/cameras, heart rate monitors, etc.; paragraph 0288, Figs. 9A-B, wearable system with sensors such as EMG sensors, sensors 810; computer processor such as microcontroller 934 receiving inputs from sensors; communicating with dongle portion 920 and outputting processing output to dongle portion; paragraph 0289, dongle portion 920 receiving signals from wearable portion 910, and providing them to host computer for further processing, display, effecting control of objects, etc.; paragraph 0292, describing Fig. 1, system including processors 112 programmed to communicate with sensors 110 and executing algorithms; paragraph 0301, system including display controller configured to display visual representations; computer processors implementing trained inference models that receive sensor signals as input and provide information to generate control signals for system as output; paragraph 0302, user interface; paragraph 0303, various types of devices in which processors may be included; paragraph 0304, multiple processors; paragraph 0306, system including processors, camera, and display; paragraph 310, camera information/cameras; paragraph 0311, camera mounted on user; paragraph 0344, embodiments implemented in software executed on processors; paragraph 0345, non-transitory computer-readable storage medium/memory encoded with computer program/instructions executed on processor to perform described functions; paragraph 0352, sensor including light emitter/detector; paragraph 0385, Fig. 13, vibrotactile system 1300 may be standalone system, or may be configured with interaction with various types of other devices, and includes processor 1360; paragraph 0610, cameras; paragraph 0729, photoplethysmography/PPG sensors detecting vascular changes/changes in blood volume; paragraph 1384, Fig. 35B, wearable device 35102 capable of controlling computer devices via neuromuscular signals of users, including set of sensors 35104 that detect/sense neuromuscular signals traversing body of user; processing device 35106 communicatively coupled to sensors, and memory 35108 including models and computer-executable instructions that when executed by processor cause it to identify force generated, generate commands for HCI system, etc.; i.e. the device includes at least one processor and memory storing instructions for execution by the processor, and is further in communication with at least one display/display controller/generation component, and at least one optical sensor, such as a camera (which may be mounted on the user)); and the method, comprising:
displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected (e.g. paragraph 1460, describing radial menu 351000 in Figs. 35O-V, where the menu may include visual indicators including a selection indicator 351020; paragraph 1461, selection indicator 351020 may indicate a currently selected action indicator; paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual, auditory, or haptic feedback; visual feedback representing change of a characteristic of visual element within radial menu in response to selection of the visual indicator, such as position change, orientation change, color change, size change, transparency change, fill change, emphasis change, shadow change, animation change, font change, line type change, line width change, combinations, etc.; paragraph 1473, describing Figs. 36B-C, radial menu 351110 including a set of action indicators including indicators 351111 and 351113 displayed in GUI along arc, where as shown in Fig. 36B includes an indication that indicator 35111 is selected; demonstrating/confirming selection of visual indicator using visual feedback; paragraph 1475, describing Figs. 36D-E, displaying sequential menu 351120 including a sequence of visual indicators depicted as A through F, where as shown in Fig. 36 D includes in indication that indicator D/351123 is selected; i.e. both of the menus/interfaces shown in the cited figures include at least first, second, and third UI objects along with an indication that an object of these is selected, such as a visual feedback including a shading/coloration of a selected indicator; paragraphs 1385-1387, determining based on neuromuscular signals state of body part of user and generating input commands accounting for state of user’s body part; sensors constitute any type of form of sensor capable of detecting/sensing neuromuscular signals in user’s body including EMG sensors, MMG sensors, SMG sensors; sensors may also include auxiliary sensors such as IMU, position tracking, imaging (camera), and radiation based sensors, microphones, heart rate monitors, etc.; paragraph 1394, wearable device 35102 coupled with computing system/display; user placing body in state/condition to control/modify presentation/performance of computing system, generating neuromuscular signals representative/indicative/suggestive of state/action);
while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture (e.g. paragraph 0310, camera information used to improve interpretation of neuromuscular signals; camera used to record image of user, may be within particular range/wavelengths of light, images produced by visible/non-visible light, including showing heat distribution in user’s body, providing indication of blood flow within user and used to infer condition of user such as forces being exerted by fingers; paragraph 0311, camera mounted on user; paragraph 0312, camera data in addition to neuromuscular activity signals used to reconstruct positions, geometries, and forces being applied by user’s arm/hand; paragraph 0323, auxiliary signals from auxiliary devices such as camera supplementing neuromuscular signals; paragraph 0610, cameras used to improve interpretation of signals; paragraph 0611, camera data used to provide real time determination of musculoskeletal representations of gestures; paragraph 0729, PPG sensors detecting vascular changes; paragraph 1271, camera/imaging data; paragraph 1387, sensors including auxiliary sensors such as cameras; paragraph 1395, user pose/gesture generating neuromuscular signals as they traverse down arm toward hand; sensors 35104 detecting/sensing neuromuscular signals and generating data representative of the signals; paragraph 1396-1398, sensor data undergoing processing, analyzing data to determine state of body parts of user, producing/outputting classification indicating state of body part; paragraph 1398, gestures; paragraph 1400, causing computing system to perform actions mapped to state of body parts; paragraph 1401, describing various actions mapped to different states of user’s body parts; paragraph 1444, capturing gestures with camera; photographic data of hand gestures; paragraph 1473, body state mapped to selection of next indicator in a given sequence; identification of body state; paragraph 1474, body states including wrist extension, flexion, rotation, radial deviation, ulnar deviation, etc.; paragraph 1475, body states mapped to selection of visual indicators in sequence; identification of body states; i.e. detecting a body state of the user, such as a hand gesture, including by using an optical sensor such as a camera,  PPG, etc., possibly in combination with other sensors),
in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is a first type of gesture, displaying, via the display generation component, an indication that the second user interface object is selected (e.g. paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual feedback; paragraph 1473, Figs. 36B-C, first body state mapped to selection of the next indicator in a given sequence; in response to identification of first body state, deselecting indicator 35111 and selecting indicator 351113; demonstrating/confirming selection of visual indicator using visual feedback; paragraph 1475, Figs. 36D-E, first body state mapped to selection of next visual indicator in the sequence; i.e. in response to detecting a first type of gesture, as indicated by a first body state, the indication that a first UI object is selected may be changed to an indication that a second UI object is selected, such as by moving the indicator of the selection to the second UI object, where the second UI object is a next UI object in a sequence with respect to the first UI object); and
in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected (e.g. paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual feedback; paragraph 1473-1474, Figs. 36B-C, first body state mapped to selection of next indicator in given sequence; demonstrating/confirming selection of visual indicator using visual feedback; first and second body states may represent counterparts of one another; 1475, Figs. 36D-E, second body state mapped to selection of previous visual indicator in the sequence; in response to identification of second body state, deselecting subsequent indicator 351123 and selecting prior indicator 351121; i.e. in response to detecting a second/different type of gesture, as indicated by a second body state, the indication that a first UI object is selected may be changed to an indication that a third UI object is selected, such as by moving the indicator of the selection to the third UI object, where the third UI object is a previous UI object in a sequence with respect to the first UI object; in addition, because these gestures cause selection to move to a next/previous object in sequence, receipt of multiple of either of these gestures may also cause a different/third UI object to be selected; for example, where multiple instances/two instances of the first type of gesture are received, the object selected as indicated may be first moved from the first UI object to the second/next UI object in the sequence, and then moved again from the second/next UI object to a third/subsequent UI object in the sequence); and
in accordance with a determination that the hand gesture is a third type of gesture that is different from the first type of gesture and the second type of gesture, performing an operation corresponding to selection of the first user interface object (e.g. paragraph 1281, fist squeeze used for discrete event such as to engage or disengage a particular feature within a given application or a continuous event, such as to maintain activation of a particular feature within a given application; paragraph 1283, selecting from presented options by performing hand gesture such as clenching the fist; paragraph 1468, Fig. 35Q, illustrating finger pinch pose 351006, which may be mapped to command/instruction to click visual indicator in the radial menu, analogous to mouse button click or keypress; alternatively combination of finger pinch pose and certain degree of force mapped to click command; paragraph 1469, Fig. 35 U, clicking/engaging with or activating function of the currently selected action indicator; clicking action indicator in response to identified click gesture; upon performing the click action, causing visual indicator to appear to depress and/or release, similar to a physical button; i.e. clicking the currently selected action indicator causes the corresponding function to be activated); and
in accordance with a determination that the hand gesture is a fourth type of gesture that is different from the first type of gesture, the second type of gesture, and the third type of gesture, displaying, via the display generation component, a menu (e.g. paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user (as compared to other movements/poses causing item selection/clicking); paragraph 1485, displaying interface 351300 of Fig. 36G, which may be a popup box/dialog that appears in response to user input; i.e. where a some suitable user input, such as a fist clenching gesture, may be used to open various menus, such as a menu including selectable options).
Osborn does not explicitly disclose that the second hardware input mechanism is a second hardware input mechanism that is physically rotatable by a user of the computer system and that is physically depressible by the user of the computer system.  However, Chae teaches that the second hardware input mechanism is a second hardware input mechanism that is physically rotatable by a user of the computer system and that is physically depressible by the user of the computer system (e.g. paragraph 0004-0005, mobile terminal types including wrist watch type; ultra-compact mobile terminal, such as a watch phone; paragraph 0009-0010, user input unit includes scroll key/wheel that is operable in an up rotational direction, a down rotational direction, and a push direction; paragraph 0106, Fig. 5, scroll key 131 for displaying and selecting key groups; paragraph 0107, when not in character input mode, scroll key 131 may perform movement between menus and symbols; when scroll key 131 is pressed, indicator displayed; moving indicator on menu screen by operating scroll key 131; paragraph 0109, scroll key may be disposed on front, rear, or side of terminal; not limited to described embodiments, can be modified to different form such as a wheel key; paragraph 0115, scroll key allows rotation input and push input; paragraph 0161, Fig. 20, showing ultra-compact device such as watch phone including scroll key 131).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Chae in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) to incorporate the teachings of Chae (directed to an ultra compact mobile terminal, such as a watch, having a scroll key/wheel input device) to include, as a hardware input mechanism (i.e. for the device of Osborne), a scroll key/wheel device which is both rotatable and depressible (as taught by Chae).  One of ordinary skill would have been motivated to perform such a modification in order to provide an ultra compact mobile terminal for inputting characters using the smallest number of keys, and for conveniently displaying menus and changing screen images, as described in Chae (paragraph 0007).
Osborn does not explicitly disclose wherein:
a first control and a second control are available via the menu;
selection of the first control causes the computer system to perform an operation corresponding to a press of the first hardware input mechanism; and
selection of the second control causes the computer system to perform an operation corresponding to a physical rotation, relative to the computer system, of the second hardware input mechanism.
However, Undasan teaches wherein:
a first control and a second control are available via the menu (e.g. paragraph 0024, rendering cursor icon within GUI; paragraph 0029, cursor icon including selection regions/objects each of which may emulate pre-defined function corresponding to actuation of an actuator; paragraph 0045, describing Fig. 6, cursor icon 600 including selection regions 608-618);
selection of the first control causes the computer system to perform an operation corresponding to a press of the first hardware input mechanism (e.g. paragraph 0029, emulating functions of a mouse; icon displayed on GUI display, including permanently; selection regions emulate pre-defined function corresponding to actuation of an actuator, such as pressing a mouse button; paragraph 0045, cursor icon substitutes conventional mouse functionality; paragraph 0046, Fig. 6, selection regions represent actuators found on two-button scroll wheel mouse, namely left button down 608, left double click 610, right button down 612, right double click; moving cursor to selection region to invoke mouse actuation corresponding to the respective region; invoking left button down actuation, invoking actuation left button up); and
selection of the second control causes the computer system to perform an operation corresponding to a physical rotation, relative to the computer system, of the second hardware input mechanism (e.g. paragraph 0029, emulating functions of a mouse; icon displayed on GUI display, including permanently; selection regions emulate pre-defined function corresponding to actuation of an actuator, such as turning a scroll wheel; paragraph 0045, cursor icon substitutes conventional mouse functionality; paragraph 0046, Fig. 6, selection regions including scroll up 616 and scroll down 618; moving cursor to selection region to invoke mouse actuation corresponding to the respective region; invoking scroll up actuation; emulating sequence of scroll up actuations by generating appropriate data as if these were actually generated by a user operating a mouse scroll wheel).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Chae, and Undasan in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) and Chae (directed to an ultra compact mobile terminal, such as a watch, having a scroll key/wheel input device), to incorporate the teachings of Undasan (directed to GUI navigation, including via a GUI item/element which emulates hardware inputs such as button and scroll wheel inputs) to include the capability to display, via the menu (i.e. of Osborn), a first control which, when selected, causes the device to perform operations corresponding to pressing of a physical button of a hardware input device (such as a mouse as taught by Undasan, or various of the physical input buttons as taught by Osborn and Chae) and a second control which, when selected, causes the device to perform operations corresponding to rotation of a hardware input mechanism (i.e. such as rotation of a scroll wheel, where such as the mouse scroll wheel of Undasan, or the similar scroll key/wheel of Chae), as taught by Undasan.  One of ordinary skill would have been motivated to perform such a modification in order to provide an improved methods for moving a GUI object, to enable a user to interact with and control software applications in conjunction with a pointing device and a GUI display, as described in Undasan (paragraph 0005).
With respect to claims 12, 32, and 50, Osborn in view of Chae, further in view of Undasan teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the fourth type of gesture does not include multiple instance of the first type of gesture and the third type of gesture does not include multiple instances of the second type of gesture (e.g. paragraph 0620, finger/hand gesture combined with force, user clenching first, etc., to open menu/dialog; paragraph 1468, finger pinch pose used to click selected item; paragraph 1381, first pose to display menu; paragraph 1474, first and second body states may be counterparts of one another; body states including wrist extension, flexion, rotation, radial deviation, ulnar deviation, etc.; i.e. where a fist clenching or other user input used for opening a menu, analogous to a fourth type of gesture does not include multiple instances of a first body state such as wrist extension, flexion, rotation, deviation, etc., analogous to a first type of gesture, and where a finger pinch pose for clicking a selected item, analogous to a third type of gesture, does not include multiple instances of a second body state such as wrist extension, flexion, rotation, deviation, etc., counterpart to a first body state, analogous to a second type of gesture ).
With respect to claims 14, 34, and 52, Osborn in view of Chae, further in view of Undasan teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the menu includes one or more selectable options, wherein the one or more selectable options includes a first selectable user interface object for changing operations that one or more hand gestures can cause to the computer system to perform, wherein selection of the first selectable user interface object causes the computer system to display a plurality of settings, and wherein each setting controls an operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system (e.g. paragraph 1483, Fig. 36S, multi-state user interface configurable/programmable by user; user able to specify mappings between body states and responsive actions; such as mapping body part sates of ring/middle/pinky/index finger pinches, first pose, and open hand pose to scroll up/down, activate/deactivate link, increment/decrement selector actions; paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H; paragraph 1486, Fig. 36I, user modifying mappings between body states and actions; configuring web navigation settings by selecting setting options shown in drop-down menu of web browser; user clicking and selecting “scroll:options” to configure type of pose that initiates scrolling control or feature; paragraph 1487, Fig. 36J, link activate setting used to highlight links included in web page; user clicking on “linksActivate:options” to configure type of pose and predetermined pose duration that initiates the activation and rendering of link displayed in webpage; paragraph 1488, Fig. 36K, user clicking on “click:options” to configure type of pose and predetermined duration that initiates a click action in an application; i.e. selection of the “Control Options” dropdown object of Fig. 36G causes the plurality of settings to be shown in Fig. 36H, where these settings allow control of which gestures/poses are mapped to which operations).
With respect to claims 15, 35, and 53, Osborn in view of Chae, further in view of Undasan teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein:
the menu includes one or more selectable options (e.g. paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H);
the one or more selectable options includes a second selectable option for transitioning the computer system to a fourth operation mode and a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode (e.g. paragraph 1485, Fig. 36G, multi-state user interface facilitates enabling and disabling body state control using “enabled” button; drop down menu control from which user is able to select certain display settings and map body states to responsive actions; see also “reset settings” button in Fig. 36G;  i.e. where an “enabled” button can be selected to transition the system to a state/mode in which the body state controls are enabled, and to a state/mode in which body state controls are disabled, analogous to at least a second selectable option for transitioning the computer system to a fourth operation mode (i.e. a body state controls disabled mode); further where the user selects the drop down menu and modifies the mappings of body states and responsive actions, this causes the system to be transitioned from a first mode (i.e. a mode in which a current set of body state/action mappings are utilized) to a different/fifth mode (i.e. a mode in which a modified/custom set of body state/action mappings are utilized), analogous to a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode; in addition, where body state/action mappings are already customized/modified, and the user selects the option to “reset settings” this would cause the system to be transitioned from a first mode (i.e. a mode in which a current set of modified/customized body state/action mappings are utilized) to a different/fifth mode (i.e. a mode in which a default/reset set of body state/action mappings are utilized), analogous to a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode);
selection of the second selectable option causes the computer system to transition to the fourth operation mode (e.g. paragraph 1485, Fig. 36G, enabling/disabling body state control in using “enabled” button, as discussed above); and
selection of the third selectable option causes the computer system to transition to the fifth operation mode (e.g. paragraph 1485, Fig. 36G, user selecting drop down menu control to map body states to responsive actions; user selecting to “reset settings”, as discussed above).
With respect to claims 16, 36, and 54, Osborn in view of Chae, further in view of Undasan teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the menu includes one or more selectable options, wherein the one or more selectable options include a fourth selectable option for displaying one or more additional selectable options, and wherein selection of the fourth selectable option causes the computer system to display the one or more additional selectable options that were not previously displayed before selection of the fourth selectable option (e.g. paragraph 1483, Fig. 36S, multi-state user interface configurable/programmable by user; user able to specify mappings between body states and responsive actions; such as mapping body part sates of ring/middle/pinky/index finger pinches, first pose, and open hand pose to scroll up/down, activate/deactivate link, increment/decrement selector actions; paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H; paragraph 1486, Fig. 36I, user modifying mappings between body states and actions; configuring web navigation settings by selecting setting options shown in drop-down menu of web browser; user clicking and selecting “scroll:options” to configure type of pose that initiates scrolling control or feature; paragraph 1487, Fig. 36J, link activate setting used to highlight links included in web page; user clicking on “linksActivate:options” to configure type of pose and predetermined pose duration that initiates the activation and rendering of link displayed in webpage; paragraph 1488, Fig. 36K, user clicking on “click:options” to configure type of pose and predetermined duration that initiates a click action in an application; i.e. selection of the “Control Options” dropdown object of Fig. 36G, analogous to a fourth selectable option, causes additional selectable options, not previously displayed, to be displayed, as shown in Fig. 36H).
With respect to claims 19, 39, and 57, Osborn in view of Chae, further in view of Undasan teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
after displaying the user interface that includes the first user interface object, detecting, via at least the optical sensor, a fourth hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, fist clenching; paragraph 1463, Fig. 35O, fist pose; i.e. while/after the menu including the first user interface object is being/has been previously displayed, or at some later time, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the fourth hand gesture:
in accordance with a determination that the fourth hand gesture is a fifth type of gesture that is different from the first type of gesture and the second type of gesture, transitioning the computer system from an inactive state to an active state (e.g. paragraph 0509, mapping neuromuscular signals to control signals to control device based on type of gesture; paragraph 0618, determining the gesture being performed; paragraph 0620, fist clenching by user results in wake signal being sent to the system; paragraph 0703, first muscular activation state detected used to determine that a wake up mode is to be controlled, and second muscular activation state used to second activation state used to control initialization operation of the wake up mode; paragraph 0706, wake up/initialization mode; paragraph 1380, mappings causing device to perform action in response to detection of specific body part states; paragraph 1463, Fig. 35O, fist pose mapped to command to display, enter, and activate radial menu; i.e. when the system determines, based on a mapping body part states/gestures to actions, that the user is performing a fist pose hand gesture or other suitable gesture, the system may perform a corresponding action of activating a menu, or waking the system from an inactive/shut down state, analogous to transitioning the system from an inactive state to an active state).
With respect to claims 20, 40, and 58, Osborn in view of Chae, further in view of Undasan teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying, via the display generation component, the indication that the second user interface object is selected, detecting, via at least the optical sensor, a fifth hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraphs 1473-1475, first body state mapped to selection of next indicator in given sequence; first and second body states may represent counterparts; first body state mapped to selection of next visual indicator in sequence, second body state mapped to selection of previous visual indicator in sequence; i.e. while the interface/menu including the first, second, and third objects is being displayed, including after the user has performed a gesture causing a second user interface object to be selected and while the second user interface object is displayed with an indication that it is selected, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the fifth hand gesture:
in accordance with a determination that the fifth hand gesture is the first type of gesture, displaying, via the display generation component, an indication that a fourth first user interface object is selected (e.g. paragraphs 1473-1475, first body state mapped to selection of next indicator in given sequence; in response to identification of first body state, deselecting indicator 351111 and selecting indicator 351113, providing visual feedback; Figs. 36B-E; i.e., where the user has already performed the first type of gesture once, causing a second user interface object to be selected as shown in Fig. 36C, the user may perform another instance of the first type of gesture, which is mapped to selection of a next indicator, such that after this second instance of the first type of gesture is performed, a next/subsequent indicator (with respect to currently selected indicator 351113, such as the next indicator below it) will be selected, and indicated as selected via visual feedback, analogous to displaying an indication that a fourth UI object is selected in accordance with a determination that the fifth hand gesture is the first type of gesture (Examiner notes that a similar rationale would be applicable in the interfaces of Figs. 36D-E, such that if a user performs a first instance of the first type of gesture to cause item “E” to be selected as the next in sequence, and then performs a second instance of the first type of gesture, this will cause the item “F” to be selected as the next item in the sequence, and indicated as such)); and
in accordance a determination that the fifth hand gesture is the second type of gesture, displaying, via the display generation component, the indication that the first user interface object is selected (paragraphs 1473-1475, first and second body states may represent counterparts; second body state mapped to selection of previous visual indicator in sequence; i.e., where the user has already performed the first type of gesture, causing a second user interface object to be selected as shown in Fig. 36C, the user may perform the second type of gesture, which is mapped to selection of a previous indicator, such that after this second type of gesture is performed, a previous indicator (with respect to currently selected indicator 351113, such as the previous indicator above it) will be selected, and indicated as selected via visual feedback (i.e. the selection will be reverted back to the state shown in Fig. 36B), analogous to displaying an indication that the first UI object is selected in accordance with a determination that the fifth hand gesture is the second type of gesture (Examiner notes that a similar rationale would be applicable in the interfaces of Figs. 36D-E, such that if a user performs a first instance of the first type of gesture to cause item “E” to be selected as the next in sequence, and then performs the second type of gesture, this will cause the item “D” to be selected as the previous item in the sequence, and indicated as such, similar to what is shown in Fig. 36D)).
With respect to claims 21, 41, and 59, Osborn in view of Chae, further in view of Undasan teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying the user interface that includes the first user interface object, detecting a request to transition the computer system from the first operation mode to a fourth operation mode (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 1470, Fig. 35 R, open hand pose, mapped to command to deactivate the menu; i.e. while the menu is being displayed, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected, including a gesture to request transitioning from a first operation mode (such as one in which a menu is activated) to a fourth operation mode (such as one in which the menu is deactivated/disabled));
in response to detecting the request to transition the computer system from the first operation mode to the fourth operation mode, transitioning the computer system from the first operation mode to the fourth operation mode (e.g. paragraph 1461, disabled radial menu; paragraph 1471, deactivating radial menu; i.e. in response to detecting the user’s open hand gesture, the radial menu is disabled/deactivated, transitioning from an operation mode in which the radial menu is activated (analogous to a first operation mode) to an operation mode in which the radial menu is deactivated (analogous to a fourth operation mode));
while the computer system is in the fourth operation mode and while displaying the user interface that includes the first user interface object, the second user interface object, and the third user interface object, and the indication that the first user interface object is selected (e.g. paragraph 1471, state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; i.e. even after the radial menu is deactivated, all user interface objects, and selection indicator (indicating which object is selected) may continue to be displayed on the display), detecting, via at least the optical sensor, a sixth hand gesture (e.g. continuously receiving neuromuscular/musculoskeletal signals and continuously performing corresponding gesture recognition as cited above); and
in response detecting, via least the optical sensor, the sixth hand gesture:
in accordance with a determination that the sixth hand gesture is the first type of gesture, continuing to display the indication that the first user interface object is selected (e.g. paragraph 1411, user rotating arm without making fist pose has no effect on the radial menu; paragraph 1461, disable radial menu is a mode in which the user is not able to interact with the radial menu; paragraph 1470, deactivated radial menu; paragraph 1471 state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; radial menu will not recognize/respond to subsequent selection (wrist gesture) or click command until receiving a subsequent activation command); i.e. after the radial menu is deactivated, all user interface objects including the selection indicator are continuously displayed and, while in the deactivated state, the user’s hand gestures, such as a first type of gesture to select a next object, will have no effect, and will not be recognized/responded to, such that the menu will continue to be displayed including the selection indicator (indicating a current action indicator as selected) in an unchanged state); and
in accordance with a determination that the sixth hand gesture is the second type of gesture, continuing to display the indication that the first user interface object is selected (e.g. paragraph 1411, user rotating arm without making fist pose has not effect on the radial menu; paragraph 1461, disable radial menu is a mode in which the user is not able to interact with the radial menu; paragraph 1470, deactivated radial menu; paragraph 1471 state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; radial menu will not recognize/respond to subsequent selection (wrist gesture) or click command until receiving a subsequent activation command); i.e. after the radial menu is deactivated, all user interface objects including the selection indicator are continuously displayed and, while in the deactivated state, the user’s hand gestures, such as a second type of gesture to select a previous object, will have no effect, and will not be recognized/responded to, such that the menu will continue to be displayed including the selection indicator (indicating a current action indicator as selected) in an unchanged state).
Claims 13, 18, 33, 38, 51, and 56 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Chae, further in view of Undasan, further in view of Kim et al. (US 20160029899 A1).
With respect to claims 18, 38, and 56, Osborn in view of Chae, further in view of Undasan teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying the menu, detecting, via at least the optical sensor, a third hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 1470, Fig. 35 R, open hand pose; i.e. while a menu is being displayed, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that the third hand gesture is a type of gesture, ceasing to display the menu (e.g. paragraph 0509, mapping neuromuscular signals to control signals to control device based on type of gesture; paragraph 0618, determining the gesture being performed; paragraph 1381, open hand pose may cause the wearable device to direct the computing device to close the menu; paragraph 1380, mappings causing device to perform action in response to detection of specific body part states; paragraph 1470, open hand pose mapped to command to hide, exit, or deactivate menu; following identification of open hand pose, some or all of menu may be obscured or hidden from view within the GUI; i.e. when the system determines, based on a mapping body part states/gestures to actions, that the user is performing an open hand gesture or other suitable gesture, the system may perform a corresponding action of ceasing to display a currently displayed menu).
Osborn does not explicitly disclose that the type of gesture is a fourth type of gesture.  However, Kim teaches that the type of gesture is a fourth type of gesture (e.g. paragraph 0256, Figs. 17A-B, while message check screen outputted through display, if command for switching to lock screen is received, stopping output of message check screen and outputting/switching to lock screen; command may include first input (which is shown as a clenched fist gesture in Figs. 17A-B); i.e. where a user performing a fist gesture to initiate/switch to a lock screen display would cause the system to cease displaying all currently displayed content, including menus).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Chae, Undasan, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Chae (directed to an ultra compact mobile terminal, such as a watch, having a scroll key/wheel input device), and Undasan (directed to GUI navigation, including via a GUI item/element which emulates hardware inputs such as button and scroll wheel inputs), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability, while displaying user interface content (i.e. such as a GUI including currently displayed menus, as taught by Osborn) to receive a hand gesture to transition to a lock screen state/mode, such as in response to a clenched fist hand gesture, and in response, cease displaying currently displayed content (such as the menu of Osborn) and display a lock screen (i.e. as taught by Kim).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claims 13, 33, and 51, Osborn in view of Chae, further in view of Undasan teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
detecting, via at least the optical sensor, a third hand gesture; and in response to detecting, via at least the optical sensor, the third hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected):
in accordance with a determination that the third gesture is the fourth type of gesture, performing an action (e.g. paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user); and
in accordance with a determination that a notification was received and in accordance with a determination that the third hand gesture is a type of gesture, performing an action related to the notification (e.g. paragraph 1649 and corresponding table, using swipe to engage/ignore notification; paragraphs 1651-1653, flick gesture may be used as dismissal mechanism for notifications; notifying user that next meeting is coming up, flick dismissing notification outright, index pinch tap engaging timer function, rolling wrist to select timer duration, etc.).
Osborn does not explicitly disclose that the determinations (for performing the action related to the notification) are that the notification was received within a threshold period of time and that the third hand gesture is the fourth type of gesture.  However, Kim teaches in accordance with a determination that a notification was received within a threshold period of time and in accordance with a determination that the third hand gesture is the fourth type of gesture, performing an action related to the notification (e.g. paragraph 0014, determining level of folding fingers of the user as being in various states (i.e. gestures) based on measured blood flow rate; paragraph 0186, Fig. 5A, user determined in sleep state, generated notifications controlled to not be outputted to user; when user wakes up, user can be informed of saved notifications; paragraph 0187, after detecting user has got out of sleep state, if prescribed command (such as making a fist twice) is received, outputting weather information through the display; paragraph 0190, if user wakes from sleep state, outputting notification screen 501 for cumulative notifications (notification list of plurality of notifications) to user, if prescribed command is received, such as input of making a fist twice, as shown in Figs. 5A-C; i.e. where a notification is received within a time period during which the user is sleeping (analogous to a threshold period of time) and then an input of making a fist gesture is received, an action related to the notification is performed, such as displaying the notification; paragraph 0227, Fig. 11A, call signal currently received, displaying call screen 1101 indicating call signal is currently being received; paragraph 0229, Figs. 11A-C, when call signal currently received, if first input maintained over prescribed time, making response to call signal, and outputting call connected screen 1103; i.e. as shown in Figs. 11A-C, if, during the time period that a call signal is being currently received and a corresponding notification screen is displayed (e.g., during the time period beginning when the call signal is first received and ending when the signal is no longer received, analogous to a threshold period of time within which a notification is received), if a user makes an input such as holding a fist for a time period, the call is answered, analogous to performing an action (answering the call) related to the notification (the screen indicating the call is being received)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Chae, Undasan, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Chae (directed to an ultra compact mobile terminal, such as a watch, having a scroll key/wheel input device), and Undasan (directed to GUI navigation, including via a GUI item/element which emulates hardware inputs such as button and scroll wheel inputs), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to detect the fourth type of gesture (i.e. of Osborn, such as a fist gesture for causing a menu to be displayed) and, based on determining that a received gesture is the fourth type of gesture/fist gesture and further determining that a notification has been received within a threshold time period (such as during a time period in which the user was sleeping, or during a time period in which a call signal is currently received and a corresponding notification is displayed), performing a related action, such as displaying the notification, or answering a corresponding phone call, etc., as taught by Kim.  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
Claims 2, 24, and 42 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Chae, further in view of Undasan, further in view of Chung et al. (US 20220326779 A1).
With respect to claims 2, 24, and 42, Osborn in view of Chae, further in view of Undasan teaches all of the limitations of claims 1, 22, and 23 as previously discussed.  Although Osborn teaches that auxiliary sensors which may contribute to gesture detection may include heart rate sensors, PPG sensors, and cameras used to detect changes in blood flow within the user (e.g. paragraph 0281, sensors including cardio sensor to measure user’s pulse, heart rate; paragraph 0286, auxiliary device/sensor including heart-rate monitor; paragraph 0310, nonvisible light used to capture image of heat distribution, providing indication of blood flow, used to infer condition of user such as forces being exerted by certain fingers; paragraph 0729, PPG sensors detecting vascular changes), Osborn does not explicitly disclose wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor.
However, Chung teaches wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor (e.g. paragraph 0093, sensor module sensing information for movement of electronic device, and may include photoplethysmogram (PPG) sensor; paragraph 0098, recognizing OCO and shake gestures, gestures of turning hand, moving hand with palm open, lifting arm, moving thumb, moving up/down hand, etc.; gesture may be recognized based on user’s biometric signal (e.g. a signal indicating relaxation/contraction of blood vessels or a heartbeat signal); paragraph 0104, gesture application invoking sensor API to activate sensors, such as PPG sensor; gesture application identifying information for gestures; paragraph 0136, OCO gesture, PPG sensor detecting change due to relaxation and contraction of blood vessels when user opens and closes her hand; ).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Chae, Undasan, and Chung in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Chae (directed to an ultra compact mobile terminal, such as a watch, having a scroll key/wheel input device), and Undasan (directed to GUI navigation, including via a GUI item/element which emulates hardware inputs such as button and scroll wheel inputs), to incorporate the teachings of Chung (directed to recognizing gestures) to include the capability to detect the hand gesture based on heart rate data detected via the optical sensor, such as by detecting a variety of gestures using a PPG sensor, based on a user biometric signal indicating relaxation/contraction of blood vessels or heartbeat (as taught by Chung).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability to recognize a gesture when there is a gesture recognition request from a specific application, eliminating the need for sensors to stay on and the gesture recognition process to keep on running, reducing power consumption and resource waste, and in order to make it possible to immediately recognize relatively small gestures, as described in Chung (paragraphs 0014-0015).
Claims 17, 37, and 55 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Chae, further in view of Undasan, further in view of Saurabh et al. (US 20180181261 A1).
With respect to claims 17, 37, and 55, Osborn in view of Chae, further in view of Undasan teaches all of the limitations of claims 1, 22, and 23 as previously discussed.  Osborn does not explicitly disclose wherein the menu is displayed:
in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location; and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location.
However, Saurabh teaches wherein the menu is displayed:
in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location (e.g. abstract, positioning popup interface within display frame in relation to corresponding target element; relative positioning of popup interface to associated target element dynamically selected and determined based on location of corresponding target element within quadrants/subquandrants of display frame; paragraph 0001, presenting additional information/menu in the form of a popup interface; paragraph 0004, rendering popup interface at position determined to be best fit position with regard to target element’s position; paragraph 0005, determining dynamically made where to display the popup interface by determining quadrant the target element is actually displayed within; paragraph 0027, pop interface may refer to a context menu; paragraph 0028, target element linked to popup interface; target element may be word or icon, and popup interface may be a selection menu related to the word or icon; paragraph 0077, Fig. 8, target element 802 residing within subquadrant 5/quadrant 1, resulting popup interface 804 displayed at bottom position relative to target element; i.e. where a respective user interface element (target element) is displayed at a position within a first quadrant (analogous to a first location), the corresponding popup menu is also displayed at a position within the same quadrant/location); and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location (e.g. abstract, positioning popup interface within display frame in relation to corresponding target element; relative positioning of popup interface to associated target element dynamically selected and determined based on location of corresponding target element within quadrants/subquandrants of display frame; location of popup interface is dynamically adjustable as opposed to being positioned based on a predetermined location; paragraph 0004, rendering popup interface at position determined to be best fit position with regard to target element’s position, event when that target element changes position or otherwise moves; paragraph 0005, determining dynamically made where to display the popup interface by determining quadrant the target element is actually displayed within; as the target element moves within the user interface (due to scrolling etc.) the relative position of the popup interface is dynamically adjusted so that the popup interface is continuously displayed in a position identified as best fit relative to target element; paragraph 0083, Fig. 10, target element 1002 displayed spanning quadrant1 and quadrant2, resulting popup interface 1004 displayed in bottom position relative to target element 1002; i.e., where a respective user interface element is displayed at a different location such as a location spanning multiple quadrants (analogous to not being displayed at a first location), the corresponding popup menu may also be moved/repositioned on the display so that it is also displayed at a second location which is different from the first location).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Chae, Undasan, and Saurabh in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects, including displaying associated menus), Chae (directed to an ultra compact mobile terminal, such as a watch, having a scroll key/wheel input device), and Undasan (directed to GUI navigation, including via a GUI item/element which emulates hardware inputs such as button and scroll wheel inputs), to incorporate the teachings of Saurabh (directed to a positioning mechanism for bubble as a custom tooltip, such as positioning popup interfaces/menus in relation to a corresponding target element) to include the capability to dynamically display the menu (i.e. of Osborn) in a position relative to the position of a corresponding UI object such that when the UI object is displayed at a first location, the menu is also displayed at the first location and when the UI object is not displayed at the first location, such as because it is moved to a different location, the menu is displayed at a second location different from the first location, such as at the different location that the UI object has been moved to (as taught by Saurabh).  One of ordinary skill would have been motivated to perform such a modification in order to solve disadvantages found in existing popup interfaces by providing the capability to dynamically render a popup interface at a position that is determined to be a best fit position with regard to a target elements position, event when that target element changes position or otherwise moves in some manner, as described in Saurabh (paragraphs 0002-0004).
Claims 4, 11, 26, 31, 44, and 49 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Chae, further in view of Undasan, further in view of Li (US 20160328108 A1) , further in view of Kin (US 10261595 B1), further in view of Pantel (US 20140320434 A1).
With respect to claims 4, 26, and 44, Osborn in view of Chae, further in view of Undasan teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches
wherein before detecting the hand gesture, the computer system is in a first operation mode (e.g. paragraph 1463, Fig. 35O, fist pose mapped to command to display, enter, and activate radial menu; i.e. where in the first state, at the time the radial menu is displayed, the user has provided a previous input causing the radial menu to be activated, and therefore in an operation mode in which the menu is activated, analogous to a first operation mode),
wherein performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode (e.g. paragraph 1468, Fig. 35Q, illustrating finger pinch pose 351006, which may be mapped to command/instruction to click visual indicator in the radial menu, analogous to mouse button click or keypress; alternatively combination of finger pinch pose and certain degree of force mapped to click command; paragraph 1469, Fig. 35 U, clicking/engaging with or activating function of the currently selected action indicator; clicking action indicator in response to identified click gesture; upon performing the click action, causing visual indicator to appear to depress and/or release, similar to a physical button; i.e. clicking the currently selected action indicator causes the corresponding function to be activated, and therefore causes the system to transition to an operation mode in which the corresponding function has been activated, analogous to a second operation mode)
detecting, via at least the optical sensor, a second hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that the second hand gesture is one type of gesture, toggling a first automatic scrolling operation between an active state and an inactive state (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling up a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll up a GUI; paragraph 1484, ring finger pinch mapped to scroll up so that when user makes ring finger pinch pose, the page/browser displayed scrolls up); and
in accordance with a determination that the second hand gesture is another type of gesture, reversing a first direction of the first automatic scrolling operation  (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling down a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll down a GUI; paragraph 1484, pinky finger pinch mapped to scroll down so that when user makes pinky finger pinch pose, the page/browser displayed scrolls down).
Although Osborn further teaches that user hand gestures may be mapped to different commands in different states/modes, and further teaches that a wrist roll gesture may be mapped to a scroll command (e.g. paragraph 0648, switching between modes; paragraph 0666-0667, selection of mode, such as from list or using gesture; paragraph 1380, transitioning between different mappings of body part states and responsive actions; paragraph 1408-1410, activating first mapping and/or deactivating second mapping between possible body part states and responsive actions; user switching between mappings; transition/switch between mappings; paragraph 1486, user modifying mappings between body states and actions, user configuring/mapping scroll controls to different poses; table accompanying paragraph 1649, indicating that an interaction with a list feature may include a “scroll through” interaction with a “wrist roll” control scheme), Osborn does not explicitly disclose:
wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture,
while the computer system is in the second operation mode, detecting the second hand gesture;
in response to detecting the second hand gesture:
in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation;
in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation; and
in accordance with a determination that the second hand gesture is a fourth type of gesture different from the first type of gesture, the second type of gesture, and the third type of gesture, ending the automatic scrolling operation.
However, Li teaches:
wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture (e.g. paragraph 0037-0038, utilizing auto-scroll method such that contents are presented in multiple steps/views within given time period; user starting content show with auto scroll where contents are presented and scrolled automatically and continuously; user selecting scroll options, etc.; paragraph 0046, auto scroll replaces contents on display at given rate continuously and without user involvement),
while the computer system is in the second operation mode, detecting the second hand gesture (e.g. paragraph 0046, describing user inputs/gestures provided while content is automatically scrolled, such as a finger hover/touch input gesture to pause scrolling; paragraph 0047, describing other types of gesture which may be used as command input, including user holding hand upright with palm facing a horizontal direction, user hand movements to make selections, etc.; paragraph 0053, describing user methods for stopping a scroll process temporarily, such as selecting on screen buttons, touch inputs, and use of gestures via gesture recognition system, such as by moving hand or fingertip in a given direction, where the user’s hand may be in any position, vertical, horizontal, tilted, using both a fist or spread hand for hand gestures, and using one or multiple fingertips for finger gestures);
in response to detecting the second hand gesture:
in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation (e.g. paragraph 0046-0047, user inputs to pause scrolling, including hand gestures; paragraph 0053, user input to stop scroll process temporarily, including use of hand gestures, including hand gestures in a given direction, using a tilted hand, etc.; i.e. the user may provide a hand gesture to pause scrolling, including a directional/tilted hand gesture, analogous to pausing an automatic scrolling operation in accordance with a determination that a hand gesture is a first type of hand gesture);
in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation (e.g. paragraph 0044-0045, scroll direction may be reversed using gesture, such as swipe on touch screen; via buttons, etc.; paragraph 0047, describing use of other gestures as command input, including directional movements to select an option; paragraph 0053, user turning page forward or backward during scroll session, including use of hand gestures, including hand gestures in a given direction, using a tilted hand, etc.; i.e. the user may provide a hand gesture to reverse scrolling direction, or to turn pages in any direction (including an opposite direction) during automatic scrolling, including a directional/tilted hand gesture, analogous to reversing a direction of scrolling in accordance with a determination that a hand gesture is a second type of gesture); and
in accordance with a determination that the second hand gesture is the fourth type of gesture, ending the automatic scrolling operation (e.g. paragraph 0043, selection of button to close content window and end scroll session; selection of no button to stop scroll process; scroll session controlled by device movement, etc.; paragraph 0047, describing various user gestures which may be used as command input, including hand movements to select a no icon/option; paragraph 0053, describing various hand gestures which may be used as command inputs, including hand gestures with a fist; paragraph 0054, option to quit auto scroll prepared; tapping/clicking/selecting quit button; i.e. the user may provide a different hand gesture, such as a closed fist or other hand gesture, in order to provide a command to close/exit the auto scroll operation mode).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Chae, Undasan, and Li in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Chae (directed to an ultra compact mobile terminal, such as a watch, having a scroll key/wheel input device), and Undasan (directed to GUI navigation, including via a GUI item/element which emulates hardware inputs such as button and scroll wheel inputs), to incorporate the teachings of Li (directed to presenting contents using auto scroll) to include the capability to place the system in second mode in response to a third type of user input (i.e. such as a mode of executing a corresponding functionality following a user’s selection activating a displayed item, as taught by Osborn, and a mode of initiating automatic and continuous content scrolling functionality following a user’s selection of a corresponding user interface component/option, as taught by Li), where the second mode includes an automatic scrolling function and, while in this automatic scrolling mode, receive an input/gesture to pause the automatic scrolling (including a directional/tilted hand gesture, analogous to a first type of gesture, as taught by Li, similar to the directional wrist rotation gesture as taught by Osborn), receive an input/gesture to reverse the direction of the automatic scrolling (including a directional/tilted hand gesture in a direction corresponding to a reverse direction, as taught by Li, similar to the directional wrist rotation gesture as taught by Osborn), and receive an input/gesture to exit/quit the auto scroll operation/mode (including a gesture using a hand in a fist and/or selecting a corresponding option, as taught by Li, similar to a fist gesture or other gesture as taught by Osborn).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability for the user to relax and watch contents effortlessly, as described in Li (paragraph 0037).
Although Li further teaches user inputs/gestures for changing a speed of the automatic scrolling operation (e.g. paragraphs 0037-0038, user changing scroll rate as needed/setting scroll rate, including using an adjustable button/column; paragraph 0054, user option to adjust scroll rate; paragraph 0071, user selecting option, including scroll rate, to adjust), Osborn and Li do not explicitly disclose in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation.  However, Kin teaches in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation (e.g. col. 14 lines 3-20, detecting pinch scroll gesture, if detecting pinch scroll gesture with thumb and two fingers, speeding up rate of scroll action compared to action with just one finger; after detecting pinch scroll gesture, detecting change in pose of user’s hand while in pinched position, and increasing the rate of scrolling).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Chae, Undasan, Li, and Kin in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Chae (directed to an ultra compact mobile terminal, such as a watch, having a scroll key/wheel input device), Undasan (directed to GUI navigation, including via a GUI item/element which emulates hardware inputs such as button and scroll wheel inputs), and Li (directed to presenting contents using auto scroll), to incorporate the teachings of Kin (directed to high resolution tracking and response to hand gestures through three dimensions) to include the capability, while performing automatic scrolling (as taught by Li, incorporated into the system of Osborn), to detect a pinch gesture, or a change in an existing pinch gesture (as taught by Kin, analogous to a third type of gesture as taught by Osborn, such as for selecting a displayed option), and in response, change the speed/rate of the scrolling operation (as taught by Kin, such as by pinching with an increased number of fingers, or changing a hand pose while pinching).  One of ordinary skill would have been motivated to perform such a modification in order to overcome limitations of conventional hand tracking systems, which are unable to accurately track positions of user fingers and thumbs and precise movements of user digits, causing poor representation of the user’s hands, by providing the capability to detect various gestures performed by the user’s hands and cause a change in display instructions in response to the detected gesture as described in Kin (col. 1 lines 20-43).
Assuming arguendo that Osborn, Li, and Kin do not explicitly disclose in accordance with a determination that the second hand gesture is the first type of hand gesture, pausing the automatic scrolling operation, Pantel teaches in accordance with a determination that the second hand gesture is the first type of hand gesture, pausing the automatic scrolling operation (e.g. paragraph 0204, Fig. 17, illustrating forearm-spin gesture, in which smartwatch is rotated about the x axis by twisting the forearm; paragraph 0220, forearm spin gesture can be executed in two directions, and direction can be assigned to different functions/tasks, including, at paragraph 0222, locking of the scroll capability of a page/card/tab; i.e. the user may temporarily lock/pause a scrolling operation/capability using wrist/forearm rotation in an assigned direction).  In addition, assuming arguendo that Osborn, Li, and Kin do not explicitly disclose in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation, Pantel teaches in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation (e.g. paragraph 0175, user performing drag away gesture, such as performing conventional drag gesture to scroll through a page, where user can also change direction of drag movement/page scrolling; paragraph 0204, Fig. 17, forearm spin gesture is variant of drag along/away gesture, where smart watch is rotated around axis by twisting the forearm while index finger is held in place; paragraph 0209, in forearm spin gesture, user touches screen with finger and starts to twist forearm while maintaining finger steady, resulting in dragging of the finger in manner comparable to drag along/away gestures; paragraph 0214-0215, triggering drag completed/flick based on direction/distance/speed; paragraph 0220, forearm spin gesture executed in two directions, direction of gesture assigned to different functions; i.e. the user may also utilize the forearm spin gesture in a given direction to execute functions associated with drag inputs, including causing scrolling to be performed in a given direction, such as a reversed/changed direction).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Chae, Undasan, Li, Kin, and Pantel in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Chae (directed to an ultra compact mobile terminal, such as a watch, having a scroll key/wheel input device), Undasan (directed to GUI navigation, including via a GUI item/element which emulates hardware inputs such as button and scroll wheel inputs), Li (directed to presenting contents using auto scroll), and Kin (directed to high resolution tracking and response to hand gestures through three dimensions), to incorporate the teachings of Pantel (directed to gesture control) to include the capability, while performing automatic scrolling (as taught by Li, incorporated into the system of Osborn), to detect a forearm spin gesture (as taught by Pantel, similar to a wrist rotation gesture as taught by Osborn) in a given direction and either lock/pause a scrolling operation (as taught by Pantel, similar to pausing scrolling according to a hand gesture as taught by Li), or reverse the direction of scrolling (as taught by Pantel, similar to controlling scrolling direction as taught by Osborn and Li).  One of ordinary skill would have been motivated to perform such a modification in order to provide novel gestures in addition to familiar touchscreen gestures, allowing access of several features or functions of an operating system or application using only a single button, icon, or control element, or global system functions using novel gestures at any location regardless of displayed content as described in Pantel (paragraph 0014).
With respect to claims 11, 31, and 49, Osborn in view of Chae, further in view of Undasan, further in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn and Li further teach wherein the fourth type of gesture is a type of gesture that is multiple instances of the third type of gesture (e.g. Osborn paragraph 1271 series of pinches to change mode; paragraph 1381, finger pinch causing selection of item; paragraph 1449-1450, pinch mapped to click/activation/selection action; paragraph 1468, pinch pose mapped to command/instruction to click/select item; paragraph 1538, selection occurring upon detection of repeated gesture such as multiple taps of finger; paragraph 1622, double tapping inputs; paragraph 1628, double click/double tapping finger; paragraph 1646, gestures may be arbitrarily defined; gestures can include two or multiple finger pinches, sequences; paragraph 1650, one or more pinch taps; Li paragraph 0043, selection of button to close content window and end scroll session; selection of no button to stop scroll process; scroll session controlled by device movement, etc.; paragraph 0046, double tap gesture to activate displayed item, such as link, as opposed to single tap for pausing scrolling; paragraph 0047, describing various user gestures which may be used as command input, including hand movements to select a no icon/option; paragraph 0053, describing various hand gestures which may be used as command inputs, including multi-fingertip gesture; paragraph 0054, option to quit auto scroll prepared; tapping/clicking/selecting quit button; i.e. where the third type of gesture may be a pinch gesture (as cited above with respect to Kin’s teachings of a pinch gesture for controlling scroll speed in claim 4 and Osborn’s teachings of a pinch gesture to perform a “click” selection, such as to select a displayed item), and where Li teaches that a gesture (such as a click, tap, or other gesture including a multifinger gesture) may be performed to select a displayed option to end/exit a scroll operation, Osborn’s further teachings of utilizing multiple/repeated pinch/click gestures to select/activate an item may be used for selection of the option of Li to end the scroll operation).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Chae, Undasan, Li, Kin, and Pantel in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Chae (directed to an ultra compact mobile terminal, such as a watch, having a scroll key/wheel input device), Undasan (directed to GUI navigation, including via a GUI item/element which emulates hardware inputs such as button and scroll wheel inputs), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Li (directed to presenting contents using auto scroll) to include the capability to receive a hand gesture to select a displayed item on the screen for ending/exiting an autoscrolling operation (as taught by Li) where this gesture may include multiple instances of a gesture, such as multiple pinch, click, or “pinch click” gestures in order to successfully select the item (as taught by Osborn, where Li further teaches utilizing a double tap operation for item selection) for ending autoscrolling (as taught by Li).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability for the user to relax and watch contents effortlessly, as described in Li (paragraph 0037).
Claims 3, 25, and 43 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Chae, further in view of Undasan, further in view of Li , further in view of Kin, further in view of Pantel, further in view of Friedman (US 20110090407 A1).
With respect to claims 3, 25, and 43, Osborn in view of Chae, further in view of Undasan, further in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture (e.g. paragraph 1473-1474, Figs. 36B-C, first body state mapped to selection of next indicator in given sequence; in response to identification of first body state, deselecting indicator 35111 and selecting indicator 351113; first and second body states may represent counterparts of one another; 1475, Figs. 36D-E, first body state mapped to selection of next visual indicator in the sequence; i.e. because first body state/gesture causes selection to move to a next object in sequence, receipt of multiple instances of this gesture may also cause a different/third UI object to be selected; for example, where multiple instances/two instances of the first type of gesture are received, the object selected as indicated may be first moved from the first UI object to the second/next UI object in the sequence, and then moved again from the second/next UI object to a third/subsequent UI object in the sequence).
Assuming arguendo that Osborn, Chae, Undasan, Li, Kin, and Pantel do not explicitly disclose wherein the second type of gesture that is multiple instances of the first type of gesture (i.e. with respect to the second mode, where this gesture is used to reverse a direction of scrolling), Friedman teaches wherein the second type of gesture that is multiple instances of the first type of gesture (e.g. paragraphs 0046-0047, describing Fig. 5, showing table 500 with gesture motions and respective commands, including rows 504 and 506, with gesture commands having two short down movements, or two short up movements, to scroll down or up a given number of rows, and rows 510 and 512, used to scroll time forwards, or reverse/scroll time backwards within menu (i.e. scroll left or right in a timeline menu); i.e. where a gesture to scroll in a given direction, such as in a reverse direction, may include multiple instances of a given gesture, such as doing the gesture twice with two short movements in order to scroll in a direction, including a direction opposite of a current direction).
To clarify, as cited above, Osborn teaches that in a first mode, a first type of gesture, such as wrist extension/flexion, rotation, radial/ulnar deviation, etc. may be used to change selection to a second user interface object, and Li and Pantel both teach that in a second mode, this same first type of gesture (i.e. a directional/tilted hand gesture) may be used to pause an automatic scrolling operation.  Osborn and Kim both teach that in the first mode, a second type of gesture, which is multiple instances of the first type of gesture, may be used in order to change a selection to a third user interface object, and Li and Pantel both teach that in the second mode, a second type of gesture (which is at least one instance of the first type of gesture) may be used to reverse a direction of the automatic scrolling operation.  Therefore, the only thing missing from this combination is that the second type of gesture (which may include at least one instance of the first type of gesture as taught by Li and Pantel), performed in the second mode for reversing the direction of the scrolling operation, is performed using multiple instances of the gesture.  However, as discussed and cited above, Friedman teaches that the second type of gesture, which can be used to set/reverse a given scrolling direction, can be performed using multiple instances of the gesture.
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Chae, Undasan, Li, Kin, Pantel, and Friedman in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Chae (directed to an ultra compact mobile terminal, such as a watch, having a scroll key/wheel input device), Undasan (directed to GUI navigation, including via a GUI item/element which emulates hardware inputs such as button and scroll wheel inputs), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions) and Pantel (directed to gesture control), to incorporate the teachings of Friedman (directed to gesture controls) to include the capability, while performing automatic scrolling in a given direction (as taught by Li, incorporated into the system of Osborn), to detect multiple instances of a given gesture (i.e. such as the forearm spin gesture as taught by Pantel, similar to a wrist rotation gesture as taught by Osborn) in a given direction and reverse the direction of scrolling (as taught by Friedman, similar to controlling scrolling direction as taught by Osborn, Li, and Pantel; i.e. utilizing multiple instances of a gesture, as taught by Friedman, in order to change scrolling direction, such as multiple instances of the wrist tilt/rotation gesture as taught by at least Osborn and Pantel).  One of ordinary skill would have been motivated to perform such a modification in order to provide a simplified yet flexible user interface as described in Friedman (paragraph 0018).
Claims 10, 30, and 48, are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Chae, further in view of Undasan, further in view of Li, further in view of Kin, further in view of Pantel, further in view of Kim et al. (US 20160029899 A1).
With respect to claims 10, 30, and 48, Osborn in view of Chae, further in view of Undasan, further in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
in accordance with a determination that the second hand gesture is one type of gesture, displaying a notification that indicates a state of the automatic scrolling operation (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling up a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll up a GUI; paragraph 1484, ring finger pinch mapped to scroll up so that when user makes ring finger pinch pose, the page/browser displayed scrolls up; paragraph 1459, providing visual feedback of both identified pose and identified force; when action includes scrolling GUI, displaying cursor in connection with scrolling, which may be presented as a horizontal line with a bar extending above or below the line, depending on the scrolling direction; paragraph 1480, providing current state/setting of interface, current body state, and current action corresponding to current body state; paragraph 1481, graphical elements displaying current interface state, current body state, and responsive action; i.e. based on determining that the user has provided a gesture for scrolling, the system may initiate the scroll operation and may also provide a notification/indicator regarding the state of the operation, such as a cursor indicating a scrolling direction and magnitude, etc.).
Kim teaches that the one type of gesture is the first type of gesture (e.g. paragraphs 0251-254, Figs. 16A-C, outputting notification screen of received text message, receiving message check command such as folding input and outputting message check screen 1601 for displaying details of received text message; scroll action on message check screen performed by rotation of wrist; wrist rotated in first direction, scrolling to move outputted message check screen; direction for rotating wrist includes scroll direction; if scroll in bottom direction desired, rotating wrist in bottom direction; i.e. the user may provide a wrist rotation input in a first or second direction in order to perform a scroll operation in a designated direction, as shown between Fig. 16B and Fig. 16C).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Chae, Undasan, Li, Kin, Pantel, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Chae (directed to an ultra compact mobile terminal, such as a watch, having a scroll key/wheel input device), Undasan (directed to GUI navigation, including via a GUI item/element which emulates hardware inputs such as button and scroll wheel inputs), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to receive a hand gesture to initiate automatic scrolling in a direction, and a hand gesture to reverse the scrolling direction (i.e. such as using different fingers to perform pinch gestures to scroll up/down as taught by Kim), where the hand gesture may be a wrist rotation gesture in different directions corresponding to the desired scroll direction (i.e. as taught by Kim, where Osborn also teaches the use of wrist rotation for at least scrolling a wrist, and further teaches that the wrist rotation may be performed in two different directions, and further teaches, based on detecting the scroll gesture, outputting a notification/indicator of the scrolling operation state).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
Claims 8, 28, and 46 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Chae, further in view of Undasan, further in view of Li, further in view of Kin, further in view of Pantel, further in view of Platzer et al. (US 20130239046 A1).
With respect to claims 8, 28, and 46, Osborn in view of Chae, further in view of Undasan, further in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn and Li further teach wherein:
the automatic scrolling operation scrolls a sequence of a plurality of interface objects in a second direction (e.g. Osborn paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling down a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll down a GUI; paragraph 1484, pinky finger pinch mapped to scroll down so that when user makes pinky finger pinch pose, the page/browser displayed scrolls down; Li paragraph 0038, user selecting scroll options, including scroll direction; paragraph 0044-0045, scroll direction may be reversed using gesture, such as swipe on touch screen; via buttons, etc.).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Chae, Undasan, Kin, Pantel, and Li in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Chae (directed to an ultra compact mobile terminal, such as a watch, having a scroll key/wheel input device), Undasan (directed to GUI navigation, including via a GUI item/element which emulates hardware inputs such as button and scroll wheel inputs), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Li (directed to presenting contents using auto scroll) to include the capability to perform the automatic scrolling operation in a second direction (i.e. in a specified direction, as taught by Li).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability for the user to relax and watch contents effortlessly, as described in Li (paragraph 0037).
Osborn does not explicitly disclose wherein the one or more programs further includes instructions for:
while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects; and
in response to detecting the end of the sequence of the plurality of user interface objects, performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction.
However, Platzer teaches wherein the one or more programs further includes instructions for:
while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects (e.g. paragraph 0077, Figs. 5A-C, scrolling list of items to terminus of the list; paragraph 0078, terminus is edge of scrolled region; while list is being scrolled; terminus is virtual boundary associated with displayed objects; displayed objects reach or intersect with the terminus 514; i.e. the system detects the end of a list of displayed UI objects, as indicated by a terminus at the edge of the scrolled list); and
in response to detecting the end of the sequence of the plurality of user interface objects, performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction (e.g. paragraph 0077, Figs. 5A-C, reaching terminus of list, smoothly bouncing displayed items at end of list, reversing direction; paragraph 0078, as illustrated in Fig. 5B, when objects reach or intersect with the terminus 514, the movement corresponding to scrolling may stop and, as illustrated in Fig. 5C, subsequently reverse direction; i.e. scrolling list reaches and bounces at its terminus; i.e. in response to reaching/detecting the terminus while scrolling in a second direction (such as opposite of a first direction), the system automatically performs a “bounce” by scrolling the list of displayed objects in the opposite direction (which may be a third direction, which may be the same as a first direction)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Chae, Undasan, Li, Kin, Pantel, and Platzer in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects, such as scrolling menus/lists of UI objects, webpages, etc.), Chae (directed to an ultra compact mobile terminal, such as a watch, having a scroll key/wheel input device), Undasan (directed to GUI navigation, including via a GUI item/element which emulates hardware inputs such as button and scroll wheel inputs), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Platzer (directed to scrolling operations) to include the capability to, while scrolling user interface content, including a plurality of objects, in a given direction (i.e. as taught by Osborn, Kim, and Platzer, such as in a second direction which is opposite of a first scroll direction, or some other direction, where at least Jose teaches a scrolling direction may be set as left-right, right-left, up-down, or down-up), detect a terminus indicating the end of the plurality of objects and, in response automatically scroll the content including the plurality of objects in a reverse direction (as taught by Platzer, where this reverse direction may be a third direction which is different from the second direction, but potentially the same as a first direction).  One of ordinary skill would have been motivated to perform such a modification in order to provide operations desired by users, such as scrolling operations on a device, including on device having limited display size, user interface, etc. which limits the ease of use of the device, as described in Platzer (paragraphs 0006, 0008).
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain,” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting in re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (GCPA 1968)). Further, a reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill the art, including nonpreferred embodiments. Merck & Co, v. Biocraft Laboratories, 874 F.2d 804, 10 USPQ2d 1843 (Fed. Cir.), cert, denied, 493 U.S. 975 (1989). See also Upsher-Smith Labs. v. Pamlab, LLC, 412 F,3d 1319, 1323, 75 USPQ2d 1213, 1215 (Fed. Cir, 2005): Celeritas Technologies Ltd. v. Rockwell International Corp., 150 F.3d 1354, 1361, 47 USPQ2d 1516, 1522-23 (Fed. Cir. 1998).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Yoon et al. (US 20150061842 A1) teaches various gestures for controlling a mobile device such as a wrist worn device (Figs. 7-33).
Offenberg et al (US 20150248235 A1) teaches various gestures for controlling a mobile device such as a wrist worn device (Figs. 27-92), as well as for selecting among displayed items with corresponding visual indications, scrolling, etc. (Figs. 102A-D).
Hazra et al. (US 20170123487 A1) teaches various gestures for controlling a mobile device including a wrist worn device (Figs. 5, 8-15).
Lorenz et al. (US 20200401292 A1) teaches implementing virtual buttons as a virtual scroll wheel which, when activated, a response is provided that simulates a mechanical scroll wheel clicking as it turns (e.g. paragraph 0076).
Rodriguez (US 20190146219 A1) teaches a user simulating a crown twisting motion of a crown of a wristwatch in order to cause GUI scrolling (e.g. paragraph 0071, Fig. 7).
Zambetti et al (US 10001817 B2) teaches a device having a rotatable input mechanism, such as a digital crown, usable for a wide variety of user interface interactions (abstract,  Figs. 6A-F, 10A-E, 13A-K, 14-53).
Macfarlane (US 20100026626 A1) teaches a clickable scroll wheel incorporated within the housing of a portable computing device (e.g. Fig. 5B; paragraph 0081, 0099, 0107).
Fleizach et al. (US 21020306748 A1) teaches a first control and a second control are available via the menu, where selection of the first control causes the computer system to perform an operation corresponding to a press of the first hardware input mechanism and selection of the second control causes the computer system to perform an operation corresponding to rotation of the second hardware input mechanism (e.g. paragraph 0232-0233, Fig. 5B, menu 508 displayed in response to predefined input, and includes plurality of icons 510 including virtual touches selection 510-1, virtual gestures 510-2, virtual device 510-3, and home button 510-4; paragraph 0234-0247, Figs. 5C-O, virtual touches icon 510-1 selected, number of contacts selected, performing gestures on device based on selected number of contacts; virtual gestures icon 510-2 selected, respective gesture is selected, corresponding gesture performed on device; paragraph 0247, Fig. 5P-, virtual device icon 510-3 selected; paragraph 0248, Fig. 5Q, menu of virtual device operations representing plurality of icons representing types of virtual device operations; virtual device icons for muting/unmuting 526-1, device lock 526-4, volume up 526-5, volume down 526-6; virtual device operations including operations usually performed via physical controls on the device, including muting and unmuting the ringer and sound effects on the device, which is normally performed via a mute switch; locking the device, which is normally initiated via a lock/unlock switch; increase volume, which is normally performed via a volume up button; decrease volume, which is normally performed via a volume down button; virtual device rotation icon 526-2 selected; virtual device operations including operations usually performed via physical controls on the device, including shaking device to perform predefined operation, rotating device to rotate display orientation, which are typically detected via accelerometer in the device; paragraph 0249-0250, Fig. 5R-S, icons 530 representing types of virtual device orientations, or rotations of device orientation; user interface rotated 90 degrees clockwise in response to selection of rotate-left orientation icon 530-2, without actual physical rotation of the device; paragraph 0251-0252, Figs. 5T-U, home button icon 510-4 selected, and home screen displayed in response; i.e. selection of a touch-related, home screen, or other device control menu option causes performance of a device function which corresponds to a hardware input mechanism press, such as pressing a volume button, mute or lock switch, home screen button, etc.).
Coenen et al. (US 20190191208 A1) teaches utilizing multiple instances in a gesture in order to change a scrolling speed (e.g. abstract, paragraph 0011, 0082, scrolling speed increased by performing multiple gestures in quick succession)

CITED REFERENCE SUMMARIES: Ref US20190191208A1: Claims:
: A method of providing an interactive user interface for a television display, responsive to a user-operated remote control having a touch-sensitive surface, wherein scrolling of a user interface object is responsive to a directional gesture on the touch-sensitive surface, and wherein the speed of scrolling is dependent on the length of time that a touch is held on the touch-sensitive surface at the end of the directional gesture.
: The method of claim 1 , wherein the speed of scrolling increases from a first speed to a second speed after the touch is held for a predetermined period of time.
: The method of claim 2 , wherein the speed of scrolling increases smoothly from the first speed to the second speed.
: The method of claim 1 , wherein the user interface object comprises an electronic programme guide (âEPGâ).
: The method of claim 4 , wherein the EPG includes a banner showing information relating to a highlighted programme within the EPG, and wherein the banner is automatically hidden during scrolling of the EPG.
: The method of claim 4 , wherein highlighting of a selected programme within the EPG is inhibited during scrolling of the EPG.
: A method of providing an interactive user interface for a television display, responsive to a user-operated remote control having a touch-sensitive surface, wherein the user interface is responsive to gestures performed by the user on the touch-sensitive surface, and wherein the response of the user interface to the gestures is determined by a set of cascaded style sheets defining variable parameters for the interaction of specific ones of the gestures with user interface objects.
: The method of claim 7 , wherein the set of cascaded style sheets define global gesture settings and specific gesture settings which inherit the global gesture settings and further define parameters for specific ones of the gestures.
: The method of claim 7 , wherein the set of cascaded style sheets define user interface object settings.
: The method of claim 9 , wherein the user interface object settings further define which gestures may be used with specific user interface objects.
: The method of claim 7 , wherein the interactive user interface provides a position indicator that interacts with user interface objects, the set of cascaded style sheets defining parameters of the position indicator when interacting with specific user interface objects.
: The method of claim 7 , including remotely modifying at least one of the set of cascaded style sheets.
: The method of claim 7 , including modifying at least one of the set of cascaded styles sheets in response to a selection by the user.
: A method of controlling playback on a display of one or more recorded television programmes, responsive to a user-operated remote control having a touch-sensitive surface, the method comprising skipping the position of playback forward or backward by a predetermined relative time in response to a corresponding user operation on the touch-sensitive surface; wherein the relative time is determined according to the length of the programme being played back.
: The method of claim 14 , where the corresponding user operation comprises a directional gesture on the touch-sensitive surface.
: The method of claim 14 , including fast forwarding or rewinding playback in response to a directional gesture on the touch-sensitive surface.
: The method of claim 16 , wherein the directional gesture and corresponding user operation are performed on separate areas of the touch-sensitive surface.
: The method of claim 16 , wherein the playback is fast forwarded or rewound in response to the direction of said directional output.
: The method of claim 16 , wherein the speed of said fast forwarding or rewinding is controlled according to a magnitude of the directional output.
: The method of claim 16 , wherein the speed of said fast forwarding or rewinding is controlled according to a non-directional output from the touch-sensitive surface.
: A method of providing an interactive user interface for a television display, responsive to a user-operated remote control having a touch-sensitive surface, the method comprising displaying a plurality of programme items in a two-dimensional grid in which programme items are arranged horizontally by broadcast time and vertically by channel, together with a position indicator, wherein the grid scrolls in a horizontal direction, while maintaining the horizontal position of the position indicator fixed, in response to a horizontal gesture on the touch-sensitive surface, until the position indicator reaches a time boundary of the programme items, where the position indicator is enabled to move in said horizontal direction in order to reach a programme item past the time boundary.
: The method of claim 21 , wherein the grid is scrolled in a vertical direction in response to a vertical direction command from a user, while maintaining the vertical position of the position indicator fixed.
: A method of providing an interactive user interface for a television display, responsive to a user-operated remote control having a touch-sensitive surface, wherein scrolling of a user interface object is responsive to a directional gesture on the touch-sensitive surface, and wherein the speed of scrolling is dependent on the number of successive said directional gestures in the same direction.
: The method of claim 23 , wherein the direction of the gesture is resolved into a horizontal or vertical direction, and the speed of scrolling is dependent on the number of successive said directional gestures in the same resolved horizontal or vertical direction.
- 26 . (canceled)
Ref US20120306748A1: Claims:
An electronic device, comprising: a display; a touch-sensitive surface; one or more processors; memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: displaying, on the display, a first visual indicator that corresponds to a virtual touch; receiving a first input from an adaptive input device; and, in response to receiving the first input from the adaptive input device, displaying a first menu on the display, the first menu including a virtual touches selection icon, wherein in response to detecting selection of the virtual touches selection icon, a menu of virtual multitouch contacts is displayed.
a touch-sensitive surface;
one or more processors;
one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: displaying, on the display, a first visual indicator that corresponds to a virtual touch; receiving a first input from an adaptive input device; and, in response to receiving the first input from the adaptive input device, displaying a first menu on the display, the first menu including a virtual touches selection icon, wherein in response to detecting selection of the virtual touches selection icon, a menu of virtual multitouch contacts is displayed.
displaying, on the display, a first visual indicator that corresponds to a virtual touch;
receiving a first input from an adaptive input device; and,
in response to receiving the first input from the adaptive input device, displaying a first menu on the display, the first menu including a virtual touches selection icon, wherein in response to detecting selection of the virtual touches selection icon, a menu of virtual multitouch contacts is displayed.
The device of claim 1 , wherein the first visual indicator comprises one of: a ring, donut, circle, oval, ellipse, arrow, cross, I-beam, star, or virtual finger.
The device of claim 1 , wherein the first input moves the first visual indicator to a predefined region of the display.
The device of claim 1 , wherein the first input corresponds to activation of a control on the adaptive input device.
The device of claim 1 , wherein the first input corresponds to moving a control on the adaptive input device from side-to-side.
The device of claim 1 wherein the adaptive input device includes a joystick and the first input corresponds to moving the joystick from side-to-side.
The device of claim 1 , wherein the first menu includes icons displayed radially about a center of the first menu.
The device of claim 1 , wherein a first user interface is displayed on the display immediately prior to receiving the first input from the adaptive input device, and the first menu is displayed over the first user interface.
The device of claim 1 , including instructions for: in response to receiving the first input from the adaptive input device, displaying the first visual indicator in a center region of the first menu.
in response to receiving the first input from the adaptive input device, displaying the first visual indicator in a center region of the first menu.
The device of claim 1 , including instructions for: detecting selection of the virtual touches selection icon; in response to detecting selection of the virtual touches selection icon, displaying the menu of virtual multitouch contacts, the menu of virtual multitouch contacts including a plurality of icons representing types of virtual multitouch contacts; detecting selection of a respective virtual multitouch contacts icon in the menu of virtual multitouch contacts; and, in response to detecting selection of the respective virtual multitouch contacts icon in the menu of virtual multitouch contacts, displaying one or more second visual indicators that correspond to the respective virtual multitouch contacts icon.
detecting selection of the virtual touches selection icon;
in response to detecting selection of the virtual touches selection icon, displaying the menu of virtual multitouch contacts, the menu of virtual multitouch contacts including a plurality of icons representing types of virtual multitouch contacts;
detecting selection of a respective virtual multitouch contacts icon in the menu of virtual multitouch contacts; and,
in response to detecting selection of the respective virtual multitouch contacts icon in the menu of virtual multitouch contacts, displaying one or more second visual indicators that correspond to the respective virtual multitouch contacts icon.
The device of claim 10 , wherein displaying the second visual indicators includes displaying a number of visual indicators that correspond to the respective virtual multitouch contacts icon.
The device of claim 10 , including instructions for: receiving a second input from the adaptive input device; and, in response to receiving the second input from the adaptive input device: moving the one or more second visual indicators, and performing an operation in accordance with the movement of the one or more second visual indicators.
receiving a second input from the adaptive input device; and,
in response to receiving the second input from the adaptive input device: moving the one or more second visual indicators, and performing an operation in accordance with the movement of the one or more second visual indicators.
moving the one or more second visual indicators, and
performing an operation in accordance with the movement of the one or more second visual indicators.
The device of claim 10 , including instructions for: while displaying the one or more second visual indicators, waiting to receive input from the adaptive input device for more than a predefined time period; and, in response to waiting to receive input from the adaptive input device for more than the predefined time period, replacing display of the one or more second visual indicators with display of the first visual indicator.
while displaying the one or more second visual indicators, waiting to receive input from the adaptive input device for more than a predefined time period; and,
in response to waiting to receive input from the adaptive input device for more than the predefined time period, replacing display of the one or more second visual indicators with display of the first visual indicator.
The device of claim 1 , including instructions for: displaying a virtual gestures icon; detecting selection of the virtual gestures icon; in response to detecting selection of the virtual gestures icon, displaying a menu of virtual gestures, the menu of virtual gestures including a plurality of icons representing types of virtual gestures; detecting selection of a respective virtual gesture icon in the menu of virtual gestures; and, in response to detecting selection of the respective virtual gesture icon in the menu of virtual gestures, performing an action that corresponds to the respective virtual gesture.
displaying a virtual gestures icon;
detecting selection of the virtual gestures icon;
in response to detecting selection of the virtual gestures icon, displaying a menu of virtual gestures, the menu of virtual gestures including a plurality of icons representing types of virtual gestures;
detecting selection of a respective virtual gesture icon in the menu of virtual gestures; and,
in response to detecting selection of the respective virtual gesture icon in the menu of virtual gestures, performing an action that corresponds to the respective virtual gesture.
The device of claim 1 , including instructions for: displaying a virtual gestures icon; detecting selection of the virtual gestures icon; in response to detecting selection of the virtual gestures icon, displaying a menu of virtual gestures, the menu of virtual gestures including a plurality of icons representing types of virtual gestures; detecting selection of a respective virtual gesture icon in the menu of virtual gestures; in response to detecting selection of the respective virtual gesture icon in the menu of virtual gestures, displaying a number of visual indicators that correspond to the respective virtual gesture icon; receiving a second input from the adaptive input device; and, in response to receiving the second input from the adaptive input device, performing an operation in accordance with the respective virtual gesture.
detecting selection of a respective virtual gesture icon in the menu of virtual gestures;
in response to detecting selection of the respective virtual gesture icon in the menu of virtual gestures, displaying a number of visual indicators that correspond to the respective virtual gesture icon;
in response to receiving the second input from the adaptive input device, performing an operation in accordance with the respective virtual gesture.
The device of claim 15 , including instructions for: in response to receiving the second input from the adaptive input device, moving the displayed visual indicators that correspond to the respective virtual gesture to simulate movement of actual contacts on the touch sensitive surface.
in response to receiving the second input from the adaptive input device, moving the displayed visual indicators that correspond to the respective virtual gesture to simulate movement of actual contacts on the touch sensitive surface.
The device of claim 1 , including instructions for: displaying a pinch/depinch gesture icon; detecting selection of the virtual pinch/depinch gesture icon; in response to detecting selection of the virtual pinch/depinch gesture icon, displaying two visual indicators that correspond to contacts in the virtual pinch/depinch gesture; receiving a second input from the adaptive input device; and, in response to receiving the second input from the adaptive input device, performing an operation in accordance with the virtual pinch/depinch gesture.
displaying a pinch/depinch gesture icon;
detecting selection of the virtual pinch/depinch gesture icon;
in response to detecting selection of the virtual pinch/depinch gesture icon, displaying two visual indicators that correspond to contacts in the virtual pinch/depinch gesture;
in response to receiving the second input from the adaptive input device, performing an operation in accordance with the virtual pinch/depinch gesture.
The device of claim 17 , wherein a zooming operation is performed in accordance with the virtual pinch/depinch gesture.
The device of claim 1 , including instructions for: displaying a virtual device icon; detecting selection of the virtual device icon; in response to detecting selection of the virtual device icon, displaying a menu of virtual device operations, the menu of virtual device operations including a plurality of icons representing types of virtual device operations; detecting selection of a respective virtual device operation icon in the menu of virtual device operations; and, in response to detecting selection of the respective virtual device operation icon in the menu of virtual device operations, performing an action that corresponds to the respective virtual device operation.
displaying a virtual device icon;
detecting selection of the virtual device icon;
in response to detecting selection of the virtual device icon, displaying a menu of virtual device operations, the menu of virtual device operations including a plurality of icons representing types of virtual device operations;
detecting selection of a respective virtual device operation icon in the menu of virtual device operations; and,
in response to detecting selection of the respective virtual device operation icon in the menu of virtual device operations, performing an action that corresponds to the respective virtual device operation.
The device of claim 1 , including instructions for: displaying a virtual device rotation icon; detecting selection of the virtual device rotation icon; in response to detecting selection of the virtual device rotation icon, displaying a menu of virtual device orientations, the menu of virtual device orientations including a plurality of icons representing types of virtual device orientations; detecting selection of a respective virtual device orientation icon in the menu of virtual device orientations; and, in response to detecting selection of the respective virtual device orientation icon in the menu of virtual device orientations, orienting the display in accordance with the respective virtual device orientation.
displaying a virtual device rotation icon;
detecting selection of the virtual device rotation icon;
in response to detecting selection of the virtual device rotation icon, displaying a menu of virtual device orientations, the menu of virtual device orientations including a plurality of icons representing types of virtual device orientations;
detecting selection of a respective virtual device orientation icon in the menu of virtual device orientations; and,
in response to detecting selection of the respective virtual device orientation icon in the menu of virtual device orientations, orienting the display in accordance with the respective virtual device orientation.
The device of claim 1 , wherein the device includes a home button, the device including instructions for: displaying a home button icon; detecting selection of the home button icon; and, in response to detecting selection of the home button icon, performing an action that corresponds to activation of the home button.
displaying a home button icon;
detecting selection of the home button icon; and,
in response to detecting selection of the home button icon, performing an action that corresponds to activation of the home button.
The device of claim 1 , including instructions for: while displaying a menu on the display, constraining movement of the first visual indicator such that the first visual indicator remains on the menu.
while displaying a menu on the display, constraining movement of the first visual indicator such that the first visual indicator remains on the menu.
The device of claim 1 , including instructions for: while displaying the first menu on the display, r
[Truncated to input limit]