Office Action 3: Non-Final Rejection (2024-01-04)
Rejection types: 103 - Obviousness

CLAIMS:
AMENDMENTS TO THE CLAIMS

This listing of the claims will replace all prior versions and listing of claims in the application:

1\. (Currently Amended) A computer system that is in communication with a display generation component and an optical sensor, the computer system comprising:one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected;
while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture, wherein before detecting the hand gesture, the computer system is in a first operation mode;
in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is a first type of gesture, displaying, via the display generation component, an indication that the second user interface object is selected;
in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected; and in accordance with a determination that the hand gesture is a third type of gesture that is different from the first type of gesture and the second type of gesture, performing an operation corresponding to selection of the first user interface object, wherein performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode, wherein when the computer system is in the second operation mode, the computer system performs an automatic

scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture;while the computer system is in the second operation mode, detecting, via at least the optical sensor, a second hand gesture; and
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation ; \[\[and\]\] in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation;
in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation; and in accordance with a determination that the second hand gesture is a fourth type of gesture different from the first type of gesture, the second type of gesture, and the third type of gesture, ending the automatic scrolling operation.

2\. (Original) The computer system of claim 1, wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor.

3\. (Original) The computer system of claim 1, wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture.

4\. (Currently Amended) The computer system of claim 1, wherein the one or more programs further include instructions for:in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is \[\[a\]\] the fourth type of gesture , displaying, via the display generation component, a menu that includes one or more selectable options.

5\. (Canceled)

6\. (Canceled)

7\. (Canceled)

8\. (Previously Presented) The computer system of claim 1, wherein the automatic scrolling operation scrolls a sequence of a plurality of interface objects in a second direction, and wherein the one or more programs further include instructions for:while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects; and
in response to detecting the end of the sequence of the plurality of user interface objects,
performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction.

9\. (Canceled)

10\. (Previously Presented) The computer system of claim 1, wherein the one or more programs further include instructions for:in accordance with a determination that the second hand gesture is the first type of gesture, displaying a notification that indicates a state of the automatic scrolling operation.

11\. (Currently Amended) The computer system of claim \[\[4\]\] 1, wherein the third type of gesture is a type of gesture that is multiple instances of the fourth type of gesture.

12\. (Currently Amended) The computer system of claim \[\[4\]\] 1, wherein the fourth type of gesture does not include multiple instances of the first type of gesture and the third type of gesture does not include multiple instances of the second type of gesture.

13\. (Currently Amended) The computer system of claim \[\[4\]\] 1, wherein the one or more programs further include instructions for:detecting, via at least the optical sensor, a third hand gesture; and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that a notification was received within a threshold period of time and in accordance with a determination that the third hand gesture is the fourth type of gesture, performing an action related to the notification.

14\. (Previously Presented) The computer system of claim 4, wherein the one or more selectable options includes a first selectable user interface object for changing operations that one or more hand gestures can cause to the computer system to perform, wherein selection of the first selectable user interface object causes the computer system to display a plurality of settings, wherein each setting controls an operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system.

15\. (Original) The computer system of claim 4, wherein:the one or more selectable options includes a second selectable option for transitioning the computer system to a fourth operation mode and a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode;
selection of the second selectable option causes the computer system to transition to the fourth operation mode; and
selection of the third selectable option causes the computer system to transition to the fifth operation mode.

16\. (Original) The computer system of claim 4, wherein the one or more selectable options include a fourth selectable option for displaying one or more additional selectable options, and wherein selection of the fourth selectable option causes the computer system to display the one or more

additional selectable options that were not previously displayed before selection of the fourth selectable option.

17\. (Original) The computer system of claim 4, wherein the menu is displayed:in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location; and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location.

18\. (Original) The computer system of claim 4, wherein the one or more programs further include instructions for:while displaying the menu that includes one or more selectable options, detecting, via at least the optical sensor, a third hand gesture; and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that the third hand gesture is the fourth type of gesture, ceasing to display the menu that includes the one or more selectable options.

19\. (Original) The computer system of claim 1, wherein the one or more programs further include instructions for:after displaying the user interface that includes the first user interface object, detecting, via at least the optical sensor, a fourth hand gesture; and
in response to detecting, via at least the optical sensor, the fourth hand gesture:
in accordance with a determination that the fourth hand gesture is a fifth type of gesture that is different from the first type of gesture and the second type of gesture, transitioning the computer system from an inactive state to an active state.

20\. (Previously Presented) The computer system of claim 1, wherein the one or more programs further include instructions for:while displaying, via the display generation component, the indication that the second user interface object is selected, detecting, via at least the optical sensor, a fifth hand gesture; and
in response to detecting, via at least the optical sensor, the fifth hand gesture:
in accordance with a determination that the fifth hand gesture is the first type of gesture, displaying, via the display generation component, an indication that a fourth user interface object is selected; and
in accordance a determination that the fifth hand gesture is the second type of gesture, displaying, via the display generation component, the indication that the first user interface object is selected.

21\. (Previously Presented) The computer system of claim 1, wherein the one or more programs further include instructions for:while displaying the user interface that includes the first user interface object, detecting a request to transition the computer system from the first operation mode to a fourth operation mode;
in response to detecting the request to transition the computer system from the first operation mode to the fourth operation mode, transitioning the computer system from the first operation mode to the fourth operation mode;
while the computer system is in the fourth operation mode and while displaying the user interface that includes the first user interface object, the second user interface object, and the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a sixth hand gesture; and
in response detecting, via least the optical sensor, the sixth hand gesture:
in accordance with a determination that the sixth hand gesture is the first type of gesture, continuing to display the indication that the first user interface object is selected; and
in accordance with a determination that the sixth hand gesture is the second type of gesture, continuing to display the indication that the first user interface object is selected.

22\. (Currently Amended) A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system that is in communication with a display generation component and an optical sensor, the one or more programs including instructions for:displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected;
while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture, wherein before detecting the hand gesture, the computer system is in a first operation mode;
in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is a first type of gesture,
displaying, via the display generation component, an indication that the second user interface object is selected;in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected; and
in accordance with a determination that the hand gesture is a third type of gesture that is different from the first type of gesture and the second type of gesture, performing an operation corresponding to selection of the first user interface object, wherein performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode, wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture;
while the computer system is in the second operation mode, detecting, via at least the optical sensor, a second hand gesture; and
in response to detecting, via at least the optical sensor, the second hand gesture:

in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation ; \[\[and\]\] in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation;
in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation; and
in accordance with a determination that the second hand gesture is a fourth type of gesture different from the first type of gesture, the second type of gesture, and the third type of gesture, ending the automatic scrolling operation.

23\. (Currently Amended) A method, comprising:at a computer system that is in communication with a display generation component and an optical sensor:
displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected;
while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture, wherein before detecting the hand gesture, the computer system is in a first operation mode;
in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is a first type of gesture, displaying, via the display generation component, an indication that the second user interface object is selected;
in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected; and

in accordance with a determination that the hand gesture is a third type of gesture that is different from the first type of gesture and the second type of gesture, performing an operation corresponding to selection of the first user interface object, wherein performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode, wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture;
while the computer system is in the second operation mode, detecting, via at least the optical sensor, a second hand gesture; and
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation ; \[\[and\]\] in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation;
in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation; and in accordance with a determination that the second hand gesture is a fourth type of gesture different from the first type of gesture, the second type of gesture, and the third type of gesture, ending the automatic scrolling operation.

24\. (Previously Presented) The non-transitory computer-readable storage medium of claim 22, wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor.

25\. (Previously Presented) The non-transitory computer-readable storage medium of claim 22, wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture.

26\. (Currently Amended) The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs further include instructions for:in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is \[\[a\]\] the fourth type of gesture , displaying, via the display generation component, a menu that includes one or more selectable options.

27\. (Canceled)

28\. (Previously Presented) The non-transitory computer-readable storage medium of claim 22, wherein the automatic scrolling operation scrolls a sequence of a plurality of interface objects in a second direction, and wherein the one or more programs further include instructions for:while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects; and
in response to detecting the end of the sequence of the plurality of user interface objects,
performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction.

29\. (Canceled)

30\. (Previously Presented) The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs further include instructions for:in accordance with a determination that the second hand gesture is the first type of gesture, displaying a notification that indicates a state of the automatic scrolling operation.

31\. (Currently Amended) The non-transitory computer-readable storage medium of claim 22\[\[26\]\], wherein the third type of gesture is a type of gesture that is multiple instances of the fourth type of gesture.

32\. (Currently Amended) The non-transitory computer-readable storage medium of claim 22\[\[26\]\], wherein the fourth type of gesture does not include multiple instances of the first type of gesture and the third type of gesture does not include multiple instances of the second type of gesture.

33\. (Currently Amended) The non-transitory computer-readable storage medium of claim 22\[\[26\]\], wherein the one or more programs further include instructions for:detecting, via at least the optical sensor, a third hand gesture; and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that a notification was received within a threshold period of time and in accordance with a determination that the third hand gesture is the fourth type of gesture, performing an action related to the notification.

34\. (Previously Presented) The non-transitory computer-readable storage medium of claim 26, wherein the one or more selectable options includes a first selectable user interface object for changing operations that one or more hand gestures can cause to the computer system to perform, wherein selection of the first selectable user interface object causes the computer system to display a plurality of settings, wherein each setting controls an operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system.

35\. (Previously Presented) The non-transitory computer-readable storage medium of claim 26, wherein:the one or more selectable options includes a second selectable option for transitioning the computer system to a fourth operation mode and a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode;
selection of the second selectable option causes the computer system to transition to the fourth operation mode; and

selection of the third selectable option causes the computer system to transition to the fifth operation mode.

36\. (Previously Presented) The non-transitory computer-readable storage medium of claim 26, wherein the one or more selectable options include a fourth selectable option for displaying one or more additional selectable options, and wherein selection of the fourth selectable option causes the computer system to display the one or more additional selectable options that were not previously displayed before selection of the fourth selectable option.

37\. (Previously Presented) The non-transitory computer-readable storage medium of claim 26, wherein the menu is displayed:in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location; and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location.

38\. (Previously Presented) The non-transitory computer-readable storage medium of claim 26, wherein the one or more programs further include instructions for:while displaying the menu that includes one or more selectable options, detecting, via at least the optical sensor, a third hand gesture; and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that the third hand gesture is the fourth type of gesture, ceasing to display the menu that includes the one or more selectable options.

39\. (Previously Presented) The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs further include instructions for:after displaying the user interface that includes the first user interface object, detecting, via at least the optical sensor, a fourth hand gesture; and
in response to detecting, via at least the optical sensor, the fourth hand gesture:

in accordance with a determination that the fourth hand gesture is a fifth type of gesture that is different from the first type of gesture and the second type of gesture, transitioning the computer system from an inactive state to an active state.

40\. (Previously Presented) The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs further include instructions for:while displaying, via the display generation component, the indication that the second user interface object is selected, detecting, via at least the optical sensor, a fifth hand gesture; and
in response to detecting, via at least the optical sensor, the fifth hand gesture:
in accordance with a determination that the fifth hand gesture is the first type of gesture, displaying, via the display generation component, an indication that a fourth user interface object is selected; and
in accordance a determination that the fifth hand gesture is the second type of gesture, displaying, via the display generation component, the indication that the first user interface object is selected.

41\. (Previously Presented) The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs further include instructions for:while displaying the user interface that includes the first user interface object, detecting a request to transition the computer system from the first operation mode to a fourth operation mode;
in response to detecting the request to transition the computer system from the first operation mode to the fourth operation mode, transitioning the computer system from the first operation mode to the fourth operation mode;
while the computer system is in the fourth operation mode and while displaying the user interface that includes the first user interface object, the second user interface object, and the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a sixth hand gesture; and
in response detecting, via least the optical sensor, the sixth hand gesture:

in accordance with a determination that the sixth hand gesture is the first type of gesture, continuing to display the indication that the first user interface object is selected; and
in accordance with a determination that the sixth hand gesture is the second type of gesture, continuing to display the indication that the first user interface object is selected.

42\. (Previously Presented) The method of claim 23, wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor.

43\. (Previously Presented) The method of claim 23, wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture.

44\. (Currently Amended) The method of claim 23, further comprising:in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is \[\[a\]\] the fourth type of gesture , displaying, via the display generation component, a menu that includes one or more selectable options.

45\. (Canceled)

46\. (Previously Presented) The method of claim 23, wherein the automatic scrolling operation scrolls a sequence of a plurality of interface objects in a second direction, and wherein the method further comprises:while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects; and
in response to detecting the end of the sequence of the plurality of user interface objects,
performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction.

47\. (Canceled)

48\. (Previously Presented) The method of claim 23, further comprising:in accordance with a determination that the second hand gesture is the first type of gesture, displaying a notification that indicates a state of the automatic scrolling operation.

49\. (Currently Amended) The method of claim 23 \[\[44\]\], wherein the third type of gesture is a type of gesture that is multiple instances of the fourth type of gesture.

50\. (Currently Amended) The method of claim 23 \[\[44\]\], wherein the fourth type of gesture does not include multiple instances of the first type of gesture and the third type of gesture does not include multiple instances of the second type of gesture.

51\. (Currently Amended) The method of claim 23 \[\[44\]\], further comprising:detecting, via at least the optical sensor, a third hand gesture; and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that a notification was received within a threshold period of time and in accordance with a determination that the third hand gesture is the fourth type of gesture, performing an action related to the notification.

52\. (Previously Presented) The method of claim 44, wherein the one or more selectable options includes a first selectable user interface object for changing operations that one or more hand gestures can cause to the computer system to perform, wherein selection of the first selectable user interface object causes the computer system to display a plurality of settings, wherein each setting controls an operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system.

53\. (Previously Presented) The method of claim 44, wherein:the one or more selectable options includes a second selectable option for transitioning the computer system to a fourth operation mode and a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode;
selection of the second selectable option causes the computer system to transition to the fourth operation mode; and
selection of the third selectable option causes the computer system to transition to the fifth operation mode.

54\. (Previously Presented) The method of claim 44, wherein the one or more selectable options include a fourth selectable option for displaying one or more additional selectable options, and wherein selection of the fourth selectable option causes the computer system to display the one or more additional selectable options that were not previously displayed before selection of the fourth selectable option.

55\. (Previously Presented) The method of claim 44, wherein the menu is displayed:in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location; and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location.

56\. (Previously Presented) The method of claim 44, further comprising:while displaying the menu that includes one or more selectable options, detecting, via at least the optical sensor, a third hand gesture; and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that the third hand gesture is the fourth type of gesture, ceasing to display the menu that includes the one or more selectable options.

57\. (Previously Presented) The method of claim 23, further comprising:

after displaying the user interface that includes the first user interface object, detecting, via at least the optical sensor, a fourth hand gesture; and
in response to detecting, via at least the optical sensor, the fourth hand gesture:
in accordance with a determination that the fourth hand gesture is a fifth type of gesture that is different from the first type of gesture and the second type of gesture, transitioning the computer system from an inactive state to an active state.

58\. (Previously Presented) The method of claim 23, further comprising:while displaying, via the display generation component, the indication that the second user interface object is selected, detecting, via at least the optical sensor, a fifth hand gesture; and
in response to detecting, via at least the optical sensor, the fifth hand gesture:
in accordance with a determination that the fifth hand gesture is the first type of gesture, displaying, via the display generation component, an indication that a fourth user interface object is selected; and
in accordance a determination that the fifth hand gesture is the second type of gesture, displaying, via the display generation component, the indication that the first user interface object is selected.

59\. (Previously Presented) The method of claim 23, further comprising:while displaying the user interface that includes the first user interface object, detecting a request to transition the computer system from the first operation mode to a fourth operation mode;
in response to detecting the request to transition the computer system from the first operation mode to the fourth operation mode, transitioning the computer system from the first operation mode to the fourth operation mode;
while the computer system is in the fourth operation mode and while displaying the user interface that includes the first user interface object, the second user interface object, and the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a sixth hand gesture; and
in response detecting, via least the optical sensor, the sixth hand gesture:

in accordance with a determination that the sixth hand gesture is the first type of gesture, continuing to display the indication that the first user interface object is selected; and
in accordance with a determination that the sixth hand gesture is the second type of gesture, continuing to display the indication that the first user interface object is selected.

OFFICE ACTION:
DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the Amendment filed on November 30, 2023.  Claims 1, 4, 11-13, 22, 23, 26, 31-33, 44, and 49-51 are amended.  Claims 5, 6, 7, 9, 27, 29, 45, and 47 are cancelled.  Therefore, Claims 1-4 8, 10-26, 28, 30-44, 46, and 48-59 are pending in the case.  Claims 1, 22, and 23 are the independent claims.
This action is non-final.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection.  Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114.  Applicant's submission filed on November 30, 2023 has been entered.
Applicant’s Response
In the Amendment filed on November 30, 2023, Applicant amended the claims and provided arguments in response to rejection of the claims under 35 USC 103 in the previous office action.
Response to Argument/Amendment
Applicant’s amendments to the claims in response to the rejection of the claims under 35 USC 103 in the previous office action are acknowledged, and Applicant’s associated arguments have been fully considered.  Applicant argues that the cited references (Osborn, Kim, and Jose) fail to teach or suggest four different types of hand gestures for controlling four respective different autoscrolling functions (pausing, reversing direction, changing speed, and ending), including changing speed of scrolling based on type of hand gesture detected, detecting a first type of hand gesture to pause an autoscrolling operation and a different fourth type of gesture to perform the different action of ending the autoscrolling operation, and using a third type of gesture for changing speed of the automatic scrolling operation which is the same third type of gesture that selects an object to transition the system into the autoscrolling mode.
Applicant’s arguments are persuasive, and the rejection is withdrawn.  However, new grounds of rejection are provided below.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims under pre-AIA  35 U.S.C. 103(a), the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were made absent any evidence to the contrary.  Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and invention dates of each claim that was not commonly owned at the time a later invention was made in order for the examiner to consider the applicability of pre-AIA  35 U.S.C. 103(c) and potential pre-AIA  35 U.S.C. 102€, (f) or (g) prior art under pre-AIA  35 U.S.C. 103(a).
Claims 1, 3, 4, 12, 14-16, 19-23, 25-27, 32, 34-36, 39-41, 43-45, 50, 52-54, 57, and 58 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn et al. (US 20230072423 A1) in view of Li (US 20160328108 A1) , further in view of Kin (US 10261595 B1), further in view of Pantel (US 20140320434 A1).
With respect to claims 1, 22, and 23, Osborn teaches a computer system that is in communication with a display generation component and an optical sensor, the computer system comprising: one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method; a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system that is in communication with a display generation component and an optical sensor, the one or more programs including instructions for performing the method (e.g. paragraph 0257, supplementing sensor data with images of user captured by camera; paragraph 0286, auxiliary devices/sensors including imaging devices/cameras, heart rate monitors, etc.; paragraph 0288, Figs. 9A-B, wearable system with sensors such as EMG sensors, sensors 810; computer processor such as microcontroller 934 receiving inputs from sensors; communicating with dongle portion 920 and outputting processing output to dongle portion; paragraph 0289, dongle portion 920 receiving signals from wearable portion 910, and providing them to host computer for further processing, display, effecting control of objects, etc.; paragraph 0292, describing Fig. 1, system including processors 112 programmed to communicate with sensors 110 and executing algorithms; paragraph 0301, system including display controller configured to display visual representations; computer processors implementing trained inference models that receive sensor signals as input and provide information to generate control signals for system as output; paragraph 0302, user interface; paragraph 0303, various types of devices in which processors may be included; paragraph 0304, multiple processors; paragraph 0306, system including processors, camera, and display; paragraph 310, camera information/cameras; paragraph 0311, camera mounted on user; paragraph 0344, embodiments implemented in software executed on processors; paragraph 0345, non-transitory computer-readable storage medium/memory encoded with computer program/instructions executed on processor to perform described functions; paragraph 0352, sensor including light emitter/detector; paragraph 0385, Fig. 13, vibrotactile system 1300 may be standalone system, or may be configured with interaction with various types of other devices, and includes processor 1360; paragraph 0610, cameras; paragraph 0729, photoplethysmography/PPG sensors detecting vascular changes/changes in blood volume; paragraph 1384, Fig. 35B, wearable device 35102 capable of controlling computer devices via neuromuscular signals of users, including set of sensors 35104 that detect/sense neuromuscular signals traversing body of user; processing device 35106 communicatively coupled to sensors, and memory 35108 including models and computer-executable instructions that when executed by processor cause it to identify force generated, generate commands for HCI system, etc.; i.e. the device includes at least one processor and memory storing instructions for execution by the processor, and is further in communication with at least one display/display controller/generation component, and at least one optical sensor, such as a camera (which may be mounted on the user)); and the method, comprising:
displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected (e.g. paragraph 1460, describing radial menu 351000 in Figs. 35O-V, where the menu may include visual indicators including a selection indicator 351020; paragraph 1461, selection indicator 351020 may indicate a currently selected action indicator; paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual, auditory, or haptic feedback; visual feedback representing change of a characteristic of visual element within radial menu in response to selection of the visual indicator, such as position change, orientation change, color change, size change, transparency change, fill change, emphasis change, shadow change, animation change, font change, line type change, line width change, combinations, etc.; paragraph 1473, describing Figs. 36B-C, radial menu 351110 including a set of action indicators including indicators 351111 and 351113 displayed in GUI along arc, where as shown in Fig. 36B includes an indication that indicator 35111 is selected; demonstrating/confirming selection of visual indicator using visual feedback; paragraph 1475, describing Figs. 36D-E, displaying sequential menu 351120 including a sequence of visual indicators depicted as A through F, where as shown in Fig. 36 D includes in indication that indicator D/351123 is selected; i.e. both of the menus/interfaces shown in the cited figures include at least first, second, and third UI objects along with an indication that an object of these is selected, such as a visual feedback including a shading/coloration of a selected indicator; paragraphs 1385-1387, determining based on neuromuscular signals state of body part of user and generating input commands accounting for state of user’s body part; sensors constitute any type of form of sensor capable of detecting/sensing neuromuscular signals in user’s body including EMG sensors, MMG sensors, SMG sensors; sensors may also include auxiliary sensors such as IMU, position tracking, imaging (camera), and radiation based sensors, microphones, heart rate monitors, etc.; paragraph 1394, wearable device 35102 coupled with computing system/display; user placing body in state/condition to control/modify presentation/performance of computing system, generating neuromuscular signals representative/indicative/suggestive of state/action);
while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture (e.g. paragraph 0310, camera information used to improve interpretation of neuromuscular signals; camera used to record image of user, may be within particular range/wavelengths of light, images produced by visible/non-visible light, including showing heat distribution in user’s body, providing indication of blood flow within user and used to infer condition of user such as forces being exerted by fingers; paragraph 0311, camera mounted on user; paragraph 0312, camera data in addition to neuromuscular activity signals used to reconstruct positions, geometries, and forces being applied by user’s arm/hand; paragraph 0323, auxiliary signals from auxiliary devices such as camera supplementing neuromuscular signals; paragraph 0610, cameras used to improve interpretation of signals; paragraph 0611, camera data used to provide real time determination of musculoskeletal representations of gestures; paragraph 0729, PPG sensors detecting vascular changes; paragraph 1271, camera/imaging data; paragraph 1387, sensors including auxiliary sensors such as cameras; paragraph 1395, user pose/gesture generating neuromuscular signals as they traverse down arm toward hand; sensors 35104 detecting/sensing neuromuscular signals and generating data representative of the signals; paragraph 1396-1398, sensor data undergoing processing, analyzing data to determine state of body parts of user, producing/outputting classification indicating state of body part; paragraph 1398, gestures; paragraph 1400, causing computing system to perform actions mapped to state of body parts; paragraph 1401, describing various actions mapped to different states of user’s body parts; paragraph 1444, capturing gestures with camera; photographic data of hand gestures; paragraph 1473, body state mapped to selection of next indicator in a given sequence; identification of body state; paragraph 1474, body states including wrist extension, flexion, rotation, radial deviation, ulnar deviation, etc.; paragraph 1475, body states mapped to selection of visual indicators in sequence; identification of body states; i.e. detecting a body state of the user, such as a hand gesture, including by using an optical sensor such as a camera,  PPG, etc., possibly in combination with other sensors),
wherein before detecting the hand gesture, the computer system is in a first operation mode (e.g. paragraph 1463, Fig. 35O, fist pose mapped to command to display, enter, and activate radial menu; i.e. where in the first state, at the time the radial menu is displayed, the user has provided a previous input causing the radial menu to be activated, and therefore in an operation mode in which the menu is activated, analogous to a first operation mode);
in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is a first type of gesture, displaying, via the display generation component, an indication that the second user interface object is selected (e.g. paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual feedback; paragraph 1473, Figs. 36B-C, first body state mapped to selection of the next indicator in a given sequence; in response to identification of first body state, deselecting indicator 35111 and selecting indicator 351113; demonstrating/confirming selection of visual indicator using visual feedback; paragraph 1475, Figs. 36D-E, first body state mapped to selection of next visual indicator in the sequence; i.e. in response to detecting a first type of gesture, as indicated by a first body state, the indication that a first UI object is selected may be changed to an indication that a second UI object is selected, such as by moving the indicator of the selection to the second UI object, where the second UI object is a next UI object in a sequence with respect to the first UI object); and
in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected (e.g. paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual feedback; paragraph 1473-1474, Figs. 36B-C, first body state mapped to selection of next indicator in given sequence; demonstrating/confirming selection of visual indicator using visual feedback; first and second body states may represent counterparts of one another; 1475, Figs. 36D-E, second body state mapped to selection of previous visual indicator in the sequence; in response to identification of second body state, deselecting subsequent indicator 351123 and selecting prior indicator 351121; i.e. in response to detecting a second/different type of gesture, as indicated by a second body state, the indication that a first UI object is selected may be changed to an indication that a third UI object is selected, such as by moving the indicator of the selection to the third UI object, where the third UI object is a previous UI object in a sequence with respect to the first UI object; in addition, because these gestures cause selection to move to a next/previous object in sequence, receipt of multiple of either of these gestures may also cause a different/third UI object to be selected; for example, where multiple instances/two instances of the first type of gesture are received, the object selected as indicated may be first moved from the first UI object to the second/next UI object in the sequence, and then moved again from the second/next UI object to a third/subsequent UI object in the sequence); and
in accordance with a determination that the hand gesture is a third type of gesture that is different from the first type of gesture and the second type of gesture, performing an operation corresponding to selection of the first user interface object, wherein performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode (e.g. paragraph 1468, Fig. 35Q, illustrating finger pinch pose 351006, which may be mapped to command/instruction to click visual indicator in the radial menu, analogous to mouse button click or keypress; alternatively combination of finger pinch pose and certain degree of force mapped to click command; paragraph 1469, Fig. 35 U, clicking/engaging with or activating function of the currently selected action indicator; clicking action indicator in response to identified click gesture; upon performing the click action, causing visual indicator to appear to depress and/or release, similar to a physical button; i.e. clicking the currently selected action indicator causes the corresponding function to be activated, and therefore causes the system to transition to an operation mode in which the corresponding function has been activated, analogous to a second operation mode);
detecting, via at least the optical sensor, a second hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that the second hand gesture is one type of gesture, toggling a first automatic scrolling operation between an active state and an inactive state (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling up a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll up a GUI; paragraph 1484, ring finger pinch mapped to scroll up so that when user makes ring finger pinch pose, the page/browser displayed scrolls up); and
in accordance with a determination that the second hand gesture is another type of gesture, reversing a first direction of the first automatic scrolling operation  (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling down a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll down a GUI; paragraph 1484, pinky finger pinch mapped to scroll down so that when user makes pinky finger pinch pose, the page/browser displayed scrolls down).
Although Osborn further teaches that user hand gestures may be mapped to different commands in different states/modes, and further teaches that a wrist roll gesture may be mapped to a scroll command (e.g. paragraph 0648, switching between modes; paragraph 0666-0667, selection of mode, such as from list or using gesture; paragraph 1380, transitioning between different mappings of body part states and responsive actions; paragraph 1408-1410, activating first mapping and/or deactivating second mapping between possible body part states and responsive actions; user switching between mappings; transition/switch between mappings; paragraph 1486, user modifying mappings between body states and actions, user configuring/mapping scroll controls to different poses; table accompanying paragraph 1649, indicating that an interaction with a list feature may include a “scroll through” interaction with a “wrist roll” control scheme), Osborn does not explicitly disclose:
wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture,
while the computer system is in the second operation mode, detecting the second hand gesture;
in response to detecting the second hand gesture:
in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation;
in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation; and
in accordance with a determination that the second hand gesture is a fourth type of gesture different from the first type of gesture, the second type of gesture, and the third type of gesture, ending the automatic scrolling operation.
However, Li teaches:
wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture (e.g. paragraph 0037-0038, utilizing auto-scroll method such that contents are presented in multiple steps/views within given time period; user starting content show with auto scroll where contents are presented and scrolled automatically and continuously; user selecting scroll options, etc.; paragraph 0046, auto scroll replaces contents on display at given rate continuously and without user involvement),
while the computer system is in the second operation mode, detecting the second hand gesture (e.g. paragraph 0046, describing user inputs/gestures provided while content is automatically scrolled, such as a finger hover/touch input gesture to pause scrolling; paragraph 0047, describing other types of gesture which may be used as command input, including user holding hand upright with palm facing a horizontal direction, user hand movements to make selections, etc.; paragraph 0053, describing user methods for stopping a scroll process temporarily, such as selecting on screen buttons, touch inputs, and use of gestures via gesture recognition system, such as by moving hand or fingertip in a given direction, where the user’s hand may be in any position, vertical, horizontal, tilted, using both a fist or spread hand for hand gestures, and using one or multiple fingertips for finger gestures);
in response to detecting the second hand gesture:
in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation (e.g. paragraph 0046-0047, user inputs to pause scrolling, including hand gestures; paragraph 0053, user input to stop scroll process temporarily, including use of hand gestures, including hand gestures in a given direction, using a tilted hand, etc.; i.e. the user may provide a hand gesture to pause scrolling, including a directional/tilted hand gesture, analogous to pausing an automatic scrolling operation in accordance with a determination that a hand gesture is a first type of hand gesture);
in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation (e.g. paragraph 0044-0045, scroll direction may be reversed using gesture, such as swipe on touch screen; via buttons, etc.; paragraph 0047, describing use of other gestures as command input, including directional movements to select an option; paragraph 0053, user turning page forward or backward during scroll session, including use of hand gestures, including hand gestures in a given direction, using a tilted hand, etc.; i.e. the user may provide a hand gesture to reverse scrolling direction, or to turn pages in any direction (including an opposite direction) during automatic scrolling, including a directional/tilted hand gesture, analogous to reversing a direction of scrolling in accordance with a determination that a hand gesture is a second type of gesture); and
in accordance with a determination that the second hand gesture is a fourth type of gesture different from the first type of gesture, the second type of gesture, and the third type of gesture, ending the automatic scrolling operation (e.g. paragraph 0043, selection of button to close content window and end scroll session; selection of no button to stop scroll process; scroll session controlled by device movement, etc.; paragraph 0047, describing various user gestures which may be used as command input, including hand movements to select a no icon/option; paragraph 0053, describing various hand gestures which may be used as command inputs, including hand gestures with a fist; paragraph 0054, option to quit auto scroll prepared; tapping/clicking/selecting quit button; i.e. the user may provide a different hand gesture, such as a closed fist or other hand gesture, in order to provide a command to close/exit the auto scroll operation mode).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Li in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), to incorporate the teachings of Li (directed to presenting contents using auto scroll) to include the capability to place the system in second mode in response to a third type of user input (i.e. such as a mode of executing a corresponding functionality following a user’s selection activating a displayed item, as taught by Osborn, and a mode of initiating automatic and continuous content scrolling functionality following a user’s selection of a corresponding user interface component/option, as taught by Li), where the second mode includes an automatic scrolling function and, while in this automatic scrolling mode, receive an input/gesture to pause the automatic scrolling (including a directional/tilted hand gesture, analogous to a first type of gesture, as taught by Li, similar to the directional wrist rotation gesture as taught by Osborn), receive an input/gesture to reverse the direction of the automatic scrolling (including a directional/tilted hand gesture in a direction corresponding to a reverse direction, as taught by Li, similar to the directional wrist rotation gesture as taught by Osborn), and receive an input/gesture to exit/quit the auto scroll operation/mode (including a gesture using a hand in a fist and/or selecting a corresponding option, as taught by Li, similar to a fist gesture or other gesture as taught by Osborn).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability for the user to relax and watch contents effortlessly, as described in Li (paragraph 0037).
Although Li further teaches user inputs/gestures for changing a speed of the automatic scrolling operation (e.g. paragraphs 0037-0038, user changing scroll rate as needed/setting scroll rate, including using an adjustable button/column; paragraph 0054, user option to adjust scroll rate; paragraph 0071, user selecting option, including scroll rate, to adjust), Osborn and Li do not explicitly disclose in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation.  However, Kin teaches in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation (e.g. col. 14 lines 3-20, detecting pinch scroll gesture, if detecting pinch scroll gesture with thumb and two fingers, speeding up rate of scroll action compared to action with just one finger; after detecting pinch scroll gesture, detecting change in pose of user’s hand while in pinched position, and increasing the rate of scrolling).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, and Kin in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) and Li (directed to presenting contents using auto scroll), to incorporate the teachings of Kin (directed to high resolution tracking and response to hand gestures through three dimensions) to include the capability, while performing automatic scrolling (as taught by Li, incorporated into the system of Osborn), to detect a pinch gesture, or a change in an existing pinch gesture (as taught by Kin, analogous to a third type of gesture as taught by Osborn, such as for selecting a displayed option), and in response, change the speed/rate of the scrolling operation (as taught by Kin, such as by pinching with an increased number of fingers, or changing a hand pose while pinching).  One of ordinary skill would have been motivated to perform such a modification in order to overcome limitations of conventional hand tracking systems, which are unable to accurately track positions of user fingers and thumbs and precise movements of user digits, causing poor representation of the user’s hands, by providing the capability to detect various gestures performed by the user’s hands and cause a change in display instructions in response to the detected gesture as described in Kin (col. 1 lines 20-43).
Assuming arguendo that Osborn, Li, and Kin do not explicitly disclose in accordance with a determination that the second hand gesture is the first type of hand gesture, pausing the automatic scrolling operation, Pantel teaches in accordance with a determination that the second hand gesture is the first type of hand gesture, pausing the automatic scrolling operation (e.g. paragraph 0204, Fig. 17, illustrating forearm-spin gesture, in which smartwatch is rotated about the x axis by twisting the forearm; paragraph 0220, forearm spin gesture can be executed in two directions, and direction can be assigned to different functions/tasks, including, at paragraph 0222, locking of the scroll capability of a page/card/tab; i.e. the user may temporarily lock/pause a scrolling operation/capability using wrist/forearm rotation in an assigned direction).  In addition, assuming arguendo that Osborn, Li, and Kin do not explicitly disclose in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation, Pantel teaches in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation (e.g. paragraph 0175, user performing drag away gesture, such as performing conventional drag gesture to scroll through a page, where user can also change direction of drag movement/page scrolling; paragraph 0204, Fig. 17, forearm spin gesture is variant of drag along/away gesture, where smart watch is rotated around axis by twisting the forearm while index finger is held in place; paragraph 0209, in forearm spin gesture, user touches screen with finger and starts to twist forearm while maintaining finger stead, resulting in dragging of the finger in manner comparable to drag along/away gestures; paragraph 0214-0215, triggering drag completed/flick based on direction/distance/speed; paragraph 0220, forearm spin gesture executed in two directions, direction of gesture assigned to different functions; i.e. the user may also utilize the forearm spin gesture in a given direction to execute functions associated with drag inputs, including causing scrolling to be performed in a given direction, such as a reversed/changed direction).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, and Pantel in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Li (directed to presenting contents using auto scroll), and Kin (directed to high resolution tracking and response to hand gestures through three dimensions), to incorporate the teachings of Pantel (directed to gesture control) to include the capability, while performing automatic scrolling (as taught by Li, incorporated into the system of Osborn), to detect a forearm spin gesture (as taught by Pantel, similar to a wrist rotation gesture as taught by Osborn) in a given direction and either lock/pause a scrolling operation (as taught by Pantel, similar to pausing scrolling according to a hand gesture as taught by Li), or reverse the direction of scrolling (as taught by Pantel, similar to controlling scrolling direction as taught by Osborn and Li).  One of ordinary skill would have been motivated to perform such a modification in order to provide novel gestures in addition to familiar touchscreen gestures, allowing access of several features or functions of an operating system or application using only a single button, icon, or control element, or global system functions using novel gestures at any location regardless of displayed content as described in Pantel (paragraph 0014).
With respect to claims 3, 25, and 43, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture (e.g. paragraph 1473-1474, Figs. 36B-C, first body state mapped to selection of next indicator in given sequence; in response to identification of first body state, deselecting indicator 35111 and selecting indicator 351113; first and second body states may represent counterparts of one another; 1475, Figs. 36D-E, first body state mapped to selection of next visual indicator in the sequence; i.e. because first body state/gesture causes selection to move to a next object in sequence, receipt of multiple instances of this gesture may also cause a different/third UI object to be selected; for example, where multiple instances/two instances of the first type of gesture are received, the object selected as indicated may be first moved from the first UI object to the second/next UI object in the sequence, and then moved again from the second/next UI object to a third/subsequent UI object in the sequence).
With respect to claims 4, 26, and 44, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
in response to detecting, via at least the optical sensor, the hand gesture (as cited above, using camera/sensors to detect hand gesture):
in accordance with a determination that the hand gesture is the fourth type of gesture, displaying, via the display generation component, a menu that includes one or more selectable options (e.g. paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user (as compared to other movements/poses causing item selection/clicking); paragraph 1485, displaying interface 351300 of Fig. 36G, which may be a popup box/dialog that appears in response to user input; i.e. where a some suitable user input, such as a fist clenching gesture, may be used to open various menus, such as a menu including selectable options).
With respect to claims 12, 32, and 50, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the fourth type of gesture does not include multiple instance of the first type of gesture and the third type of gesture does not include multiple instances of the second type of gesture (e.g. paragraph 0620, finger/hand gesture combined with force, user clenching first, etc., to open menu/dialog; paragraph 1468, finger pinch pose used to click selected item; paragraph 1381, first pose to display menu; paragraph 1474, first and second body states may be counterparts of one another; body states including wrist extension, flexion, rotation, radial deviation, ulnar deviation, etc.; i.e. where a fist clenching or other user input used for opening a menu, analogous to a fourth type of gesture does not include multiple instances of a first body state such as wrist extension, flexion, rotation, deviation, etc., analogous to a first type of gesture, and where a finger pinch pose for clicking a selected item, analogous to a third type of gesture, does not include multiple instances of a second body state such as wrist extension, flexion, rotation, deviation, etc., counterpart to a first body state, analogous to a second type of gesture ).
With respect to claims 14, 34, and 52, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the one or more selectable options includes a first selectable user interface object for changing operations that one or more hand gestures can cause to the computer system to perform, wherein selection of the first selectable user interface object causes the computer system to display a plurality of settings, wherein each setting controls an operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system (e.g. paragraph 1483, Fig. 36S, multi-state user interface configurable/programmable by user; user able to specify mappings between body states and responsive actions; such as mapping body part sates of ring/middle/pinky/index finger pinches, first pose, and open hand pose to scroll up/down, activate/deactivate link, increment/decrement selector actions; paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H; paragraph 1486, Fig. 36I, user modifying mappings between body states and actions; configuring web navigation settings by selecting setting options shown in drop-down menu of web browser; user clicking and selecting “scroll:options” to configure type of pose that initiates scrolling control or feature; paragraph 1487, Fig. 36J, link activate setting used to highlight links included in web page; user clicking on “linksActivate:options” to configure type of pose and predetermined pose duration that initiates the activation and rendering of link displayed in webpage; paragraph 1488, Fig. 36K, user clicking on “click:options” to configure type of pose and predetermined duration that initiates a click action in an application; i.e. selection of the “Control Options” dropdown object of Fig. 36G causes the plurality of settings to be shown in Fig. 36H, where these settings allow control of which gestures/poses are mapped to which operations).
With respect to claims 15, 35, and 53, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein:
the one or more selectable options includes a second selectable option for transitioning the computer system to a fourth operation mode and a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode (e.g. paragraph 1485, Fig. 36G, multi-state user interface facilitates enabling and disabling body state control using “enabled” button; drop down menu control from which user is able to select certain display settings and map body states to responsive actions; see also “reset settings” button in Fig. 36G;  i.e. where an “enabled” button can be selected to transition the system to a state/mode in which the body state controls are enabled, and to a state/mode in which body state controls are disabled, analogous to at least a second selectable option for transitioning the computer system to a fourth operation mode (i.e. a body state controls disabled mode); further where the user selects the drop down menu and modifies the mappings of body states and responsive actions, this causes the system to be transitioned from a first mode (i.e. a mode in which a current set of body state/action mappings are utilized) to a different/fifth mode (i.e. a mode in which a modified/custom set of body state/action mappings are utilized), analogous to a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode; in addition, where body state/action mappings are already customized/modified, and the user selects the option to “reset settings” this would cause the system to be transitioned from a first mode (i.e. a mode in which a current set of modified/customized body state/action mappings are utilized) to a different/fifth mode (i.e. a mode in which a default/reset set of body state/action mappings are utilized), analogous to a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode);
selection of the second selectable option causes the computer system to transition to the fourth operation mode (e.g. paragraph 1485, Fig. 36G, enabling/disabling body state control in using “enabled” button, as discussed above); and
selection of the third selectable option causes the computer system to transition to the fifth operation mode (e.g. paragraph 1485, Fig. 36G, user selecting drop down menu control to map body states to responsive actions; user selecting to “reset settings”, as discussed above).
With respect to claims 16, 36, and 54, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the one or more selectable options include a fourth selectable option for displaying one or more additional selectable options, and wherein selection of the fourth selectable option causes the computer system to display the one or more additional selectable options that were not previously displayed before selection of the fourth selectable option (e.g. paragraph 1483, Fig. 36S, multi-state user interface configurable/programmable by user; user able to specify mappings between body states and responsive actions; such as mapping body part sates of ring/middle/pinky/index finger pinches, first pose, and open hand pose to scroll up/down, activate/deactivate link, increment/decrement selector actions; paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H; paragraph 1486, Fig. 36I, user modifying mappings between body states and actions; configuring web navigation settings by selecting setting options shown in drop-down menu of web browser; user clicking and selecting “scroll:options” to configure type of pose that initiates scrolling control or feature; paragraph 1487, Fig. 36J, link activate setting used to highlight links included in web page; user clicking on “linksActivate:options” to configure type of pose and predetermined pose duration that initiates the activation and rendering of link displayed in webpage; paragraph 1488, Fig. 36K, user clicking on “click:options” to configure type of pose and predetermined duration that initiates a click action in an application; i.e. selection of the “Control Options” dropdown object of Fig. 36G, analogous to a fourth selectable option, causes additional selectable options, not previously displayed, to be displayed, as shown in Fig. 36H).
With respect to claims 19, 39, and 57, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
after displaying the user interface that includes the first user interface object, detecting, via at least the optical sensor, a fourth hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, fist clenching; paragraph 1463, Fig. 35O, fist pose; i.e. while/after the menu including the first user interface object is being/has been previously displayed, or at some later time, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the fourth hand gesture:
in accordance with a determination that the fourth hand gesture is a fifth type of gesture that is different from the first type of gesture and the second type of gesture, transitioning the computer system from an inactive state to an active state (e.g. paragraph 0509, mapping neuromuscular signals to control signals to control device based on type of gesture; paragraph 0618, determining the gesture being performed; paragraph 0620, fist clenching by user results in wake signal being sent to the system; paragraph 0703, first muscular activation state detected used to determine that a wake up mode is to be controlled, and second muscular activation state used to second activation state used to control initialization operation of the wake up mode; paragraph 0706, wake up/initialization mode; paragraph 1380, mappings causing device to perform action in response to detection of specific body part states; paragraph 1463, Fig. 35O, fist pose mapped to command to display, enter, and activate radial menu; i.e. when the system determines, based on a mapping body part states/gestures to actions, that the user is performing a fist pose hand gesture or other suitable gesture, the system may perform a corresponding action of activating a menu, or waking the system from an inactive/shut down state, analogous to transitioning the system from an inactive state to an active state).
With respect to claims 20, 40, and 58, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying, via the display generation component, the indication that the second user interface object is selected, detecting, via at least the optical sensor, a fifth hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraphs 1473-1475, first body state mapped to selection of next indicator in given sequence; first and second body states may represent counterparts; first body state mapped to selection of next visual indicator in sequence, second body state mapped to selection of previous visual indicator in sequence; i.e. while the interface/menu including the first, second, and third objects is being displayed, including after the user has performed a gesture causing a second user interface object to be selected and while the second user interface object is displayed with an indication that it is selected, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the fifth hand gesture:
in accordance with a determination that the fifth hand gesture is the first type of gesture, displaying, via the display generation component, an indication that a fourth first user interface object is selected (e.g. paragraphs 1473-1475, first body state mapped to selection of next indicator in given sequence; in response to identification of first body state, deselecting indicator 351111 and selecting indicator 351113, providing visual feedback; Figs. 36B-E; i.e., where the user has already performed the first type of gesture once, causing a second user interface object to be selected as shown in Fig. 36C, the user may perform another instance of the first type of gesture, which is mapped to selection of a next indicator, such that after this second instance of the first type of gesture is performed, a next/subsequent indicator (with respect to currently selected indicator 351113, such as the next indicator below it) will be selected, and indicated as selected via visual feedback, analogous to displaying an indication that a fourth UI object is selected in accordance with a determination that the fifth hand gesture is the first type of gesture (Examiner notes that a similar rationale would be applicable in the interfaces of Figs. 36D-E, such that if a user performs a first instance of the first type of gesture to cause item “E” to be selected as the next in sequence, and then performs a second instance of the first type of gesture, this will cause the item “F” to be selected as the next item in the sequence, and indicated as such)); and
in accordance a determination that the fifth hand gesture is the second type of gesture, displaying, via the display generation component, the indication that the first user interface object is selected (paragraphs 1473-1475, first and second body states may represent counterparts; second body state mapped to selection of previous visual indicator in sequence; i.e., where the user has already performed the first type of gesture, causing a second user interface object to be selected as shown in Fig. 36C, the user may perform the second type of gesture, which is mapped to selection of a previous indicator, such that after this second type of gesture is performed, a previous indicator (with respect to currently selected indicator 351113, such as the previous indicator above it) will be selected, and indicated as selected via visual feedback (i.e. the selection will be reverted back to the state shown in Fig. 36B), analogous to displaying an indication that the first UI object is selected in accordance with a determination that the fifth hand gesture is the second type of gesture (Examiner notes that a similar rationale would be applicable in the interfaces of Figs. 36D-E, such that if a user performs a first instance of the first type of gesture to cause item “E” to be selected as the next in sequence, and then performs the second type of gesture, this will cause the item “D” to be selected as the previous item in the sequence, and indicated as such, similar to what is shown in Fig. 36D)).
With respect to claims 21, 41, and 59, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying the user interface that includes the first user interface object, detecting a request to transition the computer system from the first operation mode to a fourth operation mode (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 1470, Fig. 35 R, open hand pose, mapped to command to deactivate the menu; i.e. while the menu is being displayed, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected, including a gesture to request transitioning from a first operation mode (such as one in which a menu is activated) to a fourth operation mode (such as one in which the menu is deactivated/disabled));
in response to detecting the request to transition the computer system from the first operation mode to the fourth operation mode, transitioning the computer system from the first operation mode to the fourth operation mode (e.g. paragraph 1461, disabled radial menu; paragraph 1471, deactivating radial menu; i.e. in response to detecting the user’s open hand gesture, the radial menu is disabled/deactivated, transitioning from an operation mode in which the radial menu is activated (analogous to a first operation mode) to an operation mode in which the radial menu is deactivated (analogous to a fourth operation mode));
while the computer system is in the fourth operation mode and while displaying the user interface that includes the first user interface object, the second user interface object, and the third user interface object, and the indication that the first user interface object is selected (e.g. paragraph 1471, state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; i.e. even after the radial menu is deactivated, all user interface objects, and selection indicator (indicating which object is selected) may continue to be displayed on the display), detecting, via at least the optical sensor, a sixth hand gesture (e.g. continuously receiving neuromuscular/musculoskeletal signals and continuously performing corresponding gesture recognition as cited above); and
in response detecting, via least the optical sensor, the sixth hand gesture:
in accordance with a determination that the sixth hand gesture is the first type of gesture, continuing to display the indication that the first user interface object is selected (e.g. paragraph 1411, user rotating arm without making fist pose has no effect on the radial menu; paragraph 1461, disable radial menu is a mode in which the user is not able to interact with the radial menu; paragraph 1470, deactivated radial menu; paragraph 1471 state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; radial menu will not recognize/respond to subsequent selection (wrist gesture) or click command until receiving a subsequent activation command); i.e. after the radial menu is deactivated, all user interface objects including the selection indicator are continuously displayed and, while in the deactivated state, the user’s hand gestures, such as a first type of gesture to select a next object, will have no effect, and will not be recognized/responded to, such that the menu will continue to be displayed including the selection indicator (indicating a current action indicator as selected) in an unchanged state); and
in accordance with a determination that the sixth hand gesture is the second type of gesture, continuing to display the indication that the first user interface object is selected (e.g. paragraph 1411, user rotating arm without making fist pose has not effect on the radial menu; paragraph 1461, disable radial menu is a mode in which the user is not able to interact with the radial menu; paragraph 1470, deactivated radial menu; paragraph 1471 state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; radial menu will not recognize/respond to subsequent selection (wrist gesture) or click command until receiving a subsequent activation command); i.e. after the radial menu is deactivated, all user interface objects including the selection indicator are continuously displayed and, while in the deactivated state, the user’s hand gestures, such as a second type of gesture to select a previous object, will have no effect, and will not be recognized/responded to, such that the menu will continue to be displayed including the selection indicator (indicating a current action indicator as selected) in an unchanged state).
Claims 10, 11, 13, 18, 30, 31, 33, 38, 48, 49, 51, and 56 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Li, further in view of Kin, further in view of Pantel, further in view of Kim et al. (US 20160029899 A1).
With respect to claims 18, 38, and 56, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying the menu that includes one or more selectable options, detecting, via at least the optical sensor, a third hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 1470, Fig. 35 R, open hand pose; i.e. while a menu is being displayed, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that the third hand gesture is a type of gesture, ceasing to display the menu that includes the one or more selectable options (e.g. paragraph 0509, mapping neuromuscular signals to control signals to control device based on type of gesture; paragraph 0618, determining the gesture being performed; paragraph 1381, open hand pose may cause the wearable device to direct the computing device to close the menu; paragraph 1380, mappings causing device to perform action in response to detection of specific body part states; paragraph 1470, open hand pose mapped to command to hide, exit, or deactivate menu; following identification of open hand pose, some or all of menu may be obscured or hidden from view within the GUI; i.e. when the system determines, based on a mapping body part states/gestures to actions, that the user is performing an open hand gesture or other suitable gesture, the system may perform a corresponding action of ceasing to display a currently displayed menu).
Osborn does not explicitly disclose that the type of gesture is a fourth type of gesture.  However, Kim teaches that the type of gesture is a fourth type of gesture (e.g. paragraph 0256, Figs. 17A-B, while message check screen outputted through display, if command for switching to lock screen is received, stopping output of message check screen and outputting/switching to lock screen; command may include first input (which is shown as a clenched fist gesture in Figs. 17A-B); i.e. where a user performing a fist gesture to initiate/switch to a lock screen display would cause the system to cease displaying all currently displayed content, including menus).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability, while displaying user interface content (i.e. such as a GUI including currently displayed menus, as taught by Osborn) to receive a hand gesture to transition to a lock screen state/mode, such as in response to a clenched fist hand gesture, and in response, cease displaying currently displayed content (such as the menu of Osborn) and display a lock screen (i.e. as taught by Kim).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claims 13, 33, and 51, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
detecting, via at least the optical sensor, a third hand gesture; and in response to detecting, via at least the optical sensor, the third hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected):
in accordance with a determination that the third gesture is the fourth type of gesture, performing an action (e.g. paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user); and
in accordance with a determination that a notification was received and in accordance with a determination that the third hand gesture is a type of gesture, performing an action related to the notification (e.g. paragraph 1649 and corresponding table, using swipe to engage/ignore notification; paragraphs 1651-1653, flick gesture may be used as dismissal mechanism for notifications; notifying user that next meeting is coming up, flick dismissing notification outright, index pinch tap engaging timer function, rolling wrist to select timer duration, etc.).
Osborn does not explicitly disclose that the determinations (for performing the action related to the notification) are that the notification was received within a threshold period of time and that the third hand gesture is the fourth type of gesture.  However, Kim teaches in accordance with a determination that a notification was received within a threshold period of time and in accordance with a determination that the third hand gesture is the fourth type of gesture, performing an action related to the notification (e.g. paragraph 0014, determining level of folding fingers of the user as being in various states (i.e. gestures) based on measured blood flow rate; paragraph 0186, Fig. 5A, user determined in sleep state, generated notifications controlled to not be outputted to user; when user wakes up, user can be informed of saved notifications; paragraph 0187, after detecting user has got out of sleep state, if prescribed command (such as making a fist twice) is received, outputting weather information through the display; paragraph 0190, if user wakes from sleep state, outputting notification screen 501 for cumulative notifications (notification list of plurality of notifications) to user, if prescribed command is received, such as input of making a fist twice, as shown in Figs. 5A-C; i.e. where a notification is received within a time period during which the user is sleeping (analogous to a threshold period of time) and then an input of making a fist gesture is received, an action related to the notification is performed, such as displaying the notification; paragraph 0227, Fig. 11A, call signal currently received, displaying call screen 1101 indicating call signal is currently being received; paragraph 0229, Figs. 11A-C, when call signal currently received, if first input maintained over prescribed time, making response to call signal, and outputting call connected screen 1103; i.e. as shown in Figs. 11A-C, if, during the time period that a call signal is being currently received and a corresponding notification screen is displayed (e.g., during the time period beginning when the call signal is first received and ending when the signal is no longer received, analogous to a threshold period of time within which a notification is received), if a user makes an input such as holding a fist for a time period, the call is answered, analogous to performing an action (answering the call) related to the notification (the screen indicating the call is being received)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to detect the fourth type of gesture (i.e. of Osborn, such as a fist gesture for causing a menu to be displayed) and, based on determining that a received gesture is the fourth type of gesture/fist gesture and further determining that a notification has been received within a threshold time period (such as during a time period in which the user was sleeping, or during a time period in which a call signal is currently received and a corresponding notification is displayed), performing a related action, such as displaying the notification, or answering a corresponding phone call, etc., as taught by Kim.  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claims 10, 30, and 48, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
in accordance with a determination that the second hand gesture is one type of gesture, displaying a notification that indicates a state of the automatic scrolling operation (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling up a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll up a GUI; paragraph 1484, ring finger pinch mapped to scroll up so that when user makes ring finger pinch pose, the page/browser displayed scrolls up; paragraph 1459, providing visual feedback of both identified pose and identified force; when action includes scrolling GUI, displaying cursor in connection with scrolling, which may be presented as a horizontal line with a bar extending above or below the line, depending on the scrolling direction; paragraph 1480, providing current state/setting of interface, current body state, and current action corresponding to current body state; paragraph 1481, graphical elements displaying current interface state, current body state, and responsive action; i.e. based on determining that the user has provided a gesture for scrolling, the system may initiate the scroll operation and may also provide a notification/indicator regarding the state of the operation, such as a cursor indicating a scrolling direction and magnitude, etc.).
Kim teaches that the one type of gesture is the first type of gesture (e.g. paragraphs 0251-254, Figs. 16A-C, outputting notification screen of received text message, receiving message check command such as folding input and outputting message check screen 1601 for displaying details of received text message; scroll action on message check screen performed by rotation of wrist; wrist rotated in first direction, scrolling to move outputted message check screen; direction for rotating wrist includes scroll direction; if scroll in bottom direction desired, rotating wrist in bottom direction; i.e. the user may provide a wrist rotation input in a first or second direction in order to perform a scroll operation in a designated direction, as shown between Fig. 16B and Fig. 16C).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to receive a hand gesture to initiate automatic scrolling in a direction, and a hand gesture to reverse the scrolling direction (i.e. such as using different fingers to perform pinch gestures to scroll up/down as taught by Kim), where the hand gesture may be a wrist rotation gesture in different directions corresponding to the desired scroll direction (i.e. as taught by Kim, where Osborn also teaches the use of wrist rotation for at least scrolling a wrist, and further teaches that the wrist rotation may be performed in two different directions, and further teaches, based on detecting the scroll gesture, outputting a notification/indicator of the scrolling operation state).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claims 11, 31, and 49, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the third type of gesture is a type of gesture that is an instance of the fourth type of gesture (e.g. paragraph 1281, fist squeeze used for discrete event such as to engage or disengage a particular feature within a given application or a continuous event, such as to maintain activation of a particular feature within a given application; paragraph 1283, selecting from presented options by performing hand gesture such as clenching the fist).
Osborn does not teach that the third type of gesture is multiple instances of the fourth type of gesture.  However, Kim teaches the third type of gesture is multiple instances of the fourth type of gesture (e.g. paragraph 0187, user command of making fist twice to cause output of weather information; i.e. as shown in Fig. 5A, where a UI element associated with a weather information check is currently displayed, the user may perform a gesture which is multiple instances of a fist clenching gesture in order to select/display content corresponding to the UI element).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to receive a hand gesture to select a displayed item on the screen(i.e. such as a fist clench gesture as taught by Osborn) as multiple instances of the hand gesture (i.e. such as receiving multiple fist clench gestures as taught by Kim).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
Claims 2, 24, and 42 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Li, further in view of Kin, further in view of Pantel, further in view of Chung et al. (US 20220326779 A1).
With respect to claims 2, 24, and 42, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 1, 22, and 23 as previously discussed.  Although Osborn teaches that auxiliary sensors which may contribute to gesture detection may include heart rate sensors, PPG sensors, and cameras used to detect changes in blood flow within the user (e.g. paragraph 0281, sensors including cardio sensor to measure user’s pulse, heart rate; paragraph 0286, auxiliary device/sensor including heart-rate monitor; paragraph 0310, nonvisible light used to capture image of heat distribution, providing indication of blood flow, used to infer condition of user such as forces being exerted by certain fingers; paragraph 0729, PPG sensors detecting vascular changes), Osborn does not explicitly disclose wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor.
However, Chung teaches wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor (e.g. paragraph 0093, sensor module sensing information for movement of electronic device, and may include photoplethysmogram (PPG) sensor; paragraph 0098, recognizing OCO and shake gestures, gestures of turning hand, moving hand with palm open, lifting arm, moving thumb, moving up/down hand, etc.; gesture may be recognized based on user’s biometric signal (e.g. a signal indicating relaxation/contraction of blood vessels or a heartbeat signal); paragraph 0104, gesture application invoking sensor API to activate sensors, such as PPG sensor; gesture application identifying information for gestures; paragraph 0136, OCO gesture, PPG sensor detecting change due to relaxation and contraction of blood vessels when user opens and closes her hand; ).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Chung in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Chung (directed to recognizing gestures) to include the capability to detect the hand gesture based on heart rate data detected via the optical sensor, such as by detecting a variety of gestures using a PPG sensor, based on a user biometric signal indicating relaxation/contraction of blood vessels or heartbeat (as taught by Chung).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability to recognize a gesture when there is a gesture recognition request from a specific application, eliminating the need for sensors to stay on and the gesture recognition process to keep on running, reducing power consumption and resource waste, and in order to make it possible to immediately recognize relatively small gestures, as described in Chung (paragraphs 0014-0015).
Claims 8, 28, and 46 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Li, further in view of Kin, further in view of Pantel, further in view of Platzer et al. (US 20130239046 A1).
With respect to claims 8, 28, and 46, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn and Li further teach wherein:
the automatic scrolling operation scrolls a sequence of a plurality of interface objects in a second direction (e.g. Osborn paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling down a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll down a GUI; paragraph 1484, pinky finger pinch mapped to scroll down so that when user makes pinky finger pinch pose, the page/browser displayed scrolls down; Li paragraph 0038, user selecting scroll options, including scroll direction; paragraph 0044-0045, scroll direction may be reversed using gesture, such as swipe on touch screen; via buttons, etc.).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Kin, Pantel, and Li in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Li (directed to presenting contents using auto scroll) to include the capability to perform the automatic scrolling operation in a second direction (i.e. in a specified direction, as taught by Li).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability for the user to relax and watch contents effortlessly, as described in Li (paragraph 0037).
Osborn does not explicitly disclose wherein the one or more programs further includes instructions for:
while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects; and
in response to detecting the end of the sequence of the plurality of user interface objects, performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction.
However, Platzer teaches wherein the one or more programs further includes instructions for:
while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects (e.g. paragraph 0077, Figs. 5A-C, scrolling list of items to terminus of the list; paragraph 0078, terminus is edge of scrolled region; while list is being scrolled; terminus is virtual boundary associated with displayed objects; displayed objects reach or intersect with the terminus 514; i.e. the system detects the end of a list of displayed UI objects, as indicated by a terminus at the edge of the scrolled list); and
in response to detecting the end of the sequence of the plurality of user interface objects, performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction (e.g. paragraph 0077, Figs. 5A-C, reaching terminus of list, smoothly bouncing displayed items at end of list, reversing direction; paragraph 0078, as illustrated in Fig. 5B, when objects reach or intersect with the terminus 514, the movement corresponding to scrolling may stop and, as illustrated in Fig. 5C, subsequently reverse direction; i.e. scrolling list reaches and bounces at its terminus; i.e. in response to reaching/detecting the terminus while scrolling in a second direction (such as opposite of a first direction), the system automatically performs a “bounce” by scrolling the list of displayed objects in the opposite direction (which may be a third direction, which may be the same as a first direction)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Platzer in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects, such as scrolling menus/lists of UI objects, webpages, etc.), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Platzer (directed to scrolling operations) to include the capability to, while scrolling user interface content, including a plurality of objects, in a given direction (i.e. as taught by Osborn, Kim, and Platzer, such as in a second direction which is opposite of a first scroll direction, or some other direction, where at least Jose teaches a scrolling direction may be set as left-right, right-left, up-down, or down-up), detect a terminus indicating the end of the plurality of objects and, in response automatically scroll the content including the plurality of objects in a reverse direction (as taught by Platzer, where this reverse direction may be a third direction which is different from the second direction, but potentially the same as a first direction).  One of ordinary skill would have been motivated to perform such a modification in order to provide operations desired by users, such as scrolling operations on a device, including on device having limited display size, user interface, etc. which limits the ease of use of the device, as described in Platzer (paragraphs 0006, 0008).
Claims 17, 37, and 55 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Li, further in view of Kin, further in view of Pantel, further in view of Saurabh et al. (US 20180181261 A1).
With respect to claims 17, 37, and 55, Osborn in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed.  Osborn does not explicitly disclose wherein the menu is displayed:
in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location; and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location.
However, Saurabh teaches wherein the menu is displayed:
in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location (e.g. abstract, positioning popup interface within display frame in relation to corresponding target element; relative positioning of popup interface to associated target element dynamically selected and determined based on location of corresponding target element within quadrants/subquandrants of display frame; paragraph 0001, presenting additional information/menu in the form of a popup interface; paragraph 0004, rendering popup interface at position determined to be best fit position with regard to target element’s position; paragraph 0005, determining dynamically made where to display the popup interface by determining quadrant the target element is actually displayed within; paragraph 0027, pop interface may refer to a context menu; paragraph 0028, target element linked to popup interface; target element may be word or icon, and popup interface may be a selection menu related to the word or icon; paragraph 0077, Fig. 8, target element 802 residing within subquadrant 5/quadrant 1, resulting popup interface 804 displayed at bottom position relative to target element; i.e. where a respective user interface element (target element) is displayed at a position within a first quadrant (analogous to a first location), the corresponding popup menu is also displayed at a position within the same quadrant/location); and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location (e.g. abstract, positioning popup interface within display frame in relation to corresponding target element; relative positioning of popup interface to associated target element dynamically selected and determined based on location of corresponding target element within quadrants/subquandrants of display frame; location of popup interface is dynamically adjustable as opposed to being positioned based on a predetermined location; paragraph 0004, rendering popup interface at position determined to be best fit position with regard to target element’s position, event when that target element changes position or otherwise moves; paragraph 0005, determining dynamically made where to display the popup interface by determining quadrant the target element is actually displayed within; as the target element moves within the user interface (due to scrolling etc.) the relative position of the popup interface is dynamically adjusted so that the popup interface is continuously displayed in a position identified as best fit relative to target element; paragraph 0083, Fig. 10, target element 1002 displayed spanning quadrant1 and quadrant2, resulting popup interface 1004 displayed in bottom position relative to target element 1002; i.e., where a respective user interface element is displayed at a different location such as a location spanning multiple quadrants (analogous to not being displayed at a first location), the corresponding popup menu may also be moved/repositioned on the display so that it is also displayed at a second location which is different from the first location).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Saurabh in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects, including displaying associated menus), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Saurabh (directed to a positioning mechanism for bubble as a custom tooltip, such as positioning popup interfaces/menus in relation to a corresponding target element) to include the capability to dynamically display the menu (i.e. of Osborn) in a position relative to the position of a corresponding UI object such that when the UI object is displayed at a first location, the menu is also displayed at the first location and when the UI object is not displayed at the first location, such as because it is moved to a different location, the menu is displayed at a second location different from the first location, such as at the different location that the UI object has been moved to (as taught by Saurabh).  One of ordinary skill would have been motivated to perform such a modification in order to solve disadvantages found in existing popup interfaces by providing the capability to dynamically render a popup interface at a position that is determined to be a best fit position with regard to a target elements position, event when that target element changes position or otherwise moves in some manner, as described in Saurabh (paragraphs 0002-0004).
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain,” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting in re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (GCPA 1968)). Further, a reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill the art, including nonpreferred embodiments. Merck & Co, v. Biocraft Laboratories, 874 F.2d 804, 10 USPQ2d 1843 (Fed. Cir.), cert, denied, 493 U.S. 975 (1989). See also Upsher-Smith Labs. v. Pamlab, LLC, 412 F,3d 1319, 1323, 75 USPQ2d 1213, 1215 (Fed. Cir, 2005): Celeritas Technologies Ltd. v. Rockwell International Corp., 150 F.3d 1354, 1361, 47 USPQ2d 1516, 1522-23 (Fed. Cir. 1998).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Yoon et al. (US 20150061842 A1) teaches various gestures for controlling a mobile device such as a wrist worn device (Figs. 7-33).
Offenberg et al (US 20150248235 A1) teaches various gestures for controlling a mobile device such as a wrist worn device (Figs. 27-92), as well as for selecting among displayed items with corresponding visual indications, scrolling, etc. (Figs. 102A-D).
Hazra et al. (US 20170123487 A1) teaches various gestures for controlling a mobile device including a wrist worn device (Figs. 5, 8-15).

APPLICANT REBUTTAL: REMARKS
Claims 1-4, 8, 10-26, 28, 30-44, 46, and 48-59 were previously pending. By virtue of this
amendment, claims 1, 3, 4, 8, 10, 11, 14-18, 22, 23, 25, 26, 28, 30, 31, 34-38, 43, 44, 46, 48, 49, and 52-56 have been amended. The amendments to independent claims 1, 22, and 23 are supported by at least paragraph [0205] and FIG. 14 of the Application as-filed. The amendments to the dependent claims are for providing consistency across the claims due to the amendment to claims 1, 22, and 23 and are fully supported by the Application as-filed. Thus, no new matter has been added. Claims 1-4, 8, 10-26, 28, 30-44, 46, and 48-59 remain pending.
Applicant respectfully requests reconsideration of this application based on the foregoing amendments and the following remarks.
Examiner Interview
Applicant thanks Examiner Stanley for the telephonic interview on January 16, 2024 with Applicant's representatives Horace Zhang and Christopher Eide. During the interview, the attendees discussed the 103 rejection of claim 1 over Osborn, Li, Kin, and Pantel. In particular, Applicant's representatives proposed amending the independent claims as indicated above. The Examiner indicated that the amendment likely overcomes the present 103 rejection of the independent claims, but that further search and consideration would be required to determine allowability.
The attendees further discussed the 103 rejection of claim 3 over Osborn, Li, Kin, and Pantel and the 103 rejection of claim 11 over Osborn, Li, Kin, Pantel, and Kim. In particular, Applicant's representatives explained that the respective cited references fail to teach or suggest claims 3 and 11. The Examiner indicated that he understood Applicant's explanation and that further
consideration of claims 3 and 11 would be required. (Applicant Initiated Interview Summary dated January 23, 2024.)
Claim Re
j
ections 35 USC 103
Independent Claims
Claims 1, 3, 4, 12, 14-16, 19-23, 25-27, 32, 34-36, 39-41, 43-45, 50, 52-54, 57, and 58 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn et al. (US 20230072423 Al) in view of Li (US 20160328108 Al), further in view of Kin (US 10261595 81), further in view of Pantel (US 20140320434 Al).
Applicant has amended independent claim 1 (and similarly independent claims 22 and 23) to recite, inter alia:
in accordance with a determination that the hand gesture is a fourth type of gesture that is different from the first type of gesture, the second type of gesture, and the third type of gesture, displaying, via the display generation component, a menu, wherein: a first control and a second control are available via the menu; selection of the first control causes the computer system to perform an operation corresponding to a press of the first hardware input mechanism; and selection of the second control causes the computer system to perform an operation corresponding to a rotation of the second hardware input mechanism.
Amended claims 1, 22, and 23 (and all claims depending therefrom) are allowable over the cited references for at least the reasons discussed below.
As an initial matter, the Examiner agreed during the interview that the amendment would overcome the present 103 rejection. For this reason alone, Applicant respectfully requests that the 103 rejection of claims 1, 22, and 23 (and all claims depending therefrom) be withdrawn and the claims be allowed.
Further, the cited references fail to teach or suggest the amended limitations of claims 1, 22, and 23. Specifically, while Osborn teaches displaying a menu in response to detecting a hand
gesture, Osborn fails to teach or suggest that the menu includes controls to perform respective operations that correspond to pressing a hardware input device and rotating a hardware input device. Li, Kin, and Pantel fail to cure the deficiencies of Osborn.
In view of the above, Applicant respectfully requests that the 103 rejection of claims 1, 22,
and 23 (and all claims depending therefrom) be withdrawn and the claims be allowed.
Claims 3
,
,
,
,
,
and 49
Claims 3 and 11 depend from claim 4. Claims 3, 4, and 11 are reproduced below.
4. The computer system of claim 1, wherein before detecting the hand gesture, the computer system is in a first operation mode, wherein performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode, wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture, and wherein the one or more programs further include instructions for:
while the computer system is in the second operation mode, detecting, via at least the optical sensor, a second hand gesture; and
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation; in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation; in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation; and in accordance with a determination that the second hand gesture is the fourth type of gesture, ending the automatic scrolling operation.
3. The computer system of claim 4, wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture.
11. The computer system of claim 4, wherein the third type of gesture is a type of gesture that is multiple instances of the fourth type of gesture.
Accordingly, claim 3 specifies that the computer system detects the "first type of gesture" to pause autoscroll and detects multiple instances of the "first type of gesture" to reverse the autoscroll direction.
Similarly, claim 11 specifies that computer system detects the "third type of gesture" to
change the speed of autoscrolling and detects multiple instances of the "third type of gesture" to end autoscrolling.
Claims 3, 11, 25, 31, 43, and 49 are allowable for at least the reasons discussed below.
Initially, claims 3, 11, 25, 31, 43, and 49 depend variously from independent claims 1, 22, and 23, which are allowable for at least the reasons discussed above. Thus, claims 3, 11, 25, 31, 43, and 49 are allowable for at least the reason that they each depend from a respective allowable independent claim. Claims 3, 11, 25, 31, 43, and 49 are allowable for at least the further reasons discussed below.
Claims 3, 25, and 43 stand rejected under 103 over Osborn, Li, Kin, and Pantel. The cited portions of Osborn teach that a gesture navigates by one unit (e.g., moves forward by one). This, according to the Examiner, suggests that multiple instances of the gesture navigate by multiple units.
However, none of Osborn, Li, Kin, or Pantel teach that that the gesture that reverses autoscroll direction is multiple instances ofthe gesture that pauses autoscroll.
Claims 11, 31, and 49 stand rejected under 103 over Osborn, Li, Kin, Pantel, and Kim.
Specifically, the Office Action relies on Osborn and Kim for allegedly teaching claim 11. The cited portions of Osborn teach using a fist squeeze gesture to engage (e.g., select) an element or to maintain activation of an element. The cited portions of Kim teach using a double fist squeeze gesture to display information.
However, none of Osborn, Li, Kin, Pantel, or Kim teach or suggest that the gesture that ends autoscrolling is multiple instances ofthe gesture that changes the speed of autoscrolling.
Applicant thus respectfully requests the 103 rejection of claims 3, 11, 25, 31, 43, and 49 be withdrawn and the claims be allowed.
Remainin
g
Claims
Claims 10, 11, 13, 18, 30, 31, 33, 38, 48, 49, 51, and 56 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Li, further in view of Kin, further in view of Pantel, further in view of Kim et al. (US 20160029899 Al ).
Claims 10, 11, 13, 18, 30, 31, 33, 38, 48, 49, 51, and 56 depend variously from independent claims 1, 22, and 23, which are allowable for at least the reasons discussed above. Thus, claims 10, 11, 13, 18, 30, 31, 33, 38, 48, 49, 51, and 56 are allowable for at least the reason that they each depend from a respective allowable independent claim. The addition of Kim fails to cure the deficiencies of Osborn, Li, Kin, and Pantel described above, nor is it alleged to. Applicant thus respectfully requests the 103 rejection of claims 10, 11, 13, 18, 30, 31, 33, 38, 48, 49, 51, and 56 be withdrawn and the claims be allowed.
Claims 2, 24, and 42 are rejected under 35 U.S.C.103 as being unpatentable over Osborn in view of Li, further in view of Kin, further in view of Pantel, further in view of Chung et al. (US 20220326779 Al).
Claims 2, 24, and 42 depend variously from independent claims 1, 22, and 23, which are allowable for at least the reasons discussed above. Thus, claims 2, 24, and 42 are allowable for at least the reason that they each depend from a respective allowable independent claim. The addition of Chung fails to cure the deficiencies of Osborn, Li, Kin, and Pantel described above, nor is it alleged to. Applicant thus respectfully requests the 103 rejection of claims 2, 24, and 42 be withdrawn and the claims be allowed.
Claims 8, 28, and 46 are rejected under 35 U.S.C.103 as being unpatentable over Osborn in view of Li, further in view of Kin, further in view of Pantel, further in view of Platzer et al. (US 20130239046 Al).
Claims 8, 28, and 46 depend variously from independent claims 1, 22, and 23, which are allowable for at least the reasons discussed above. Thus, claims 8, 28, and 46 are allowable for at least the reason that they each depend from a respective allowable independent claim. The addition of Platzer fails to cure the deficiencies of Osborn, Li, Kin, and Pantel described above, nor is it alleged to. Applicant thus respectfully requests the 103 rejection of claims 8, 28, and 46 be withdrawn and the claims be allowed.
Claims 17, 37, and 55 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Li, further in view of Kin, further in view of Pantel, further in view of Saurabh et al. (US 20180181261 Al).
Claims 17, 37, and 55 depend variously from independent claims 1, 22, and 23, which are allowable for at least the reasons discussed above. Thus, claims 17, 37, and 55 are allowable for at least the reason that they each depend from a respective allowable independent claim. The addition of Saurabh fails to cure the deficiencies of Osborn, Li, Kin, and Pantel described above, nor is it alleged to. Applicant thus respectfully requests the 103 rejection of claims 17, 37, and 55 be withdrawn and the claims be allowed.
CONCLUSION

NEXT OFFICE ACTION: DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the Amendment filed on January 30, 2024.  Claims 1, 3, 4, 8, 10, 11, 14-18, 22, 23, 25, 26, 28, 30, 31, 34-38, 43, 44, 46, 48, 49, and 52-56 are amended.  Claims 5, 6, 7, 9, 27, 29, 45, and 47 are cancelled.  Therefore, Claims 1-4 8, 10-26, 28, 30-44, 46, and 48-59 are pending in the case.  Claims 1, 22, and 23 are the independent claims.
This action is final.
Applicant’s Response
In the Amendment filed on January 30, 2024, Applicant amended the claims and provided arguments in response to rejection of the claims under 35 USC 103 in the previous office action.
Response to Argument/Amendment
Applicant’s amendments to the claims in response to the rejection of the claims under 35 USC 103 in the previous office action are acknowledged, and Applicant’s associated arguments have been fully considered.
Regarding the independent claims, Applicant argues that the cited references (Osborn, Li, Kin, and Pantel) fail to teach or suggest displaying a menu that includes controls to perform respective operations that correspond to pressing a hardware input device and rotating a hardware input device, as recited in the amended claims.  Applicant’s arguments are persuasive, and the rejection is withdrawn.
Regarding claims 3, 25, and 43, Applicant argues that Osborn, Li, Kin, and Pantel do not teach that the gesture that reverses autoscroll direction is multiple instances of the gesture that pauses autoscroll.  The claims recite “wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture.”  As cited in the previous office action, Osborn does appear to teach this limitation as recited with respect to at least the first operation mode recited in claim 4 (i.e. receiving a single instance of a gesture in order to display an indication that a second object is selected and receiving multiple instances of the gesture in order to display an indication that a third object is selected).  However, Examiner agrees that the cited references do not appear to teach this limitation with respect to the second operation mode recited in claim 4 (i.e. as argued by Applicant, where the second type of gesture, used to reverse autoscroll direction, is multiple instances of the first type of gesture, used to pause autoscroll).  Given that the claims recite first and second types of gestures (i.e. corresponding to classes/categories of gestures, and not necessarily to any specific gesture within the type/class/category), and further recite that these are used for different purposes in different modes, it is not entirely clear that the claims, under the broadest reasonable interpretation, require the elements specifically argued by applicant (i.e. that the specific gesture (as opposed to type of gesture) that reverses autoscroll must be multiple instances of the specific gesture that pauses autoscroll, where these specific gestures have this same one-to-multiple relationship across both recited operation modes).  However, this argument is moot in view of the new grounds of rejection provided below.
Regarding claims 11, 31, and 49, Applicant argues that Osborn, Li, Kin, Pantel, and Kim do not teach that the gesture that ends autoscrolling is multiple instances of the gesture that changes the speed of autoscrolling.  First, the claims actually recite “wherein the third type of gesture is a type of gesture that is multiple instances of the fourth type of gesture.”  Examiner respectfully submits that this appears to require the opposite of what is argued by Applicant, i.e. appears to require that the gesture that changes the speed of autoscrolling (i.e. the third gesture in the second mode as recited in claim 4) is multiple instances of the gesture that ends autoscrolling (i.e. the fourth gesture in the second mode as recited in claim 4).  Assuming that this understanding is correct, Applicant’s argument cannot be persuasive, as it is directed to limitations not present in the claims.  In addition, similar to claims  3, 25, and 43, given that the claims recite third and fourth types of gestures (i.e. corresponding to classes/categories of gestures, and not necessarily to any specific gesture within the type/class/category), and further recite that these are used for different purposes in different modes, it is not entirely clear that the claims, under the broadest reasonable interpretation, require the elements specifically argued by applicant (i.e. that the specific gesture (as opposed to type of gesture) that ends autoscrolling must be multiple instances of the specific gesture that changes the speed of autoscrolling, as argued by Applicant, or the reverse situation discussed above in which the specific gesture that changes the speed of autoscrolling must be multiple instances of the specific gesture that ends autoscrolling, where these specific gestures have this same one-to-multiple relationship across both recited operation modes).  However, this argument is moot in view of the new grounds of rejection provided below.
As noted above, Applicant’s arguments are persuasive, and the rejection is withdrawn.  However, new grounds of rejection are provided below.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims under pre-AIA  35 U.S.C. 103(a), the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were made absent any evidence to the contrary.  Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and invention dates of each claim that was not commonly owned at the time a later invention was made in order for the examiner to consider the applicability of pre-AIA  35 U.S.C. 103(c) and potential pre-AIA  35 U.S.C. 102€, (f) or (g) prior art under pre-AIA  35 U.S.C. 103(a).
Claims 1, 12, 14, 15, 16, 19, 20, 21, 22, 23, 32, 34, 35, 36, 39, 40, 41, 50, 52, 53, 54, 57, 58, and 59 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn et al. (US 20230072423 A1) in view of Fleizach et al. (US 21020306748 A1).
With respect to claims 1, 22, and 23, Osborn teaches a computer system that is in communication with a display generation component and an optical sensor, the computer system comprising: a first hardware input mechanism (e.g. paragraph 0386, vibrotactile system having touch-sensitive pads, pressure sensors, on/off button, etc.; paragraph 1605, text entry control signal via physical or electrical button, switch, touchscreen, etc.); a second hardware input mechanism (e.g. paragraph 0278, system including IMUs such as accelerometer, gyroscope, magnetometer, etc.; paragraph 0352, sensors including accelerometers, gyroscopes, magnetometers, etc.; paragraph 0386, vibrotactile system having motion/position sensors); one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method; a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system that is in communication with a display generation component and an optical sensor, the one or more programs including instructions for performing the method (e.g. paragraph 0257, supplementing sensor data with images of user captured by camera; paragraph 0286, auxiliary devices/sensors including imaging devices/cameras, heart rate monitors, etc.; paragraph 0288, Figs. 9A-B, wearable system with sensors such as EMG sensors, sensors 810; computer processor such as microcontroller 934 receiving inputs from sensors; communicating with dongle portion 920 and outputting processing output to dongle portion; paragraph 0289, dongle portion 920 receiving signals from wearable portion 910, and providing them to host computer for further processing, display, effecting control of objects, etc.; paragraph 0292, describing Fig. 1, system including processors 112 programmed to communicate with sensors 110 and executing algorithms; paragraph 0301, system including display controller configured to display visual representations; computer processors implementing trained inference models that receive sensor signals as input and provide information to generate control signals for system as output; paragraph 0302, user interface; paragraph 0303, various types of devices in which processors may be included; paragraph 0304, multiple processors; paragraph 0306, system including processors, camera, and display; paragraph 310, camera information/cameras; paragraph 0311, camera mounted on user; paragraph 0344, embodiments implemented in software executed on processors; paragraph 0345, non-transitory computer-readable storage medium/memory encoded with computer program/instructions executed on processor to perform described functions; paragraph 0352, sensor including light emitter/detector; paragraph 0385, Fig. 13, vibrotactile system 1300 may be standalone system, or may be configured with interaction with various types of other devices, and includes processor 1360; paragraph 0610, cameras; paragraph 0729, photoplethysmography/PPG sensors detecting vascular changes/changes in blood volume; paragraph 1384, Fig. 35B, wearable device 35102 capable of controlling computer devices via neuromuscular signals of users, including set of sensors 35104 that detect/sense neuromuscular signals traversing body of user; processing device 35106 communicatively coupled to sensors, and memory 35108 including models and computer-executable instructions that when executed by processor cause it to identify force generated, generate commands for HCI system, etc.; i.e. the device includes at least one processor and memory storing instructions for execution by the processor, and is further in communication with at least one display/display controller/generation component, and at least one optical sensor, such as a camera (which may be mounted on the user)); and the method, comprising:
displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected (e.g. paragraph 1460, describing radial menu 351000 in Figs. 35O-V, where the menu may include visual indicators including a selection indicator 351020; paragraph 1461, selection indicator 351020 may indicate a currently selected action indicator; paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual, auditory, or haptic feedback; visual feedback representing change of a characteristic of visual element within radial menu in response to selection of the visual indicator, such as position change, orientation change, color change, size change, transparency change, fill change, emphasis change, shadow change, animation change, font change, line type change, line width change, combinations, etc.; paragraph 1473, describing Figs. 36B-C, radial menu 351110 including a set of action indicators including indicators 351111 and 351113 displayed in GUI along arc, where as shown in Fig. 36B includes an indication that indicator 35111 is selected; demonstrating/confirming selection of visual indicator using visual feedback; paragraph 1475, describing Figs. 36D-E, displaying sequential menu 351120 including a sequence of visual indicators depicted as A through F, where as shown in Fig. 36 D includes in indication that indicator D/351123 is selected; i.e. both of the menus/interfaces shown in the cited figures include at least first, second, and third UI objects along with an indication that an object of these is selected, such as a visual feedback including a shading/coloration of a selected indicator; paragraphs 1385-1387, determining based on neuromuscular signals state of body part of user and generating input commands accounting for state of user’s body part; sensors constitute any type of form of sensor capable of detecting/sensing neuromuscular signals in user’s body including EMG sensors, MMG sensors, SMG sensors; sensors may also include auxiliary sensors such as IMU, position tracking, imaging (camera), and radiation based sensors, microphones, heart rate monitors, etc.; paragraph 1394, wearable device 35102 coupled with computing system/display; user placing body in state/condition to control/modify presentation/performance of computing system, generating neuromuscular signals representative/indicative/suggestive of state/action);
while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture (e.g. paragraph 0310, camera information used to improve interpretation of neuromuscular signals; camera used to record image of user, may be within particular range/wavelengths of light, images produced by visible/non-visible light, including showing heat distribution in user’s body, providing indication of blood flow within user and used to infer condition of user such as forces being exerted by fingers; paragraph 0311, camera mounted on user; paragraph 0312, camera data in addition to neuromuscular activity signals used to reconstruct positions, geometries, and forces being applied by user’s arm/hand; paragraph 0323, auxiliary signals from auxiliary devices such as camera supplementing neuromuscular signals; paragraph 0610, cameras used to improve interpretation of signals; paragraph 0611, camera data used to provide real time determination of musculoskeletal representations of gestures; paragraph 0729, PPG sensors detecting vascular changes; paragraph 1271, camera/imaging data; paragraph 1387, sensors including auxiliary sensors such as cameras; paragraph 1395, user pose/gesture generating neuromuscular signals as they traverse down arm toward hand; sensors 35104 detecting/sensing neuromuscular signals and generating data representative of the signals; paragraph 1396-1398, sensor data undergoing processing, analyzing data to determine state of body parts of user, producing/outputting classification indicating state of body part; paragraph 1398, gestures; paragraph 1400, causing computing system to perform actions mapped to state of body parts; paragraph 1401, describing various actions mapped to different states of user’s body parts; paragraph 1444, capturing gestures with camera; photographic data of hand gestures; paragraph 1473, body state mapped to selection of next indicator in a given sequence; identification of body state; paragraph 1474, body states including wrist extension, flexion, rotation, radial deviation, ulnar deviation, etc.; paragraph 1475, body states mapped to selection of visual indicators in sequence; identification of body states; i.e. detecting a body state of the user, such as a hand gesture, including by using an optical sensor such as a camera,  PPG, etc., possibly in combination with other sensors),
in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is a first type of gesture, displaying, via the display generation component, an indication that the second user interface object is selected (e.g. paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual feedback; paragraph 1473, Figs. 36B-C, first body state mapped to selection of the next indicator in a given sequence; in response to identification of first body state, deselecting indicator 35111 and selecting indicator 351113; demonstrating/confirming selection of visual indicator using visual feedback; paragraph 1475, Figs. 36D-E, first body state mapped to selection of next visual indicator in the sequence; i.e. in response to detecting a first type of gesture, as indicated by a first body state, the indication that a first UI object is selected may be changed to an indication that a second UI object is selected, such as by moving the indicator of the selection to the second UI object, where the second UI object is a next UI object in a sequence with respect to the first UI object); and
in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected (e.g. paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual feedback; paragraph 1473-1474, Figs. 36B-C, first body state mapped to selection of next indicator in given sequence; demonstrating/confirming selection of visual indicator using visual feedback; first and second body states may represent counterparts of one another; 1475, Figs. 36D-E, second body state mapped to selection of previous visual indicator in the sequence; in response to identification of second body state, deselecting subsequent indicator 351123 and selecting prior indicator 351121; i.e. in response to detecting a second/different type of gesture, as indicated by a second body state, the indication that a first UI object is selected may be changed to an indication that a third UI object is selected, such as by moving the indicator of the selection to the third UI object, where the third UI object is a previous UI object in a sequence with respect to the first UI object; in addition, because these gestures cause selection to move to a next/previous object in sequence, receipt of multiple of either of these gestures may also cause a different/third UI object to be selected; for example, where multiple instances/two instances of the first type of gesture are received, the object selected as indicated may be first moved from the first UI object to the second/next UI object in the sequence, and then moved again from the second/next UI object to a third/subsequent UI object in the sequence); and
in accordance with a determination that the hand gesture is a third type of gesture that is different from the first type of gesture and the second type of gesture, performing an operation corresponding to selection of the first user interface object (e.g. paragraph 1281, fist squeeze used for discrete event such as to engage or disengage a particular feature within a given application or a continuous event, such as to maintain activation of a particular feature within a given application; paragraph 1283, selecting from presented options by performing hand gesture such as clenching the fist; paragraph 1468, Fig. 35Q, illustrating finger pinch pose 351006, which may be mapped to command/instruction to click visual indicator in the radial menu, analogous to mouse button click or keypress; alternatively combination of finger pinch pose and certain degree of force mapped to click command; paragraph 1469, Fig. 35 U, clicking/engaging with or activating function of the currently selected action indicator; clicking action indicator in response to identified click gesture; upon performing the click action, causing visual indicator to appear to depress and/or release, similar to a physical button; i.e. clicking the currently selected action indicator causes the corresponding function to be activated); and
in accordance with a determination that the hand gesture is a fourth type of gesture that is different from the first type of gesture, the second type of gesture, and the third type of gesture, displaying, via the display generation component, a menu (e.g. paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user (as compared to other movements/poses causing item selection/clicking); paragraph 1485, displaying interface 351300 of Fig. 36G, which may be a popup box/dialog that appears in response to user input; i.e. where a some suitable user input, such as a fist clenching gesture, may be used to open various menus, such as a menu including selectable options).
Osborn does not explicitly disclose wherein:
a first control and a second control are available via the menu;
selection of the first control causes the computer system to perform an operation corresponding to a press of the first hardware input mechanism; and
selection of the second control causes the computer system to perform an operation corresponding to rotation of the second hardware input mechanism.
However, Fleizach teaches wherein:
a first control and a second control are available via the menu (e.g. paragraph 0232-0233, Fig. 5B, menu 508 displayed in response to predefined input, and includes plurality of icons 510 including virtual touches selection 510-1, virtual gestures 510-2, virtual device 510-3, and home button 510-4; paragraph 0248, Fig. 5Q, menu of virtual device operations representing plurality of icons representing types of virtual device operations);
selection of the first control causes the computer system to perform an operation corresponding to a press of the first hardware input mechanism (e.g. paragraph 0234-0247, Figs. 5C-O, virtual touches icon 510-1 selected, number of contacts selected, performing gestures on device based on selected number of contacts; virtual gestures icon 510-2 selected, respective gesture is selected, corresponding gesture performed on device; paragraph 0247, Fig. 5P-, virtual device icon 510-3 selected; paragraph 0248, Fig. 5Q, virtual device icons for muting/unmuting 526-1, device lock 526-4, volume up 526-5, volume down 526-6; virtual device operations including operations usually performed via physical controls on the device, including muting and unmuting the ringer and sound effects on the device, which is normally performed via a mute switch; locking the device, which is normally initiated via a lock/unlock switch; increase volume, which is normally performed via a volume up button; decrease volume, which is normally performed via a volume down button; paragraph 0251-0252, Figs. 5T-U, home button icon 510-4 selected, and home screen displayed in response; i.e. selection of a touch-related, home screen, or other device control menu option causes performance of a device function which corresponds to a hardware input mechanism press, such as pressing a volume button, mute or lock switch, home screen button, etc.); and
selection of the second control causes the computer system to perform an operation corresponding to rotation of the second hardware input mechanism (e.g. paragraph 0248, Fig. 5Q, virtual device rotation icon 526-2 selected; virtual device operations including operations usually performed via physical controls on the device, including shaking device to perform predefined operation, rotating device to rotate display orientation, which are typically detected via accelerometer in the device; paragraph 0249-0250, Fig. 5R-S, icons 530 representing types of virtual device orientations, or rotations of device orientation; user interface rotated 90 degrees clockwise in response to selection of rotate-left orientation icon 530-2, without actual physical rotation of the device; i.e. selection of a rotation-related menu option causes performance of a device function which corresponds to a hardware input mechanism rotation (such as rotation of a device accelerometer through space), such as a 90 degree (or greater rotation of the device)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Fleizach in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) to incorporate the teachings of Fleizach (directed to providing control of a touch-based user interface absent physical touch capabilities) to include the capability to display, via the menu (i.e. of Osborn), a first control which, when selected, causes the device to perform operations corresponding to pressing of a hardware input mechanism (i.e. such as pressing of a physical button or touching of a touchscreen or touch-sensitive surface, to perform an associated function, such as performing touch screen selections and/or gestures, or a function associated with a physical button, such as a home screen function, screen lock function, volume/mute control functions, etc.) and a second control which, when selected, causes the device to perform operations corresponding to rotation of a hardware input mechanism (i.e. such as rotation of one or more accelerometers of the device in space, to perform an associated function, such as altering a display orientation), as taught by Fleizach.  One of ordinary skill would have been motivated to perform such a modification in order to provide electronic devices with touch-based user interfaces with methods and interfaces that make these devices accessible to users without fine motor skills in their hands, as described in Fleizach (paragraph 0063).
With respect to claims 12, 32, and 50, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the fourth type of gesture does not include multiple instance of the first type of gesture and the third type of gesture does not include multiple instances of the second type of gesture (e.g. paragraph 0620, finger/hand gesture combined with force, user clenching first, etc., to open menu/dialog; paragraph 1468, finger pinch pose used to click selected item; paragraph 1381, first pose to display menu; paragraph 1474, first and second body states may be counterparts of one another; body states including wrist extension, flexion, rotation, radial deviation, ulnar deviation, etc.; i.e. where a fist clenching or other user input used for opening a menu, analogous to a fourth type of gesture does not include multiple instances of a first body state such as wrist extension, flexion, rotation, deviation, etc., analogous to a first type of gesture, and where a finger pinch pose for clicking a selected item, analogous to a third type of gesture, does not include multiple instances of a second body state such as wrist extension, flexion, rotation, deviation, etc., counterpart to a first body state, analogous to a second type of gesture ).
With respect to claims 14, 34, and 52, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the menu includes one or more selectable options, wherein the one or more selectable options includes a first selectable user interface object for changing operations that one or more hand gestures can cause to the computer system to perform, wherein selection of the first selectable user interface object causes the computer system to display a plurality of settings, and wherein each setting controls an operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system (e.g. paragraph 1483, Fig. 36S, multi-state user interface configurable/programmable by user; user able to specify mappings between body states and responsive actions; such as mapping body part sates of ring/middle/pinky/index finger pinches, first pose, and open hand pose to scroll up/down, activate/deactivate link, increment/decrement selector actions; paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H; paragraph 1486, Fig. 36I, user modifying mappings between body states and actions; configuring web navigation settings by selecting setting options shown in drop-down menu of web browser; user clicking and selecting “scroll:options” to configure type of pose that initiates scrolling control or feature; paragraph 1487, Fig. 36J, link activate setting used to highlight links included in web page; user clicking on “linksActivate:options” to configure type of pose and predetermined pose duration that initiates the activation and rendering of link displayed in webpage; paragraph 1488, Fig. 36K, user clicking on “click:options” to configure type of pose and predetermined duration that initiates a click action in an application; i.e. selection of the “Control Options” dropdown object of Fig. 36G causes the plurality of settings to be shown in Fig. 36H, where these settings allow control of which gestures/poses are mapped to which operations).
With respect to claims 15, 35, and 53, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein:
the menu includes one or more selectable options (e.g. paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H);
the one or more selectable options includes a second selectable option for transitioning the computer system to a fourth operation mode and a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode (e.g. paragraph 1485, Fig. 36G, multi-state user interface facilitates enabling and disabling body state control using “enabled” button; drop down menu control from which user is able to select certain display settings and map body states to responsive actions; see also “reset settings” button in Fig. 36G;  i.e. where an “enabled” button can be selected to transition the system to a state/mode in which the body state controls are enabled, and to a state/mode in which body state controls are disabled, analogous to at least a second selectable option for transitioning the computer system to a fourth operation mode (i.e. a body state controls disabled mode); further where the user selects the drop down menu and modifies the mappings of body states and responsive actions, this causes the system to be transitioned from a first mode (i.e. a mode in which a current set of body state/action mappings are utilized) to a different/fifth mode (i.e. a mode in which a modified/custom set of body state/action mappings are utilized), analogous to a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode; in addition, where body state/action mappings are already customized/modified, and the user selects the option to “reset settings” this would cause the system to be transitioned from a first mode (i.e. a mode in which a current set of modified/customized body state/action mappings are utilized) to a different/fifth mode (i.e. a mode in which a default/reset set of body state/action mappings are utilized), analogous to a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode);
selection of the second selectable option causes the computer system to transition to the fourth operation mode (e.g. paragraph 1485, Fig. 36G, enabling/disabling body state control in using “enabled” button, as discussed above); and
selection of the third selectable option causes the computer system to transition to the fifth operation mode (e.g. paragraph 1485, Fig. 36G, user selecting drop down menu control to map body states to responsive actions; user selecting to “reset settings”, as discussed above).
With respect to claims 16, 36, and 54, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the menu includes one or more selectable options, wherein the one or more selectable options include a fourth selectable option for displaying one or more additional selectable options, and wherein selection of the fourth selectable option causes the computer system to display the one or more additional selectable options that were not previously displayed before selection of the fourth selectable option (e.g. paragraph 1483, Fig. 36S, multi-state user interface configurable/programmable by user; user able to specify mappings between body states and responsive actions; such as mapping body part sates of ring/middle/pinky/index finger pinches, first pose, and open hand pose to scroll up/down, activate/deactivate link, increment/decrement selector actions; paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H; paragraph 1486, Fig. 36I, user modifying mappings between body states and actions; configuring web navigation settings by selecting setting options shown in drop-down menu of web browser; user clicking and selecting “scroll:options” to configure type of pose that initiates scrolling control or feature; paragraph 1487, Fig. 36J, link activate setting used to highlight links included in web page; user clicking on “linksActivate:options” to configure type of pose and predetermined pose duration that initiates the activation and rendering of link displayed in webpage; paragraph 1488, Fig. 36K, user clicking on “click:options” to configure type of pose and predetermined duration that initiates a click action in an application; i.e. selection of the “Control Options” dropdown object of Fig. 36G, analogous to a fourth selectable option, causes additional selectable options, not previously displayed, to be displayed, as shown in Fig. 36H).
With respect to claims 19, 39, and 57, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
after displaying the user interface that includes the first user interface object, detecting, via at least the optical sensor, a fourth hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, fist clenching; paragraph 1463, Fig. 35O, fist pose; i.e. while/after the menu including the first user interface object is being/has been previously displayed, or at some later time, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the fourth hand gesture:
in accordance with a determination that the fourth hand gesture is a fifth type of gesture that is different from the first type of gesture and the second type of gesture, transitioning the computer system from an inactive state to an active state (e.g. paragraph 0509, mapping neuromuscular signals to control signals to control device based on type of gesture; paragraph 0618, determining the gesture being performed; paragraph 0620, fist clenching by user results in wake signal being sent to the system; paragraph 0703, first muscular activation state detected used to determine that a wake up mode is to be controlled, and second muscular activation state used to second activation state used to control initialization operation of the wake up mode; paragraph 0706, wake up/initialization mode; paragraph 1380, mappings causing device to perform action in response to detection of specific body part states; paragraph 1463, Fig. 35O, fist pose mapped to command to display, enter, and activate radial menu; i.e. when the system determines, based on a mapping body part states/gestures to actions, that the user is performing a fist pose hand gesture or other suitable gesture, the system may perform a corresponding action of activating a menu, or waking the system from an inactive/shut down state, analogous to transitioning the system from an inactive state to an active state).
With respect to claims 20, 40, and 58, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying, via the display generation component, the indication that the second user interface object is selected, detecting, via at least the optical sensor, a fifth hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraphs 1473-1475, first body state mapped to selection of next indicator in given sequence; first and second body states may represent counterparts; first body state mapped to selection of next visual indicator in sequence, second body state mapped to selection of previous visual indicator in sequence; i.e. while the interface/menu including the first, second, and third objects is being displayed, including after the user has performed a gesture causing a second user interface object to be selected and while the second user interface object is displayed with an indication that it is selected, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the fifth hand gesture:
in accordance with a determination that the fifth hand gesture is the first type of gesture, displaying, via the display generation component, an indication that a fourth first user interface object is selected (e.g. paragraphs 1473-1475, first body state mapped to selection of next indicator in given sequence; in response to identification of first body state, deselecting indicator 351111 and selecting indicator 351113, providing visual feedback; Figs. 36B-E; i.e., where the user has already performed the first type of gesture once, causing a second user interface object to be selected as shown in Fig. 36C, the user may perform another instance of the first type of gesture, which is mapped to selection of a next indicator, such that after this second instance of the first type of gesture is performed, a next/subsequent indicator (with respect to currently selected indicator 351113, such as the next indicator below it) will be selected, and indicated as selected via visual feedback, analogous to displaying an indication that a fourth UI object is selected in accordance with a determination that the fifth hand gesture is the first type of gesture (Examiner notes that a similar rationale would be applicable in the interfaces of Figs. 36D-E, such that if a user performs a first instance of the first type of gesture to cause item “E” to be selected as the next in sequence, and then performs a second instance of the first type of gesture, this will cause the item “F” to be selected as the next item in the sequence, and indicated as such)); and
in accordance a determination that the fifth hand gesture is the second type of gesture, displaying, via the display generation component, the indication that the first user interface object is selected (paragraphs 1473-1475, first and second body states may represent counterparts; second body state mapped to selection of previous visual indicator in sequence; i.e., where the user has already performed the first type of gesture, causing a second user interface object to be selected as shown in Fig. 36C, the user may perform the second type of gesture, which is mapped to selection of a previous indicator, such that after this second type of gesture is performed, a previous indicator (with respect to currently selected indicator 351113, such as the previous indicator above it) will be selected, and indicated as selected via visual feedback (i.e. the selection will be reverted back to the state shown in Fig. 36B), analogous to displaying an indication that the first UI object is selected in accordance with a determination that the fifth hand gesture is the second type of gesture (Examiner notes that a similar rationale would be applicable in the interfaces of Figs. 36D-E, such that if a user performs a first instance of the first type of gesture to cause item “E” to be selected as the next in sequence, and then performs the second type of gesture, this will cause the item “D” to be selected as the previous item in the sequence, and indicated as such, similar to what is shown in Fig. 36D)).
With respect to claims 21, 41, and 59, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying the user interface that includes the first user interface object, detecting a request to transition the computer system from the first operation mode to a fourth operation mode (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 1470, Fig. 35 R, open hand pose, mapped to command to deactivate the menu; i.e. while the menu is being displayed, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected, including a gesture to request transitioning from a first operation mode (such as one in which a menu is activated) to a fourth operation mode (such as one in which the menu is deactivated/disabled));
in response to detecting the request to transition the computer system from the first operation mode to the fourth operation mode, transitioning the computer system from the first operation mode to the fourth operation mode (e.g. paragraph 1461, disabled radial menu; paragraph 1471, deactivating radial menu; i.e. in response to detecting the user’s open hand gesture, the radial menu is disabled/deactivated, transitioning from an operation mode in which the radial menu is activated (analogous to a first operation mode) to an operation mode in which the radial menu is deactivated (analogous to a fourth operation mode));
while the computer system is in the fourth operation mode and while displaying the user interface that includes the first user interface object, the second user interface object, and the third user interface object, and the indication that the first user interface object is selected (e.g. paragraph 1471, state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; i.e. even after the radial menu is deactivated, all user interface objects, and selection indicator (indicating which object is selected) may continue to be displayed on the display), detecting, via at least the optical sensor, a sixth hand gesture (e.g. continuously receiving neuromuscular/musculoskeletal signals and continuously performing corresponding gesture recognition as cited above); and
in response detecting, via least the optical sensor, the sixth hand gesture:
in accordance with a determination that the sixth hand gesture is the first type of gesture, continuing to display the indication that the first user interface object is selected (e.g. paragraph 1411, user rotating arm without making fist pose has no effect on the radial menu; paragraph 1461, disable radial menu is a mode in which the user is not able to interact with the radial menu; paragraph 1470, deactivated radial menu; paragraph 1471 state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; radial menu will not recognize/respond to subsequent selection (wrist gesture) or click command until receiving a subsequent activation command); i.e. after the radial menu is deactivated, all user interface objects including the selection indicator are continuously displayed and, while in the deactivated state, the user’s hand gestures, such as a first type of gesture to select a next object, will have no effect, and will not be recognized/responded to, such that the menu will continue to be displayed including the selection indicator (indicating a current action indicator as selected) in an unchanged state); and
in accordance with a determination that the sixth hand gesture is the second type of gesture, continuing to display the indication that the first user interface object is selected (e.g. paragraph 1411, user rotating arm without making fist pose has not effect on the radial menu; paragraph 1461, disable radial menu is a mode in which the user is not able to interact with the radial menu; paragraph 1470, deactivated radial menu; paragraph 1471 state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; radial menu will not recognize/respond to subsequent selection (wrist gesture) or click command until receiving a subsequent activation command); i.e. after the radial menu is deactivated, all user interface objects including the selection indicator are continuously displayed and, while in the deactivated state, the user’s hand gestures, such as a second type of gesture to select a previous object, will have no effect, and will not be recognized/responded to, such that the menu will continue to be displayed including the selection indicator (indicating a current action indicator as selected) in an unchanged state).
Claims 13, 18, 33, 38, 51, and 56 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Kim et al. (US 20160029899 A1).
With respect to claims 18, 38, and 56, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying the menu, detecting, via at least the optical sensor, a third hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 1470, Fig. 35 R, open hand pose; i.e. while a menu is being displayed, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that the third hand gesture is a type of gesture, ceasing to display the menu (e.g. paragraph 0509, mapping neuromuscular signals to control signals to control device based on type of gesture; paragraph 0618, determining the gesture being performed; paragraph 1381, open hand pose may cause the wearable device to direct the computing device to close the menu; paragraph 1380, mappings causing device to perform action in response to detection of specific body part states; paragraph 1470, open hand pose mapped to command to hide, exit, or deactivate menu; following identification of open hand pose, some or all of menu may be obscured or hidden from view within the GUI; i.e. when the system determines, based on a mapping body part states/gestures to actions, that the user is performing an open hand gesture or other suitable gesture, the system may perform a corresponding action of ceasing to display a currently displayed menu).
Osborn does not explicitly disclose that the type of gesture is a fourth type of gesture.  However, Kim teaches that the type of gesture is a fourth type of gesture (e.g. paragraph 0256, Figs. 17A-B, while message check screen outputted through display, if command for switching to lock screen is received, stopping output of message check screen and outputting/switching to lock screen; command may include first input (which is shown as a clenched fist gesture in Figs. 17A-B); i.e. where a user performing a fist gesture to initiate/switch to a lock screen display would cause the system to cease displaying all currently displayed content, including menus).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Fleizach, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) and Fleizach (directed to providing control of a touch-based user interface absent physical touch capabilities), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability, while displaying user interface content (i.e. such as a GUI including currently displayed menus, as taught by Osborn) to receive a hand gesture to transition to a lock screen state/mode, such as in response to a clenched fist hand gesture, and in response, cease displaying currently displayed content (such as the menu of Osborn) and display a lock screen (i.e. as taught by Kim).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claims 13, 33, and 51, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
detecting, via at least the optical sensor, a third hand gesture; and in response to detecting, via at least the optical sensor, the third hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected):
in accordance with a determination that the third gesture is the fourth type of gesture, performing an action (e.g. paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user); and
in accordance with a determination that a notification was received and in accordance with a determination that the third hand gesture is a type of gesture, performing an action related to the notification (e.g. paragraph 1649 and corresponding table, using swipe to engage/ignore notification; paragraphs 1651-1653, flick gesture may be used as dismissal mechanism for notifications; notifying user that next meeting is coming up, flick dismissing notification outright, index pinch tap engaging timer function, rolling wrist to select timer duration, etc.).
Osborn does not explicitly disclose that the determinations (for performing the action related to the notification) are that the notification was received within a threshold period of time and that the third hand gesture is the fourth type of gesture.  However, Kim teaches in accordance with a determination that a notification was received within a threshold period of time and in accordance with a determination that the third hand gesture is the fourth type of gesture, performing an action related to the notification (e.g. paragraph 0014, determining level of folding fingers of the user as being in various states (i.e. gestures) based on measured blood flow rate; paragraph 0186, Fig. 5A, user determined in sleep state, generated notifications controlled to not be outputted to user; when user wakes up, user can be informed of saved notifications; paragraph 0187, after detecting user has got out of sleep state, if prescribed command (such as making a fist twice) is received, outputting weather information through the display; paragraph 0190, if user wakes from sleep state, outputting notification screen 501 for cumulative notifications (notification list of plurality of notifications) to user, if prescribed command is received, such as input of making a fist twice, as shown in Figs. 5A-C; i.e. where a notification is received within a time period during which the user is sleeping (analogous to a threshold period of time) and then an input of making a fist gesture is received, an action related to the notification is performed, such as displaying the notification; paragraph 0227, Fig. 11A, call signal currently received, displaying call screen 1101 indicating call signal is currently being received; paragraph 0229, Figs. 11A-C, when call signal currently received, if first input maintained over prescribed time, making response to call signal, and outputting call connected screen 1103; i.e. as shown in Figs. 11A-C, if, during the time period that a call signal is being currently received and a corresponding notification screen is displayed (e.g., during the time period beginning when the call signal is first received and ending when the signal is no longer received, analogous to a threshold period of time within which a notification is received), if a user makes an input such as holding a fist for a time period, the call is answered, analogous to performing an action (answering the call) related to the notification (the screen indicating the call is being received)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Fleizach, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) and Fleizach (directed to providing control of a touch-based user interface absent physical touch capabilities), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to detect the fourth type of gesture (i.e. of Osborn, such as a fist gesture for causing a menu to be displayed) and, based on determining that a received gesture is the fourth type of gesture/fist gesture and further determining that a notification has been received within a threshold time period (such as during a time period in which the user was sleeping, or during a time period in which a call signal is currently received and a corresponding notification is displayed), performing a related action, such as displaying the notification, or answering a corresponding phone call, etc., as taught by Kim.  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
Claims 2, 24, and 42 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Chung et al. (US 20220326779 A1).
With respect to claims 2, 24, and 42, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed.  Although Osborn teaches that auxiliary sensors which may contribute to gesture detection may include heart rate sensors, PPG sensors, and cameras used to detect changes in blood flow within the user (e.g. paragraph 0281, sensors including cardio sensor to measure user’s pulse, heart rate; paragraph 0286, auxiliary device/sensor including heart-rate monitor; paragraph 0310, nonvisible light used to capture image of heat distribution, providing indication of blood flow, used to infer condition of user such as forces being exerted by certain fingers; paragraph 0729, PPG sensors detecting vascular changes), Osborn does not explicitly disclose wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor.
However, Chung teaches wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor (e.g. paragraph 0093, sensor module sensing information for movement of electronic device, and may include photoplethysmogram (PPG) sensor; paragraph 0098, recognizing OCO and shake gestures, gestures of turning hand, moving hand with palm open, lifting arm, moving thumb, moving up/down hand, etc.; gesture may be recognized based on user’s biometric signal (e.g. a signal indicating relaxation/contraction of blood vessels or a heartbeat signal); paragraph 0104, gesture application invoking sensor API to activate sensors, such as PPG sensor; gesture application identifying information for gestures; paragraph 0136, OCO gesture, PPG sensor detecting change due to relaxation and contraction of blood vessels when user opens and closes her hand; ).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Fleizach, and Chung in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), and Fleizach (directed to providing control of a touch-based user interface absent physical touch capabilities), to incorporate the teachings of Chung (directed to recognizing gestures) to include the capability to detect the hand gesture based on heart rate data detected via the optical sensor, such as by detecting a variety of gestures using a PPG sensor, based on a user biometric signal indicating relaxation/contraction of blood vessels or heartbeat (as taught by Chung).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability to recognize a gesture when there is a gesture recognition request from a specific application, eliminating the need for sensors to stay on and the gesture recognition process to keep on running, reducing power consumption and resource waste, and in order to make it possible to immediately recognize relatively small gestures, as described in Chung (paragraphs 0014-0015).
Claims 17, 37, and 55 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Saurabh et al. (US 20180181261 A1).
With respect to claims 17, 37, and 55, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed.  Osborn does not explicitly disclose wherein the menu is displayed:
in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location; and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location.
However, Saurabh teaches wherein the menu is displayed:
in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location (e.g. abstract, positioning popup interface within display frame in relation to corresponding target element; relative positioning of popup interface to associated target element dynamically selected and determined based on location of corresponding target element within quadrants/subquandrants of display frame; paragraph 0001, presenting additional information/menu in the form of a popup interface; paragraph 0004, rendering popup interface at position determined to be best fit position with regard to target element’s position; paragraph 0005, determining dynamically made where to display the popup interface by determining quadrant the target element is actually displayed within; paragraph 0027, pop interface may refer to a context menu; paragraph 0028, target element linked to popup interface; target element may be word or icon, and popup interface may be a selection menu related to the word or icon; paragraph 0077, Fig. 8, target element 802 residing within subquadrant 5/quadrant 1, resulting popup interface 804 displayed at bottom position relative to target element; i.e. where a respective user interface element (target element) is displayed at a position within a first quadrant (analogous to a first location), the corresponding popup menu is also displayed at a position within the same quadrant/location); and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location (e.g. abstract, positioning popup interface within display frame in relation to corresponding target element; relative positioning of popup interface to associated target element dynamically selected and determined based on location of corresponding target element within quadrants/subquandrants of display frame; location of popup interface is dynamically adjustable as opposed to being positioned based on a predetermined location; paragraph 0004, rendering popup interface at position determined to be best fit position with regard to target element’s position, event when that target element changes position or otherwise moves; paragraph 0005, determining dynamically made where to display the popup interface by determining quadrant the target element is actually displayed within; as the target element moves within the user interface (due to scrolling etc.) the relative position of the popup interface is dynamically adjusted so that the popup interface is continuously displayed in a position identified as best fit relative to target element; paragraph 0083, Fig. 10, target element 1002 displayed spanning quadrant1 and quadrant2, resulting popup interface 1004 displayed in bottom position relative to target element 1002; i.e., where a respective user interface element is displayed at a different location such as a location spanning multiple quadrants (analogous to not being displayed at a first location), the corresponding popup menu may also be moved/repositioned on the display so that it is also displayed at a second location which is different from the first location).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Fleizach, and Saurabh in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects, including displaying associated menus) and Fleizach (directed to providing control of a touch-based user interface absent physical touch capabilities), to incorporate the teachings of Saurabh (directed to a positioning mechanism for bubble as a custom tooltip, such as positioning popup interfaces/menus in relation to a corresponding target element) to include the capability to dynamically display the menu (i.e. of Osborn) in a position relative to the position of a corresponding UI object such that when the UI object is displayed at a first location, the menu is also displayed at the first location and when the UI object is not displayed at the first location, such as because it is moved to a different location, the menu is displayed at a second location different from the first location, such as at the different location that the UI object has been moved to (as taught by Saurabh).  One of ordinary skill would have been motivated to perform such a modification in order to solve disadvantages found in existing popup interfaces by providing the capability to dynamically render a popup interface at a position that is determined to be a best fit position with regard to a target elements position, event when that target element changes position or otherwise moves in some manner, as described in Saurabh (paragraphs 0002-0004).
Claims 4, 26, and 44 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Li (US 20160328108 A1) , further in view of Kin (US 10261595 B1), further in view of Pantel (US 20140320434 A1).
With respect to claims 4, 26, and 44, Osborn in view of Fleizach teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches
wherein before detecting the hand gesture, the computer system is in a first operation mode (e.g. paragraph 1463, Fig. 35O, fist pose mapped to command to display, enter, and activate radial menu; i.e. where in the first state, at the time the radial menu is displayed, the user has provided a previous input causing the radial menu to be activated, and therefore in an operation mode in which the menu is activated, analogous to a first operation mode),
wherein performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode (e.g. paragraph 1468, Fig. 35Q, illustrating finger pinch pose 351006, which may be mapped to command/instruction to click visual indicator in the radial menu, analogous to mouse button click or keypress; alternatively combination of finger pinch pose and certain degree of force mapped to click command; paragraph 1469, Fig. 35 U, clicking/engaging with or activating function of the currently selected action indicator; clicking action indicator in response to identified click gesture; upon performing the click action, causing visual indicator to appear to depress and/or release, similar to a physical button; i.e. clicking the currently selected action indicator causes the corresponding function to be activated, and therefore causes the system to transition to an operation mode in which the corresponding function has been activated, analogous to a second operation mode)
detecting, via at least the optical sensor, a second hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that the second hand gesture is one type of gesture, toggling a first automatic scrolling operation between an active state and an inactive state (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling up a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll up a GUI; paragraph 1484, ring finger pinch mapped to scroll up so that when user makes ring finger pinch pose, the page/browser displayed scrolls up); and
in accordance with a determination that the second hand gesture is another type of gesture, reversing a first direction of the first automatic scrolling operation  (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling down a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll down a GUI; paragraph 1484, pinky finger pinch mapped to scroll down so that when user makes pinky finger pinch pose, the page/browser displayed scrolls down).
Although Osborn further teaches that user hand gestures may be mapped to different commands in different states/modes, and further teaches that a wrist roll gesture may be mapped to a scroll command (e.g. paragraph 0648, switching between modes; paragraph 0666-0667, selection of mode, such as from list or using gesture; paragraph 1380, transitioning between different mappings of body part states and responsive actions; paragraph 1408-1410, activating first mapping and/or deactivating second mapping between possible body part states and responsive actions; user switching between mappings; transition/switch between mappings; paragraph 1486, user modifying mappings between body states and actions, user configuring/mapping scroll controls to different poses; table accompanying paragraph 1649, indicating that an interaction with a list feature may include a “scroll through” interaction with a “wrist roll” control scheme), Osborn does not explicitly disclose:
wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture,
while the computer system is in the second operation mode, detecting the second hand gesture;
in response to detecting the second hand gesture:
in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation;
in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation; and
in accordance with a determination that the second hand gesture is a fourth type of gesture different from the first type of gesture, the second type of gesture, and the third type of gesture, ending the automatic scrolling operation.
However, Li teaches:
wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture (e.g. paragraph 0037-0038, utilizing auto-scroll method such that contents are presented in multiple steps/views within given time period; user starting content show with auto scroll where contents are presented and scrolled automatically and continuously; user selecting scroll options, etc.; paragraph 0046, auto scroll replaces contents on display at given rate continuously and without user involvement),
while the computer system is in the second operation mode, detecting the second hand gesture (e.g. paragraph 0046, describing user inputs/gestures provided while content is automatically scrolled, such as a finger hover/touch input gesture to pause scrolling; paragraph 0047, describing other types of gesture which may be used as command input, including user holding hand upright with palm facing a horizontal direction, user hand movements to make selections, etc.; paragraph 0053, describing user methods for stopping a scroll process temporarily, such as selecting on screen buttons, touch inputs, and use of gestures via gesture recognition system, such as by moving hand or fingertip in a given direction, where the user’s hand may be in any position, vertical, horizontal, tilted, using both a fist or spread hand for hand gestures, and using one or multiple fingertips for finger gestures);
in response to detecting the second hand gesture:
in accordance with a determination that the second hand gesture is the first type of gesture, pausing the automatic scrolling operation (e.g. paragraph 0046-0047, user inputs to pause scrolling, including hand gestures; paragraph 0053, user input to stop scroll process temporarily, including use of hand gestures, including hand gestures in a given direction, using a tilted hand, etc.; i.e. the user may provide a hand gesture to pause scrolling, including a directional/tilted hand gesture, analogous to pausing an automatic scrolling operation in accordance with a determination that a hand gesture is a first type of hand gesture);
in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation (e.g. paragraph 0044-0045, scroll direction may be reversed using gesture, such as swipe on touch screen; via buttons, etc.; paragraph 0047, describing use of other gestures as command input, including directional movements to select an option; paragraph 0053, user turning page forward or backward during scroll session, including use of hand gestures, including hand gestures in a given direction, using a tilted hand, etc.; i.e. the user may provide a hand gesture to reverse scrolling direction, or to turn pages in any direction (including an opposite direction) during automatic scrolling, including a directional/tilted hand gesture, analogous to reversing a direction of scrolling in accordance with a determination that a hand gesture is a second type of gesture); and
in accordance with a determination that the second hand gesture is the fourth type of gesture, ending the automatic scrolling operation (e.g. paragraph 0043, selection of button to close content window and end scroll session; selection of no button to stop scroll process; scroll session controlled by device movement, etc.; paragraph 0047, describing various user gestures which may be used as command input, including hand movements to select a no icon/option; paragraph 0053, describing various hand gestures which may be used as command inputs, including hand gestures with a fist; paragraph 0054, option to quit auto scroll prepared; tapping/clicking/selecting quit button; i.e. the user may provide a different hand gesture, such as a closed fist or other hand gesture, in order to provide a command to close/exit the auto scroll operation mode).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Li in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), to incorporate the teachings of Li (directed to presenting contents using auto scroll) to include the capability to place the system in second mode in response to a third type of user input (i.e. such as a mode of executing a corresponding functionality following a user’s selection activating a displayed item, as taught by Osborn, and a mode of initiating automatic and continuous content scrolling functionality following a user’s selection of a corresponding user interface component/option, as taught by Li), where the second mode includes an automatic scrolling function and, while in this automatic scrolling mode, receive an input/gesture to pause the automatic scrolling (including a directional/tilted hand gesture, analogous to a first type of gesture, as taught by Li, similar to the directional wrist rotation gesture as taught by Osborn), receive an input/gesture to reverse the direction of the automatic scrolling (including a directional/tilted hand gesture in a direction corresponding to a reverse direction, as taught by Li, similar to the directional wrist rotation gesture as taught by Osborn), and receive an input/gesture to exit/quit the auto scroll operation/mode (including a gesture using a hand in a fist and/or selecting a corresponding option, as taught by Li, similar to a fist gesture or other gesture as taught by Osborn).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability for the user to relax and watch contents effortlessly, as described in Li (paragraph 0037).
Although Li further teaches user inputs/gestures for changing a speed of the automatic scrolling operation (e.g. paragraphs 0037-0038, user changing scroll rate as needed/setting scroll rate, including using an adjustable button/column; paragraph 0054, user option to adjust scroll rate; paragraph 0071, user selecting option, including scroll rate, to adjust), Osborn and Li do not explicitly disclose in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation.  However, Kin teaches in accordance with a determination that the second hand gesture is the third type of gesture, changing a speed of the automatic scrolling operation (e.g. col. 14 lines 3-20, detecting pinch scroll gesture, if detecting pinch scroll gesture with thumb and two fingers, speeding up rate of scroll action compared to action with just one finger; after detecting pinch scroll gesture, detecting change in pose of user’s hand while in pinched position, and increasing the rate of scrolling).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, and Kin in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) and Li (directed to presenting contents using auto scroll), to incorporate the teachings of Kin (directed to high resolution tracking and response to hand gestures through three dimensions) to include the capability, while performing automatic scrolling (as taught by Li, incorporated into the system of Osborn), to detect a pinch gesture, or a change in an existing pinch gesture (as taught by Kin, analogous to a third type of gesture as taught by Osborn, such as for selecting a displayed option), and in response, change the speed/rate of the scrolling operation (as taught by Kin, such as by pinching with an increased number of fingers, or changing a hand pose while pinching).  One of ordinary skill would have been motivated to perform such a modification in order to overcome limitations of conventional hand tracking systems, which are unable to accurately track positions of user fingers and thumbs and precise movements of user digits, causing poor representation of the user’s hands, by providing the capability to detect various gestures performed by the user’s hands and cause a change in display instructions in response to the detected gesture as described in Kin (col. 1 lines 20-43).
Assuming arguendo that Osborn, Li, and Kin do not explicitly disclose in accordance with a determination that the second hand gesture is the first type of hand gesture, pausing the automatic scrolling operation, Pantel teaches in accordance with a determination that the second hand gesture is the first type of hand gesture, pausing the automatic scrolling operation (e.g. paragraph 0204, Fig. 17, illustrating forearm-spin gesture, in which smartwatch is rotated about the x axis by twisting the forearm; paragraph 0220, forearm spin gesture can be executed in two directions, and direction can be assigned to different functions/tasks, including, at paragraph 0222, locking of the scroll capability of a page/card/tab; i.e. the user may temporarily lock/pause a scrolling operation/capability using wrist/forearm rotation in an assigned direction).  In addition, assuming arguendo that Osborn, Li, and Kin do not explicitly disclose in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation, Pantel teaches in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation (e.g. paragraph 0175, user performing drag away gesture, such as performing conventional drag gesture to scroll through a page, where user can also change direction of drag movement/page scrolling; paragraph 0204, Fig. 17, forearm spin gesture is variant of drag along/away gesture, where smart watch is rotated around axis by twisting the forearm while index finger is held in place; paragraph 0209, in forearm spin gesture, user touches screen with finger and starts to twist forearm while maintaining finger steady, resulting in dragging of the finger in manner comparable to drag along/away gestures; paragraph 0214-0215, triggering drag completed/flick based on direction/distance/speed; paragraph 0220, forearm spin gesture executed in two directions, direction of gesture assigned to different functions; i.e. the user may also utilize the forearm spin gesture in a given direction to execute functions associated with drag inputs, including causing scrolling to be performed in a given direction, such as a reversed/changed direction).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, and Pantel in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Li (directed to presenting contents using auto scroll), and Kin (directed to high resolution tracking and response to hand gestures through three dimensions), to incorporate the teachings of Pantel (directed to gesture control) to include the capability, while performing automatic scrolling (as taught by Li, incorporated into the system of Osborn), to detect a forearm spin gesture (as taught by Pantel, similar to a wrist rotation gesture as taught by Osborn) in a given direction and either lock/pause a scrolling operation (as taught by Pantel, similar to pausing scrolling according to a hand gesture as taught by Li), or reverse the direction of scrolling (as taught by Pantel, similar to controlling scrolling direction as taught by Osborn and Li).  One of ordinary skill would have been motivated to perform such a modification in order to provide novel gestures in addition to familiar touchscreen gestures, allowing access of several features or functions of an operating system or application using only a single button, icon, or control element, or global system functions using novel gestures at any location regardless of displayed content as described in Pantel (paragraph 0014).
Claims 3, 25, and 43 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Li , further in view of Kin, further in view of Pantel, further in view of Friedman (US 20110090407 A1).
With respect to claims 3, 25, and 43, Osborn in view of Fleizach, further in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture (e.g. paragraph 1473-1474, Figs. 36B-C, first body state mapped to selection of next indicator in given sequence; in response to identification of first body state, deselecting indicator 35111 and selecting indicator 351113; first and second body states may represent counterparts of one another; 1475, Figs. 36D-E, first body state mapped to selection of next visual indicator in the sequence; i.e. because first body state/gesture causes selection to move to a next object in sequence, receipt of multiple instances of this gesture may also cause a different/third UI object to be selected; for example, where multiple instances/two instances of the first type of gesture are received, the object selected as indicated may be first moved from the first UI object to the second/next UI object in the sequence, and then moved again from the second/next UI object to a third/subsequent UI object in the sequence).
Assuming arguendo that Osborn, Fleizach, Li, Kin, and Pantel do not explicitly disclose wherein the second type of gesture that is multiple instances of the first type of gesture (i.e. with respect to the second mode, where this gesture is used to reverse a direction of scrolling), Friedman teaches wherein the second type of gesture that is multiple instances of the first type of gesture (e.g. paragraphs 0046-0047, describing Fig. 5, showing table 500 with gesture motions and respective commands, including rows 504 and 506, with gesture commands having two short down movements, or two short up movements, to scroll down or up a given number of rows, and rows 510 and 512, used to scroll time forwards, or reverse/scroll time backwards within menu (i.e. scroll left or right in a timeline menu); i.e. where a gesture to scroll in a given direction, such as in a reverse direction, may include multiple instances of a given gesture, such as doing the gesture twice with two short movements in order to scroll in a direction, including a direction opposite of a current direction).
To clarify, as cited above, Osborn teaches that in a first mode, a first type of gesture, such as wrist extension/flexion, rotation, radial/ulnar deviation, etc. may be used to change selection to a second user interface object, and Li and Pantel both teach that in a second mode, this same first type of gesture (i.e. a directional/tilted hand gesture) may be used to pause an automatic scrolling operation.  Osborn and Kim both teach that in the first mode, a second type of gesture, which is multiple instances of the first type of gesture, may be used in order to change a selection to a third user interface object, and Li and Pantel both teach that in the second mode, a second type of gesture (which is at least one instance of the first type of gesture) may be used to reverse a direction of the automatic scrolling operation.  Therefore, the only thing missing from this combination is that the second type of gesture (which may include at least one instance of the first type of gesture as taught by Li and Pantel), performed in the second mode for reversing the direction of the scrolling operation, is performed using multiple instances of the gesture.  However, as discussed and cited above, Friedman teaches that the second type of gesture, which can be used to set/reverse a given scrolling direction, can be performed using multiple instances of the gesture.
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Friedman in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions) and Pantel (directed to gesture control), to incorporate the teachings of Friedman (directed to gesture controls) to include the capability, while performing automatic scrolling in a given direction (as taught by Li, incorporated into the system of Osborn), to detect multiple instances of a given gesture (i.e. such as the forearm spin gesture as taught by Pantel, similar to a wrist rotation gesture as taught by Osborn) in a given direction and reverse the direction of scrolling (as taught by Friedman, similar to controlling scrolling direction as taught by Osborn, Li, and Pantel; i.e. utilizing multiple instances of a gesture, as taught by Friedman, in order to change scrolling direction, such as multiple instances of the wrist tilt/rotation gesture as taught by at least Osborn and Pantel).  One of ordinary skill would have been motivated to perform such a modification in order to provide a simplified yet flexible user interface as described in Friedman (paragraph 0018).
Claims 10, 30, and 48, are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Li, further in view of Kin, further in view of Pantel, further in view of Kim et al. (US 20160029899 A1).
With respect to claims 10, 30, and 48, Osborn in view of Fleizach, further in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
in accordance with a determination that the second hand gesture is one type of gesture, displaying a notification that indicates a state of the automatic scrolling operation (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling up a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll up a GUI; paragraph 1484, ring finger pinch mapped to scroll up so that when user makes ring finger pinch pose, the page/browser displayed scrolls up; paragraph 1459, providing visual feedback of both identified pose and identified force; when action includes scrolling GUI, displaying cursor in connection with scrolling, which may be presented as a horizontal line with a bar extending above or below the line, depending on the scrolling direction; paragraph 1480, providing current state/setting of interface, current body state, and current action corresponding to current body state; paragraph 1481, graphical elements displaying current interface state, current body state, and responsive action; i.e. based on determining that the user has provided a gesture for scrolling, the system may initiate the scroll operation and may also provide a notification/indicator regarding the state of the operation, such as a cursor indicating a scrolling direction and magnitude, etc.).
Kim teaches that the one type of gesture is the first type of gesture (e.g. paragraphs 0251-254, Figs. 16A-C, outputting notification screen of received text message, receiving message check command such as folding input and outputting message check screen 1601 for displaying details of received text message; scroll action on message check screen performed by rotation of wrist; wrist rotated in first direction, scrolling to move outputted message check screen; direction for rotating wrist includes scroll direction; if scroll in bottom direction desired, rotating wrist in bottom direction; i.e. the user may provide a wrist rotation input in a first or second direction in order to perform a scroll operation in a designated direction, as shown between Fig. 16B and Fig. 16C).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Fleizach, Li, Kin, Pantel, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Fleizach (directed to providing control of a touch-based user interface absent physical touch capabilities), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to receive a hand gesture to initiate automatic scrolling in a direction, and a hand gesture to reverse the scrolling direction (i.e. such as using different fingers to perform pinch gestures to scroll up/down as taught by Kim), where the hand gesture may be a wrist rotation gesture in different directions corresponding to the desired scroll direction (i.e. as taught by Kim, where Osborn also teaches the use of wrist rotation for at least scrolling a wrist, and further teaches that the wrist rotation may be performed in two different directions, and further teaches, based on detecting the scroll gesture, outputting a notification/indicator of the scrolling operation state).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
Claims 11, 31, and 49 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Li, further in view of Kin, further in view of Pantel, further in view of Kim et al. (US 20160029899 A1), further in view of Coenen et al. (US 20190191208 A1).
With respect to claims 11, 31, and 49, Osborn in view of Fleizach, further in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the third type of gesture is a type of gesture that is an instance of the fourth type of gesture (e.g. paragraph 1281, fist squeeze used for discrete event such as to engage or disengage a particular feature within a given application or a continuous event, such as to maintain activation of a particular feature within a given application; paragraph 1283, selecting from presented options by performing hand gesture such as clenching the fist).
Osborn does not teach that the third type of gesture is multiple instances of the fourth type of gesture.  However, Kim teaches the third type of gesture is multiple instances of the fourth type of gesture (e.g. paragraph 0187, user command of making fist twice to cause output of weather information; i.e. as shown in Fig. 5A, where a UI element associated with a weather information check is currently displayed, the user may perform a gesture which is multiple instances of a fist clenching gesture in order to select/display content corresponding to the UI element).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to receive a hand gesture to select a displayed item on the screen(i.e. such as a fist clench gesture as taught by Osborn) as multiple instances of the hand gesture (i.e. such as receiving multiple fist clench gestures as taught by Kim).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
Assuming arguendo that Osborn, Fleizach,Li, Kin, Pantel, and Kim do not explicitly disclose that the third type of gesture is multiple instances of the fourth type of gesture, Coenen teaches the third type of gesture is multiple instances of the fourth type of gesture (e.g. abstract, paragraph 0011, 0082, scrolling speed increased by performing multiple gestures in quick succession).
To clarify, as cited above, Osborn teaches that in a first mode, a third type of gesture, such as finger pinching or fist squeezing, may be used to select a currently indicated object, and Kin teaches that in a second mode, this same third type of gesture (i.e. pinching with a given number of fingers, including all fingers) may be used to change a scrolling speed.  Osborn additionally teaches that in the first mode, a fourth type of gesture, which may be the same as the third type of gesture (i.e. making a fist), may be used in order to display a menu, and Li teaches that in the second mode, the fourth type of gesture, which may be the same as the third type of gesture (i.e. a gesture using a fist) may be used to end/exit the automatic scrolling operation.  Kim additionally teaches that, in the first mode, the third type of gesture may be multiple instances of the fourth type of gesture (i.e. multiple instances of fist clenching/squeezing to select an object for display).  Therefore, the only thing missing from this combination is that the third type of gesture (which may include at least one instance of the fourth type of gesture as taught by Osborn, Li, and Kin), performed in the second mode for changing speed of the scrolling operation, is performed using multiple instances of the gesture.  However, as discussed and cited above, Coenen teaches that the third type of gesture, which can be used to change speed of the scrolling operation, can be performed using multiple instances of the gesture.
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Coenen in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions) and Pantel (directed to gesture control), to incorporate the teachings of Coenen (directed to gestures for scrolling in a user interface) to include the capability, while performing automatic scrolling (as taught by Li, incorporated into the system of Osborn), to detect multiple instances of a given gesture (i.e. such as the fist clenching gesture taught by Osborn, Li, and Kim, and the similar gesture), and in response change the speed of scrolling (as taught by Coenen, i.e. utilizing multiple instances of a gesture, as taught by Coenen, in order to change scrolling speed, such as multiple instances of the fist gesture as taught by at least Osborn, Li, and Kim (where Kim also teaches use of multiple instances of the fist gesture in a different mode)).  One of ordinary skill would have been motivated to perform such a modification in order to provide gestures which are more intuitive to the user than controlling scrolling based on speed or acceleration of gestures as described in Coenen (paragraph 0011).
Claims 8, 28, and 46 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Fleizach, further in view of Li, further in view of Kin, further in view of Pantel, further in view of Platzer et al. (US 20130239046 A1).
With respect to claims 8, 28, and 46, Osborn in view of Fleizach, further in view of Li, further in view of Kin, further in view of Pantel teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn and Li further teach wherein:
the automatic scrolling operation scrolls a sequence of a plurality of interface objects in a second direction (e.g. Osborn paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling down a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll down a GUI; paragraph 1484, pinky finger pinch mapped to scroll down so that when user makes pinky finger pinch pose, the page/browser displayed scrolls down; Li paragraph 0038, user selecting scroll options, including scroll direction; paragraph 0044-0045, scroll direction may be reversed using gesture, such as swipe on touch screen; via buttons, etc.).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Fleizach, Kin, Pantel, and Li in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Fleizach (directed to providing control of a touch-based user interface absent physical touch capabilities), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Li (directed to presenting contents using auto scroll) to include the capability to perform the automatic scrolling operation in a second direction (i.e. in a specified direction, as taught by Li).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability for the user to relax and watch contents effortlessly, as described in Li (paragraph 0037).
Osborn does not explicitly disclose wherein the one or more programs further includes instructions for:
while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects; and
in response to detecting the end of the sequence of the plurality of user interface objects, performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction.
However, Platzer teaches wherein the one or more programs further includes instructions for:
while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects (e.g. paragraph 0077, Figs. 5A-C, scrolling list of items to terminus of the list; paragraph 0078, terminus is edge of scrolled region; while list is being scrolled; terminus is virtual boundary associated with displayed objects; displayed objects reach or intersect with the terminus 514; i.e. the system detects the end of a list of displayed UI objects, as indicated by a terminus at the edge of the scrolled list); and
in response to detecting the end of the sequence of the plurality of user interface objects, performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction (e.g. paragraph 0077, Figs. 5A-C, reaching terminus of list, smoothly bouncing displayed items at end of list, reversing direction; paragraph 0078, as illustrated in Fig. 5B, when objects reach or intersect with the terminus 514, the movement corresponding to scrolling may stop and, as illustrated in Fig. 5C, subsequently reverse direction; i.e. scrolling list reaches and bounces at its terminus; i.e. in response to reaching/detecting the terminus while scrolling in a second direction (such as opposite of a first direction), the system automatically performs a “bounce” by scrolling the list of displayed objects in the opposite direction (which may be a third direction, which may be the same as a first direction)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Li, Kin, Pantel, and Platzer in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects, such as scrolling menus/lists of UI objects, webpages, etc.), Li (directed to presenting contents using auto scroll), Kin (directed to high resolution tracking and response to hand gestures through three dimensions), and Pantel (directed to gesture control), to incorporate the teachings of Platzer (directed to scrolling operations) to include the capability to, while scrolling user interface content, including a plurality of objects, in a given direction (i.e. as taught by Osborn, Kim, and Platzer, such as in a second direction which is opposite of a first scroll direction, or some other direction, where at least Jose teaches a scrolling direction may be set as left-right, right-left, up-down, or down-up), detect a terminus indicating the end of the plurality of objects and, in response automatically scroll the content including the plurality of objects in a reverse direction (as taught by Platzer, where this reverse direction may be a third direction which is different from the second direction, but potentially the same as a first direction).  One of ordinary skill would have been motivated to perform such a modification in order to provide operations desired by users, such as scrolling operations on a device, including on device having limited display size, user interface, etc. which limits the ease of use of the device, as described in Platzer (paragraphs 0006, 0008).
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain,” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting in re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (GCPA 1968)). Further, a reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill the art, including nonpreferred embodiments. Merck & Co, v. Biocraft Laboratories, 874 F.2d 804, 10 USPQ2d 1843 (Fed. Cir.), cert, denied, 493 U.S. 975 (1989). See also Upsher-Smith Labs. v. Pamlab, LLC, 412 F,3d 1319, 1323, 75 USPQ2d 1213, 1215 (Fed. Cir, 2005): Celeritas Technologies Ltd. v. Rockwell International Corp., 150 F.3d 1354, 1361, 47 USPQ2d 1516, 1522-23 (Fed. Cir. 1998).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Yoon et al. (US 20150061842 A1) teaches various gestures for controlling a mobile device such as a wrist worn device (Figs. 7-33).
Offenberg et al (US 20150248235 A1) teaches various gestures for controlling a mobile device such as a wrist worn device (Figs. 27-92), as well as for selecting among displayed items with corresponding visual indications, scrolling, etc. (Figs. 102A-D).
Hazra et al. (US 20170123487 A1) teaches various gestures for controlling a mobile device including a wrist worn device (Figs. 5, 8-15).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action.  Accordingly, THIS ACTION IS MADE FINAL.  See MPEP § 706.07(a).  Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action.  In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action.  In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.

CITED REFERENCE SUMMARIES: Ref US10261595B1: Claims:
A system comprising: an electronic display configured to display one or more simulated objects in accordance with display instructions; an imaging sensor configured to capture images of a user's hands; and a console configured to: receive the captured images from the imaging sensor; extract joint information of the user's hands from the captured images; determine one or more poses based on the extracted joint information; and in response to the determination that the poses indicate the user's index finger positioned orthogonally to the user's thumb, and the thumb is in contact with the index finger, detect a directional pad display gesture; and in response to the detection of the direction pad display gesture, update the display instructions to cause the electronic display to generate a simulated directional pad adjacent to the user's thumb in a simulated environment that is presented to the user via the electronic display, the simulated directional pad indicating each of one or more selection directions each with a simulated arrow, the selection of the one or more selection directions detected based on a detection of further determined poses indicating a movement of a distal end of the user's thumb.
an electronic display configured to display one or more simulated objects in accordance with display instructions;
an imaging sensor configured to capture images of a user's hands; and
a console configured to: receive the captured images from the imaging sensor; extract joint information of the user's hands from the captured images; determine one or more poses based on the extracted joint information; and in response to the determination that the poses indicate the user's index finger positioned orthogonally to the user's thumb, and the thumb is in contact with the index finger, detect a directional pad display gesture; and in response to the detection of the direction pad display gesture, update the display instructions to cause the electronic display to generate a simulated directional pad adjacent to the user's thumb in a simulated environment that is presented to the user via the electronic display, the simulated directional pad indicating each of one or more selection directions each with a simulated arrow, the selection of the one or more selection directions detected based on a detection of further determined poses indicating a movement of a distal end of the user's thumb.
receive the captured images from the imaging sensor;
extract joint information of the user's hands from the captured images;
determine one or more poses based on the extracted joint information; and
in response to the determination that the poses indicate the user's index finger positioned orthogonally to the user's thumb, and the thumb is in contact with the index finger, detect a directional pad display gesture; and
in response to the detection of the direction pad display gesture, update the display instructions to cause the electronic display to generate a simulated directional pad adjacent to the user's thumb in a simulated environment that is presented to the user via the electronic display, the simulated directional pad indicating each of one or more selection directions each with a simulated arrow, the selection of the one or more selection directions detected based on a detection of further determined poses indicating a movement of a distal end of the user's thumb.
The system of claim 1 , wherein the console is further configured to: in response to the determination that the poses indicate the user's thumb is moving in a selected direction, the selected direction being one of the selection directions indicated on the simulated direction pad, detect a directional pad selection gesture; in response to the detection of the directional pad selection gesture, execute instructions associated with the selected direction; and in response to the detection of the directional pad selection gesture, update the display instructions to cause the electronic display to generate a change in simulated content presented to the user.
in response to the determination that the poses indicate the user's thumb is moving in a selected direction, the selected direction being one of the selection directions indicated on the simulated direction pad, detect a directional pad selection gesture;
in response to the detection of the directional pad selection gesture, execute instructions associated with the selected direction; and
in response to the detection of the directional pad selection gesture, update the display instructions to cause the electronic display to generate a change in simulated content presented to the user.
The system of claim 1 , wherein the console is further configured to: in response to the determination that the poses indicate the user's index finger is extended, the user's middle finger is positioned orthogonally to the user's thumb, and the thumb is within a minimum distance to the index finger, detect a multifunction display gesture; in response to the detection of the multifunction display gesture, generate a simulated display of multiple options for selection, the multiple options arranged radially and adjacent to the user's thumb in the simulated environment; and in response to the detection of the multifunction display gesture, update the display instructions to cause the electronic display to generate the simulated display of multiple options via the electronic display.
in response to the determination that the poses indicate the user's index finger is extended, the user's middle finger is positioned orthogonally to the user's thumb, and the thumb is within a minimum distance to the index finger, detect a multifunction display gesture;
in response to the detection of the multifunction display gesture, generate a simulated display of multiple options for selection, the multiple options arranged radially and adjacent to the user's thumb in the simulated environment; and
in response to the detection of the multifunction display gesture, update the display instructions to cause the electronic display to generate the simulated display of multiple options via the electronic display.
The system of claim 3 , wherein the console is further configured to: in response to the determination that the poses indicate the user's index finger moving in a selected option, the selected option being one of the options indicated on the simulated display of multiple options, detect a multifunction selection gesture; in response to the detection of the multifunction selection gesture, execute instructions associated with the selected option; and in response to the detection of the multifunction selection gesture, update the display instructions to cause the electronic display to generate a change in simulated content presented to the user.
in response to the determination that the poses indicate the user's index finger moving in a selected option, the selected option being one of the options indicated on the simulated display of multiple options, detect a multifunction selection gesture;
in response to the detection of the multifunction selection gesture, execute instructions associated with the selected option; and
in response to the detection of the multifunction selection gesture, update the display instructions to cause the electronic display to generate a change in simulated content presented to the user.
The system of claim 1 , wherein the console is further configured to: in response to the determination that the poses indicate the user's thumb on the user's dominant hand is substantially orthogonal to the lateral side of the user's index finger, and the distal portion of the thumb being within a minimum distance to the index finger, detect a pen-holding gesture; in response to the detection of the pen-holding gesture, generate a simulated pen between the user's thumb and the users index finger, the simulated pen capable of simulated writing on a simulated pad; and in response to the detection of the pen-holding gesture, update the display instructions to cause the electronic display to generate the simulated pen.
in response to the determination that the poses indicate the user's thumb on the user's dominant hand is substantially orthogonal to the lateral side of the user's index finger, and the distal portion of the thumb being within a minimum distance to the index finger, detect a pen-holding gesture;
in response to the detection of the pen-holding gesture, generate a simulated pen between the user's thumb and the users index finger, the simulated pen capable of simulated writing on a simulated pad; and
in response to the detection of the pen-holding gesture, update the display instructions to cause the electronic display to generate the simulated pen.
The system of claim 5 , wherein the console is further configured to: in response to the determination that the poses indicate the distal portion of the thumb being exceeding the minimum distance to the index finger, detect a contextual menu gesture; in response to the detection of the contextual menu gesture, generate a simulated contextual menu adjacent to the simulated pen; and in response to the detection of the contextual menu gesture, update the display instructions to cause the electronic display to generate the simulated contextual menu.
in response to the determination that the poses indicate the distal portion of the thumb being exceeding the minimum distance to the index finger, detect a contextual menu gesture;
in response to the detection of the contextual menu gesture, generate a simulated contextual menu adjacent to the simulated pen; and
in response to the detection of the contextual menu gesture, update the display instructions to cause the electronic display to generate the simulated contextual menu.
The system of claim 1 , wherein the console is further configured to: in response to a detection that the user is holding a physical pen device, detect a pen-holding gesture; in response to the detection that the user is holding a physical pen device, generate a simulated pen between the user's thumb and the users index finger, the simulated pen capable of writing on a simulated pad and mirroring the position of the physical pen device relative to the user's hand; and in response to the detection that the user is holding a physical pen device, update the display instructions to cause the electronic display to generate the simulated pen.
in response to a detection that the user is holding a physical pen device, detect a pen-holding gesture;
in response to the detection that the user is holding a physical pen device, generate a simulated pen between the user's thumb and the users index finger, the simulated pen capable of writing on a simulated pad and mirroring the position of the physical pen device relative to the user's hand; and
in response to the detection that the user is holding a physical pen device, update the display instructions to cause the electronic display to generate the simulated pen.
The system of claim 1 , wherein the console is further configured to: in response to the determination that the poses indicate that one of the user's fingers is elongated and that the distal portion of the one finger is intersecting a simulated pressable element in the simulated environment that is presented to the user via the electronic display, detect a button press gesture; in response to the detection of the button press gesture, update the display instructions to generate a simulated button press in the pressable element to the user via the electronic display; in response to the detection of the button press gesture, execute instructions associated with the button press gesture for the pressable element; and in response to the detection of the button press gesture, update the display instructions to cause the electronic display to generate a change in simulated content presented to the user associated with the pressable element.
in response to the determination that the poses indicate that one of the user's fingers is elongated and that the distal portion of the one finger is intersecting a simulated pressable element in the simulated environment that is presented to the user via the electronic display, detect a button press gesture;
in response to the detection of the button press gesture, update the display instructions to generate a simulated button press in the pressable element to the user via the electronic display;
in response to the detection of the button press gesture, execute instructions associated with the button press gesture for the pressable element; and
in response to the detection of the button press gesture, update the display instructions to cause the electronic display to generate a change in simulated content presented to the user associated with the pressable element.
The system of claim 8 , wherein the console is further configured to: in response to the determination that the poses indicate that the one finger is moving in a direction orthogonal to a previous movement direction of the one finger, detect a sliding gesture; in response to the detection of the sliding gesture, execute instructions associated with the pressable element for the sliding gesture to cause a translational movement in simulated content; and in response to the detection of the sliding gesture, update the display instructions to cause the electronic display to generate a translational movement in the simulated content.
in response to the determination that the poses indicate that the one finger is moving in a direction orthogonal to a previous movement direction of the one finger, detect a sliding gesture;
in response to the detection of the sliding gesture, execute instructions associated with the pressable element for the sliding gesture to cause a translational movement in simulated content; and
in response to the detection of the sliding gesture, update the display instructions to cause the electronic display to generate a translational movement in the simulated content.
The system of claim 1 , wherein the console is further configured to: in response to the determination that the poses indicate the distal portion of the user's thumb is within a minimum distance to the distal portion of the user's index finger, wherein the user's hand is moving along one axis, and wherein the user's thumb and the user's index finger surround a scrolling element detect a pinch scroll gesture; in response to the detection of the pinch scroll gesture, execute instructions associated with the scrolling element for a scrolling action to cause a translational movement in simulated content; and in response to the detection of the pinch scroll gesture, update the display instructions to cause the electronic display to generate the translational movement in the simulated content.
in response to the determination that the poses indicate the distal portion of the user's thumb is within a minimum distance to the distal portion of the user's index finger, wherein the user's hand is moving along one axis, and wherein the user's thumb and the user's index finger surround a scrolling element detect a pinch scroll gesture;
in response to the detection of the pinch scroll gesture, execute instructions associated with the scrolling element for a scrolling action to cause a translational movement in simulated content; and
in response to the detection of the pinch scroll gesture, update the display instructions to cause the electronic display to generate the translational movement in the simulated content.
The system of claim 1 , wherein the console is further configured to: in response to the determination that the poses indicate that both the user's hands are performing a pinch gesture, each hand pinching one transform element of a simulated element in the simulated environment, the distance between the two hands changing over time, detect a transform gesture; in response to the detection of the transform gesture, execute a transform on the simulated element based on the positions of each hand, the transform of the simulated element changing a dimension of the simulated element; and in response to the detection of the pinch scroll gesture, update the display instructions to cause the electronic display to generate the transform of the simulated element.
in response to the determination that the poses indicate that both the user's hands are performing a pinch gesture, each hand pinching one transform element of a simulated element in the simulated environment, the distance between the two hands changing over time, detect a transform gesture;
in response to the detection of the transform gesture, execute a transform on the simulated element based on the positions of each hand, the transform of the simulated element changing a dimension of the simulated element; and
in response to the detection of the pinch scroll gesture, update the display instructions to cause the electronic display to generate the transform of the simulated element.
The system of claim 1 , wherein the console is further configured to: in response to the determination that the poses indicate that one of the user's hand is performing a pinch gesture on a simulated element in the simulated environment, the user's hand moving over time, detect a moving gesture; in response to the detection of the moving gesture, execute a move on the simulated element based on the position of the hand; and in response to the detection of the moving gesture, update the display instructions to cause the electronic display to generate the move of the simulated element.
in response to the determination that the poses indicate that one of the user's hand is performing a pinch gesture on a simulated element in the simulated environment, the user's hand moving over time, detect a moving gesture;
in response to the detection of the moving gesture, execute a move on the simulated element based on the position of the hand; and
in response to the detection of the moving gesture, update the display instructions to cause the electronic display to generate the move of the simulated element.
The system of claim 1 , wherein the console is further configured to: in response to the determination that the poses indicate that one of the user's hand is performing a pinch gesture using two fingers and a thumb, detect a selection button gesture, and in response to the detection of the selection button gesture, execute an action associated with the selection button gesture; and in response to the detection of the selection button gesture, update the display instructions to cause the electronic display to generate a change in simulated content presented to the user.
in response to the determination that the poses indicate that one of the user's hand is performing a pinch gesture using two fingers and a thumb, detect a selection button gesture, and
in response to the detection of the selection button gesture, execute an action associated with the selection button gesture; and
in response to the detection of the selection button gesture, update the display instructions to cause the electronic display to generate a change in simulated content presented to the user.
The system of claim 1 , wherein the console is further configured to: in response to the determination that the poses indicate the user's index finger and middle finger are extended, the user's ring finger positioned orthogonally to the user's thumb, and the thumb within a minimum distance to the ring finger, detect a lock gesture; and in response to the detection of the lock gesture, execute an input lock action to disable changes in the presentation of simulated elements to the user via the electronic display for those simulated elements associated with the input lock action.
in response to the determination that the poses indicate the user's index finger and middle finger are extended, the user's ring finger
[Truncated to input limit]