Office Action 1: Non-Final Rejection (2023-09-12)
Rejection types: 102 - Lack of Novelty, 103 - Obviousness, 112 - Written Description/Enablement

CLAIMS:
CLAIMS

What is claimed is:

1\. A computer system that is in communication with a display generation component and an optical sensor, the computer system comprising:
one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected;
while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture; and
in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is a first type of gesture, displaying, via the display generation component, an indication that the second user interface object is selected; and
in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected.

2\. The computer system of claim 1, wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor.

3\. The computer system of claim 1, wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture.

4\. The computer system of claim 1, wherein the one or more programs further include instructions for:
in response to detecting, via at least the optical sensor, the hand gesture: in accordance with a determination that the hand gesture is a third type of gesture that is different from the first type of gesture and the second type of gesture, performing an operation corresponding to selection of the first user interface object; and
in accordance with a determination that the hand gesture is a fourth type of gesture that is different from the first type of gesture, the second type of gesture, and the third type of gesture, displaying, via the display generation component, a menu that includes one or more selectable options.

5\. The computer system of claim 4, wherein:
before detecting the hand gesture, the computer system is in a first operation mode; and
performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode.

6\. The computer system of claim 5, wherein the one or more programs further include instructions for:
while the computer system is in the second operation mode, detecting, via at least the optical sensor, a second hand gesture; and
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that the second hand gesture is the first type of gesture, toggling a first automatic scrolling operation between an active state and an inactive state; and
in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the first automatic scrolling operation.

7\. The computer system of claim 6, wherein the one or more programs further include instructions for:
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that the second hand gesture is the third type of gesture, performing a second operation; and in accordance with a determination that the second hand gesture is the fourth type of gesture, performing a third operation that is different from the second operation.

8\. The computer system of claim 6, wherein the one or more programs further include instructions for:
while the computer system is in the second operation mode, performing a second automatic scrolling operation that scrolls a sequence of a plurality of interface objects in a second direction;
while performing the second automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects; and
in response to detecting the end of the sequence of the plurality of user interface objects, performing a third automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction.

9\. The computer system of claim 6, wherein the one or more programs further include instructions for:
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that the second hand gesture is the fourth type of gesture, transitioning the computer system from the second operation mode to a third operation mode.

10\. The computer system of claim 6, wherein the one or more programs further include instructions for:
in accordance with a determination that the second hand gesture is the first type of gesture, displaying a notification that indicates a state of the automatic scrolling operation.

11\. The computer system of claim 4, wherein the third type of gesture is a type of gesture that is multiple instances of the fourth type of gesture.

12\. The computer system of claim 4, wherein the fourth type of gesture does not include multiple instance of the first type of gesture and the third type of gesture does not include multiple instances of the second type of gesture.

13\. The computer system of claim 4, wherein the one or more programs further include instructions for:
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that a notification was received within a threshold period of time and in accordance with a determination that the second hand gesture is the fourth type of gesture, performing an action related to the notification.

14\. The computer system of claim 4, wherein the one or more selectable options includes a first selectable user interface object for changing operations that one or more hand gestures can cause to the computer system to perform, wherein selection of the first selectable user interface objects causes the computer system to display a plurality of settings, wherein each setting controls an operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system.

15\. The computer system of claim 4, wherein:
the one or more selectable options includes a second selectable option for transitioning the computer system to a fourth operation mode and a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode;
selection of the second selectable option causes the computer system to transition to the fourth operation mode; and
selection of the third selectable option causes the computer system to transition to the fifth operation mode.
16\. The computer system of claim 4, wherein the one or more selectable options include a fourth selectable option for displaying one or more additional selectable options, and wherein selection of the fourth selectable option causes the computer system to display the one or more additional selectable options that were not previously displayed before selection of the fourth selectable option.

17\. The computer system of claim 4, wherein the menu is displayed:
in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location; and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location.

18\. The computer system of claim 4, wherein the one or more programs further include instructions for:
while displaying the menu that includes one or more selectable options, detecting, via at least the optical sensor, a third hand gesture; and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that the third hand gesture is the fourth type of gesture, ceasing to display the menu that includes the one or more selectable options.

19\. The computer system of claim 1, wherein the one or more programs further include instructions for:
after displaying the user interface that includes the first user interface object, detecting, via at least the optical sensor, a fourth hand gesture; and
in response to detecting, via at least the optical sensor, the fourth hand gesture:
in accordance with a determination that the fourth hand gesture is a fifth type of gesture that is different from the first type of gesture and the second type of gesture, transitioning the computer system from an inactive state to an active state.

20\. The computer system of claim 1, wherein the one or more programs further include instructions for:
while displaying, via the display generation component, the indication that the second user interface object is selected, detecting, via at least the optical sensor, a fifth hand gesture; and
in response to detecting, via at least the optical sensor, the fourth hand gesture: in accordance with a determination that the fifth hand gesture is the first type of gesture, displaying, via the display generation component, an indication that a fourth first user interface object is selected; and
in accordance a determination that the fifth hand gesture is the second type of gesture, displaying, via the display generation component, the indication that the first user interface object.

21\. The computer system of claim 1, wherein the one or more programs further include instructions for:
while displaying the user interface that includes the first user interface object, detecting a request to transition the computer system from a first operation mode to a fourth operation mode;
in response to detecting the request to transition the computer system from the first operation mode to the fourth operation mode, transitioning the computer system from the first operation mode to the fourth operation mode;
while the computer system is in the fourth operation mode and while displaying the user interface that includes the first user interface object, the second user interface object, and the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a sixth hand gesture;
in response detecting, via least the optical sensor, the sixth hand gesture:
in accordance with a determination that the hand gesture is the first type of gesture, continuing to display the indication that the first user interface object is selected; and
in accordance with a determination that the hand gesture is the second type of gesture, continuing to display the indication that the first user interface object is selected.

22\. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system that is in communication with a display generation component and an optical sensor, the one or more programs including instructions for:
displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected; while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture; and
in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is a first type of gesture, displaying, via the display generation component, an indication that the second user interface object is selected; and
in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected.

23\. A method, comprising:
at a computer system that is in communication with a display generation component and an optical sensor:
displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected;
while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture; and
in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is a first type of gesture, displaying, via the display generation component, an indication that the second user interface object is selected; and
in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected. 106 107 108 109 110 111 112

OFFICE ACTION:
DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the Application filed on February 14, 2023.  Claims 1-23 are pending in the case.  Claims 1, 22, and 23 are the independent claims.
This action is non-final.
Claim Objections
Claim 14 is objected to because of the following informalities:  Claim 14 recites on line 3 “objects” where “object” was perhaps intended.  Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b)  CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 13 and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA  35 U.S.C. 112, the applicant), regards as the invention.
Claim 13 recites dependency upon claim 4, and further recites “the second hand gesture.”  This limitation lacks antecedent basis, since there is no previous recitation of “a second hand gesture” in claims 1 and 4.  Examiner notes that the first recitation of “a second hand gesture” appears to occur in claim 6.
Claim 20 recites dependency upon claim 1, and further recites, on line 5, “the fourth hand gesture.”  This limitation lacks antecedent basis, since there is no previous recitation of “a fourth hand gesture” in claim 1.  Examiner notes that the first recitation of “a fourth hand gesture” appears to occur in claim 19.
Claim 20 further recites, on lines 10-11, “displaying…the indication that the first user interface object.”  It cannot be determined whether “the indication that the first user interface object” is intended to refer to the “indication that the first user interface object is selected,” as is recited in claim 1, or if it is intended to refer to some other indication regarding the first user interface object.  Therefore, the limitation is indefinite.  In the interest of providing full examination on the merits, this limitation is interpreted as referring to the same “indication that the first user interface object is selected,” as is recited in claim 1.
Claim Rejections – 35 USC § 102
In the event the determination of the status of the application as subject to AIA  35 U.S.C. 102 and 103 (or as subject to pre-AIA  35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 3-5, 12, 14-16, and 18-23 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Osborn et al. (US 20230072423 A1).
With respect to claims 1, 22, and 23, Osborn teaches a computer system that is in communication with a display generation component and an optical sensor, the computer system comprising: one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method; a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system that is in communication with a display generation component and an optical sensor, the one or more programs including instructions for performing the method (e.g. paragraph 0257, supplementing sensor data with images of user captured by camera; paragraph 0286, auxiliary devices/sensors including imaging devices/cameras, heart rate monitors, etc.; paragraph 0288, Figs. 9A-B, wearable system with sensors such as EMG sensors, sensors 810; computer processor such as microcontroller 934 receiving inputs from sensors; communicating with dongle portion 920 and outputting processing output to dongle portion; paragraph 0289, dongle portion 920 receiving signals from wearable portion 910, and providing them to host computer for further processing, display, effecting control of objects, etc.; paragraph 0292, describing Fig. 1, system including processors 112 programmed to communicate with sensors 110 and executing algorithms; paragraph 0301, system including display controller configured to display visual representations; computer processors implementing trained inference models that receive sensor signals as input and provide information to generate control signals for system as output; paragraph 0302, user interface; paragraph 0303, various types of devices in which processors may be included; paragraph 0304, multiple processors; paragraph 0306, system including processors, camera, and display; paragraph 310, camera information/cameras; paragraph 0311, camera mounted on user; paragraph 0344, embodiments implemented in software executed on processors; paragraph 0345, non-transitory computer-readable storage medium/memory encoded with computer program/instructions executed on processor to perform described functions; paragraph 0352, sensor including light emitter/detector; paragraph 0385, Fig. 13, vibrotactile system 1300 may be standalone system, or may be configured with interaction with various types of other devices, and includes processor 1360; paragraph 610, cameras; paragraph 0729, photoplethysmography/PPG sensors detecting vascular changes/changes in blood volume; paragraph 1384, Fig. 35B, wearable device 35102 capable of controlling computer devices via neuromuscular signals of users, including set of sensors 35104 that detect/sense neuromuscular signals traversing body of user; processing device 35106 communicatively coupled to sensors, and memory 35108 including models and computer-executable instructions that when executed by processor cause it to identify force generated, generate commands for HCI system, etc.; i.e. the includes at least one processor and memory storing instructions for execution by the processor, and is further in communication with at least one display/display controller/generation component, and at least one optical sensor, such as a camera (which may be mounted on the user)); and the method, comprising:
displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected (e.g. paragraph 1460, describing radial menu 351000 in Figs. 35O-V, where the menu may include visual indicators including a selection indicator 351020; paragraph 1461, selection indicator 351020 may indicate a currently selected action indicator; paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual, auditory, or haptic feedback; visual feedback representing change of a characteristic of visual element within radial menu in response to selection of the visual indicator, such as position change, orientation change, color change, size change, transparency change, fill change, emphasis change, shadow change, animation change, font change, line type change, line width change, combinations, etc.; paragraph 1473, describing Figs. 36B-C, radial menu 351110 including a set of action indicators including indicators 351111 and 351113 displayed in GUI along arc, where as shown in Fig. 36B includes an indication that indicator 35111 is selected; demonstrating/confirming selection of visual indicator using visual feedback; paragraph 1475, describing Figs. 36D-E, displaying sequential menu 351120 including a sequence of visual indicators depicted as A through F, where as shown in Fig. 36 D includes in indication that indicator D/351123 is selected; i.e. both of the menus/interfaces shown in the cited figures include at least first, second, and third UI objects along with an indication that an object of these is selected, such as a visual feedback including a shading/coloration of a selected indicator; paragraphs 1385-1387, determining based on neuromuscular signals state of body part of user and generating input commands accounting for state of user’s body part; sensors constitute any type of form of sensor capable of detecting/sensing neuromuscular signals in user’s body including EMG sensors, MMG sensors, SMG sensors; sensors may also include auxiliary sensors such as IMU, position tracking, imaging (camera), and radiation based sensors, microphones, heart rate monitors, etc.; paragraph 1394, wearable device 35102 coupled with computing system/display; user placing body in state/condition to control/modify presentation/performance of computing system, generating neuromuscular signals representative/indicative/suggestive of state/action);
while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture (e.g. paragraph 0310, camera information used to improve interpretation of neuromuscular signals; camera used to record image of user, may be within particular range/wavelengths of light, images produced by visible/non-visible light, including showing heat distribution in user’s body, providing indication of blood flow within user and used to infer condition of user such as forces being exerted by fingers; paragraph 0311, camera mounted on user; paragraph 0312, camera data in addition to neuromuscular activity signals used to reconstruct positions, geometries, and forces being applied by user’s arm/hand; paragraph 0323, auxiliary signals from auxiliary devices such as camera supplementing neuromuscular signals; paragraph 610, cameras used to improve interpretation of signals; paragraph 0611, camera data used to provide real time determination of musculoskeletal representations of gestures; paragraph 0729, PPG sensors detecting vascular changes; paragraph 1271, camera/imaging data; paragraph 1387, sensors including auxiliary sensors such as cameras; paragraph 1395, user pose/gesture generating neuromuscular signals as they traverse down arm toward hand; sensors 35104 detecting/sensing neuromuscular signals and generating data representative of the signals; paragraph 1396-1398, sensor data undergoing processing, analyzing data to determine state of body parts of user, producing/outputting classification indicating state of body part; paragraph 1398, gestures; paragraph 1400, causing computing system to perform actions mapped to state of body parts; paragraph 1401, describing various actions mapped to different states of user’s body parts; paragraph 1444, capturing gestures with camera; photographic data of hand gestures; paragraph 1473, body state mapped to selection of next indicator in a given sequence; identification of body state; paragraph 1474, body states including wrist extension, flexion, rotation, radial deviation, ulnar deviation, etc.; paragraph 1475, body states mapped to selection of visual indicators in sequence; identification of body states; i.e. detecting a body state of the user, such as a hand gesture, including by using an optical sensor such as a camera,  PPG, etc., possibly in combination with other sensors); and
in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is a first type of gesture, displaying, via the display generation component, an indication that the second user interface object is selected (e.g. paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual feedback; paragraph 1473, Figs. 36B-C, first body state mapped to selection of the next indicator in a given sequence; in response to identification of first body state, deselecting indicator 35111 and selecting indicator 351113; demonstrating/confirming selection of visual indicator using visual feedback; paragraph 1475, Figs. 36D-E, first body state mapped to selection of next visual indicator in the sequence; i.e. in response to detecting a first type of gesture, as indicated by a first body state, the indication that a first UI object is selected may be changed to an indication that a second UI object is selected, such as by moving the indicator of the selection to the second UI object, where the second UI object is a next UI object in a sequence with respect to the first UI object); and
in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected (e.g. paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual feedback; paragraph 1473-1474, Figs. 36B-C, first body state mapped to selection of next indicator in given sequence; demonstrating/confirming selection of visual indicator using visual feedback; first and second body states may represent counterparts of one another; 1475, Figs. 36D-E, second body state mapped to selection of previous visual indicator in the sequence; in response to identification of second body state, deselecting subsequent indicator 351123 and selecting prior indicator 351121; i.e. in response to detecting a second/different type of gesture, as indicated by a second body state, the indication that a first UI object is selected may be changed to an indication that a third UI object is selected, such as by moving the indicator of the selection to the third UI object, where the third UI object is a previous UI object in a sequence with respect to the first UI object; in addition, because these gestures cause selection to move to a next/previous object in sequence, receipt of multiple of either of these gestures may also cause a different/third UI object to be selected; for example, where multiple instances/two instances of the first type of gesture are received, the object selected as indicated may be first moved from the first UI object to the second/next UI object in the sequence, and then moved again from the second/next UI object to a third/subsequent UI object in the sequence).
With respect to claim 3, Osborn teaches all of the limitations of claim 1 as previously discussed, and further teaches wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture (e.g. paragraph 1473-1474, Figs. 36B-C, first body state mapped to selection of next indicator in given sequence; in response to identification of first body state, deselecting indicator 35111 and selecting indicator 351113; first and second body states may represent counterparts of one another; 1475, Figs. 36D-E, first body state mapped to selection of next visual indicator in the sequence; i.e. because first body state/gesture causes selection to move to a next object in sequence, receipt of multiple instances of this gesture may also cause a different/third UI object to be selected; for example, where multiple instances/two instances of the first type of gesture are received, the object selected as indicated may be first moved from the first UI object to the second/next UI object in the sequence, and then moved again from the second/next UI object to a third/subsequent UI object in the sequence).
With respect to claim 4, Osborn teaches all of the limitations of claim 1 as previously discussed, and further teaches wherein the one or more programs further include instructions for:
in response to detecting, via at least the optical sensor, the hand gesture (as cited above, using camera/sensors to detect hand gesture):
in accordance with a determination that the hand gesture is a third type of gesture that is different from the first type of gesture and the second type of gesture, performing an operation corresponding to selection of the first user interface object (e.g. paragraph 1468, Fig. 35Q, illustrating finger pinch pose 351006, which may be mapped to command/instruction to click visual indicator in the radial menu, analogous to mouse button click or keypress; alternatively combination of finger pinch pose and certain degree of force mapped to click command; paragraph 1469, Fig. 35 U, clicking/engaging with or activating function of the currently selected action indicator; clicking action indicator in response to identified click gesture; upon performing the click action, causing visual indicator to appear to depress and/or release, similar to a physical button); and
in accordance with a determination that the hand gesture is a fourth type of gesture that is different from the first type of gesture, the second type of gesture, and the third type of gesture, displaying, via the display generation component, a menu that includes one or more selectable options (e.g. paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user (as compared to other movements/poses causing item selection/clicking); paragraph 1485, displaying interface 351300 of Fig. 36G, which may be a popup box/dialog that appears in response to user input; i.e. where a some suitable user input, such as a fist clenching gesture, may be used to open various menus, such as a menu including selectable options).
With respect to claim 5, Osborn teaches all of the limitations of claim 4 as previously discussed, and further teaches wherein:
before detecting the hand gesture, the computer system is in a first operation mode (e.g. paragraph 1463, Fig. 35O, fist pose mapped to command to display, enter, and activate radial menu; i.e. where in the first state, at the time the radial menu is displayed, the user has provided a previous input causing the radial menu to be activated, and therefore in an operation mode in which the menu is activated, analogous to a first operation mode); and
performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode (e.g. paragraph 1468, Fig. 35Q, illustrating finger pinch pose 351006, which may be mapped to command/instruction to click visual indicator in the radial menu, analogous to mouse button click or keypress; alternatively combination of finger pinch pose and certain degree of force mapped to click command; paragraph 1469, Fig. 35 U, clicking/engaging with or activating function of the currently selected action indicator; clicking action indicator in response to identified click gesture; upon performing the click action, causing visual indicator to appear to depress and/or release, similar to a physical button; i.e. clicking the currently selected action indicator causes the corresponding function to be activated, and therefore causes the system to transition to an operation mode in which the corresponding function has been activated, analogous to a second operation mode).
With respect to claim 12, Osborn teaches all of the limitations of claim 4 as previously discussed, and further teaches wherein the fourth type of gesture does not include multiple instance of the first type of gesture and the third type of gesture does not include multiple instances of the second type of gesture (e.g. paragraph 0620, finger/hand gesture combined with force, user clenching first, etc., to open menu/dialog; paragraph 1468, finger pinch pose used to click selected item; paragraph 1381, first pose to display menu; paragraph 1474, first and second body states may be counterparts of one another; body states including wrist extension, flexion, rotation, radial deviation, ulnar deviation, etc.; i.e. where a fist clenching or other user input used for opening a menu, analogous to a fourth type of gesture does not include multiple instances of a first body state such as wrist extension, flexion, rotation, deviation, etc., analogous to a first type of gesture, and where a finger pinch pose for clicking a selected item, analogous to a third type of gesture, does not include multiple instances of a second body state such as wrist extension, flexion, rotation, deviation, etc., counterpart to a first body state, analogous to a second type of gesture ).
With respect to claim 14, Osborn teaches all of the limitations of claim 4 as previously discussed, and further teaches wherein the one or more selectable options includes a first selectable user interface object for changing operations that one or more hand gestures can cause to the computer system to perform, wherein selection of the first selectable user interface objects causes the computer system to display a plurality of settings, wherein each setting controls an operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system (e.g. paragraph 1483, Fig. 36S, multi-state user interface configurable/programmable by user; user able to specify mappings between body states and responsive actions; such as mapping body part sates of ring/middle/pinky/index finger pinches, first pose, and open hand pose to scroll up/down, activate/deactivate link, increment/decrement selector actions; paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H; paragraph 1486, Fig. 36I, user modifying mappings between body states and actions; configuring web navigation settings by selecting setting options shown in drop-down menu of web browser; user clicking and selecting “scroll:options” to configure type of pose that initiates scrolling control or feature; paragraph 1487, Fig. 36J, link activate setting used to highlight links included in web page; user clicking on “linksActivate:options” to configure type of pose and predetermined pose duration that initiates the activation and rendering of link displayed in webpage; paragraph 1488, Fig. 36K, user clicking on “click:options” to configure type of pose and predetermined duration that initiates a click action in an application; i.e. selection of the “Control Options” dropdown object of Fig. 36G causes the plurality of settings to be shown in Fig. 36H, where these settings allow control of which gestures/poses are mapped to which operations).
With respect to claim 15, Osborn teaches all of the limitations of claim 4 as previously discussed, and further teaches wherein:
the one or more selectable options includes a second selectable option for transitioning the computer system to a fourth operation mode and a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode (e.g. paragraph 1485, Fig. 36G, multi-state user interface facilitates enabling and disabling body state control using “enabled” button; drop down menu control from which user is able to select certain display settings and map body states to responsive actions; see also “reset settings” button in Fig. 36G;  i.e. where an “enabled” button can be selected to transition the system to a state/mode in which the body state controls are enabled, and to a state/mode in which body state controls are disabled, analogous to at least a second selectable option for transitioning the computer system to a fourth operation mode (i.e. a body state controls disabled mode); further where the user selects the drop down menu and modifies the mappings of body states and responsive actions, this causes the system to be transitioned from a first mode (i.e. a mode in which a current set of body state/action mappings are utilized) to a different/fifth mode (i.e. a mode in which a modified/custom set of body state/action mappings are utilized), analogous to a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode; in addition, where body state/action mappings are already customized/modified, and the user selects the option to “reset settings” this would cause the system to be transitioned from a first mode (i.e. a mode in which a current set of modified/customized body state/action mappings are utilized) to a different/fifth mode (i.e. a mode in which a default/reset set of body state/action mappings are utilized), analogous to a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode);
selection of the second selectable option causes the computer system to transition to the fourth operation mode (e.g. paragraph 1485, Fig. 36G, enabling/disabling body state control in using “enabled” button, as discussed above); and
selection of the third selectable option causes the computer system to transition to the fifth operation mode (e.g. paragraph 1485, Fig. 36G, user selecting drop down menu control to map body states to responsive actions; user selecting to “reset settings”, as discussed above).
With respect to claim 16, Osborn teaches all of the limitations of claim 4 as previously discussed, and further teaches wherein the one or more selectable options include a fourth selectable option for displaying one or more additional selectable options, and wherein selection of the fourth selectable option causes the computer system to display the one or more additional selectable options that were not previously displayed before selection of the fourth selectable option (e.g. paragraph 1483, Fig. 36S, multi-state user interface configurable/programmable by user; user able to specify mappings between body states and responsive actions; such as mapping body part sates of ring/middle/pinky/index finger pinches, first pose, and open hand pose to scroll up/down, activate/deactivate link, increment/decrement selector actions; paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H; paragraph 1486, Fig. 36I, user modifying mappings between body states and actions; configuring web navigation settings by selecting setting options shown in drop-down menu of web browser; user clicking and selecting “scroll:options” to configure type of pose that initiates scrolling control or feature; paragraph 1487, Fig. 36J, link activate setting used to highlight links included in web page; user clicking on “linksActivate:options” to configure type of pose and predetermined pose duration that initiates the activation and rendering of link displayed in webpage; paragraph 1488, Fig. 36K, user clicking on “click:options” to configure type of pose and predetermined duration that initiates a click action in an application; i.e. selection of the “Control Options” dropdown object of Fig. 36G, analogous to a fourth selectable option, causes additional selectable options, not previously displayed, to be displayed, as shown in Fig. 36H).
With respect to claim 18, Osborn teaches all of the limitations of claim 4 as previously discussed, and further teaches wherein the one or more programs further include instructions for:
while displaying the menu that includes one or more selectable options, detecting, via at least the optical sensor, a third hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 1470, Fig. 35 R, open hand pose; i.e. while a menu is being displayed, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that the third hand gesture is a type of gesture, ceasing to display the menu that includes the one or more selectable options (e.g. paragraph 0509, mapping neuromuscular signals to control signals to control device based on type of gesture; paragraph 0618, determining the gesture being performed; paragraph 1381, open hand pose may cause the wearable device to direct the computing device to close the menu; paragraph 1380, mappings causing device to perform action in response to detection of specific body part states; paragraph 1470, open hand pose mapped to command to hide, exit, or deactivate menu; following identification of open hand pose, some or all of menu may be obscured or hidden from view within the GUI; i.e. when the system determines, based on a mapping body part states/gestures to actions, that the user is performing an open hand gesture or other suitable gesture, the system may perform a corresponding action of ceasing to display a currently displayed menu).
Osborn does not explicitly disclose that the type of gesture is a fourth type of gesture.  However, Kim teaches that the type of gesture is a fourth type of gesture (e.g. paragraph 0256, Figs. 17A-B, while message check screen outputted through display, if command for switching to lock screen is received, stopping output of message check screen and outputting/switching to lock screen; command may include first input (which is shown as a clenched fist gesture in Figs. 17A-B); i.e. where a user performing a fist gesture to initiate/switch to a lock screen display would cause the system to cease displaying all currently displayed content, including menus).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability, while displaying user interface content (i.e. such as a GUI including currently displayed menus, as taught by Osborn) to receive a hand gesture to transition to a lock screen state/mode, such as in response to a clenched fist hand gesture, and in response, cease displaying currently displayed content (such as the menu of Osborn) and display a lock screen (i.e. as taught by Kim).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claim 19, Osborn teaches all of the limitations of claim 1 as previously discussed, and further teaches wherein the one or more programs further include instructions for:
after displaying the user interface that includes the first user interface object, detecting, via at least the optical sensor, a fourth hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, fist clenching; paragraph 1463, Fig. 35O, fist pose; i.e. while/after the menu including the first user interface object is being/has been previously displayed, or at some later time, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the fourth hand gesture:
in accordance with a determination that the fourth hand gesture is a fifth type of gesture that is different from the first type of gesture and the second type of gesture, transitioning the computer system from an inactive state to an active state (e.g. paragraph 0509, mapping neuromuscular signals to control signals to control device based on type of gesture; paragraph 0618, determining the gesture being performed; paragraph 0620, fist clenching by user results in wake signal being sent to the system; paragraph 0703, first muscular activation state detected used to determine that a wake up mode is to be controlled, and second muscular activation state used to second activation state used to control initialization operation of the wake up mode; paragraph 0706, wake up/initialization mode; paragraph 1380, mappings causing device to perform action in response to detection of specific body part states; paragraph 1463, Fig. 35O, fist pose mapped to command to display, enter, and activate radial menu; i.e. when the system determines, based on a mapping body part states/gestures to actions, that the user is performing a fist pose hand gesture or other suitable gesture, the system may perform a corresponding action of activating a menu, or waking the system from an inactive/shut down state, analogous to transitioning the system from an inactive state to an active state).
With respect to claim 20, Osborn teaches all of the limitations of claim 1 as previously discussed, and further teaches wherein the one or more programs further include instructions for:
while displaying, via the display generation component, the indication that the second user interface object is selected, detecting, via at least the optical sensor, a fifth hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraphs 1473-1475, first body state mapped to selection of next indicator in given sequence; first and second body states may represent counterparts; first body state mapped to selection of next visual indicator in sequence, second body state mapped to selection of previous visual indicator in sequence; i.e. while the interface/menu including the first, second, and third objects is being displayed, including after the user has performed a gesture causing a second user interface object to be selected and while the second user interface object is displayed with an indication that it is selected, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the fourth hand gesture:
in accordance with a determination that the fifth hand gesture is the first type of gesture, displaying, via the display generation component, an indication that a fourth first user interface object is selected (e.g. paragraphs 1473-1475, first body state mapped to selection of next indicator in given sequence; in response to identification of first body state, deselecting indicator 351111 and selecting indicator 351113, providing visual feedback; Figs. 36B-E; i.e., where the user has already performed the first type of gesture once, causing a second user interface object to be selected as shown in Fig. 36C, the user may perform another instance of the first type of gesture, which is mapped to selection of a next indicator, such that after this second instance of the first type of gesture is performed, a next/subsequent indicator (with respect to currently selected indicator 351113, such as the next indicator below it) will be selected, and indicated as selected via visual feedback, analogous to displaying an indication that a fourth UI object is selected in accordance with a determination that the fifth hand gesture is the first type of gesture (Examiner notes that a similar rationale would be applicable in the interfaces of Figs. 36D-E, such that if a user performs a first instance of the first type of gesture to cause item “E” to be selected as the next in sequence, and then performs a second instance of the first type of gesture, this will cause the item “F” to be selected as the next item in the sequence, and indicated as such)); and
in accordance a determination that the fifth hand gesture is the second type of gesture, displaying, via the display generation component, the indication that the first user interface object (paragraphs 1473-1475, first and second body states may represent counterparts; second body state mapped to selection of previous visual indicator in sequence; i.e., where the user has already performed the first type of gesture, causing a second user interface object to be selected as shown in Fig. 36C, the user may perform the second type of gesture, which is mapped to selection of a previous indicator, such that after this second type of gesture is performed, a previous indicator (with respect to currently selected indicator 351113, such as the previous indicator above it) will be selected, and indicated as selected via visual feedback (i.e. the selection will be reverted back to the state shown in Fig. 36B), analogous to displaying an indication that the first UI object is selected in accordance with a determination that the fifth hand gesture is the second type of gesture (Examiner notes that a similar rationale would be applicable in the interfaces of Figs. 36D-E, such that if a user performs a first instance of the first type of gesture to cause item “E” to be selected as the next in sequence, and then performs the second type of gesture, this will cause the item “D” to be selected as the previous item in the sequence, and indicated as such, similar to what is shown in Fig. 36D)).
With respect to claim 21, Osborn teaches all of the limitations of claim 1 as previously discussed, and further teaches wherein the one or more programs further include instructions for:
while displaying the user interface that includes the first user interface object, detecting a request to transition the computer system from a first operation mode to a fourth operation mode (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 1470, Fig. 35 R, open hand pose, mapped to command to deactivate the menu; i.e. while the menu is being displayed, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected, including a gesture to request transitioning from a first operation mode (such as one in which a menu is activated) to a fourth operation mode (such as one in which the menu is deactivated/disabled));
in response to detecting the request to transition the computer system from the first operation mode to the fourth operation mode, transitioning the computer system from the first operation mode to the fourth operation mode (e.g. paragraph 1461, disabled radial menu; paragraph 1471, deactivating radial menu; i.e. in response to detecting the user’s open hand gesture, the radial menu is disabled/deactivated, transitioning from an operation mode in which the radial menu is activated (analogous to a first operation mode) to an operation mode in which the radial menu is deactivated (analogous to a fourth operation mode));
while the computer system is in the fourth operation mode and while displaying the user interface that includes the first user interface object, the second user interface object, and the third user interface object, and the indication that the first user interface object is selected (e.g. paragraph 1471, state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; i.e. even after the radial menu is deactivated, all user interface objects, and selection indicator (indicating which object is selected) may continue to be displayed on the display), detecting, via at least the optical sensor, a sixth hand gesture (e.g. continuously receiving neuromuscular/musculoskeletal signals and continuously performing corresponding gesture recognition as cited above);
in response detecting, via least the optical sensor, the sixth hand gesture:
in accordance with a determination that the hand gesture is the first type of gesture, continuing to display the indication that the first user interface object is selected (e.g. paragraph 1411, user rotating arm without making fist pose has not effect on the radial menu; paragraph 1461, disable radial menu is a mode in which the user is not able to interact with the radial menu; paragraph 1470, deactivated radial menu; paragraph 1471 state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; radial menu will not recognize/respond to subsequent selection (wrist gesture) or click command until receiving a subsequent activation command); i.e. after the radial menu is deactivated, all user interface objects including the selection indicator are continuously displayed and, while in the deactivated state, the user’s hand gestures, such as a first type of gesture to select a next object, will have no effect, and will not be recognized/responded to, such that the menu will continue to be displayed including the selection indicator (indicating a current action indicator as selected) in an unchanged state); and
in accordance with a determination that the hand gesture is the second type of gesture, continuing to display the indication that the first user interface object is selected (e.g. paragraph 1411, user rotating arm without making fist pose has not effect on the radial menu; paragraph 1461, disable radial menu is a mode in which the user is not able to interact with the radial menu; paragraph 1470, deactivated radial menu; paragraph 1471 state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; radial menu will not recognize/respond to subsequent selection (wrist gesture) or click command until receiving a subsequent activation command); i.e. after the radial menu is deactivated, all user interface objects including the selection indicator are continuously displayed and, while in the deactivated state, the user’s hand gestures, such as a second type of gesture to select a previous object, will have no effect, and will not be recognized/responded to, such that the menu will continue to be displayed including the selection indicator (indicating a current action indicator as selected) in an unchanged state).
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims under pre-AIA  35 U.S.C. 103(a), the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were made absent any evidence to the contrary.  Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and invention dates of each claim that was not commonly owned at the time a later invention was made in order for the examiner to consider the applicability of pre-AIA  35 U.S.C. 103(c) and potential pre-AIA  35 U.S.C. 102€, (f) or (g) prior art under pre-AIA  35 U.S.C. 103(a).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Chung et al. (US 20220326779 A1).
With respect to claim 2, Osborn teaches all of the limitations of claim 1 as previously discussed.  Although Osborn teaches that auxiliary sensors which may contribute to gesture detection may include heart rate sensors, PPG sensors, and cameras used to detect changes in blood flow within the user (e.g. paragraph 0281, sensors including cardio sensor to measure user’s pulse, heart rate; paragraph 0286, auxiliary device/sensor including heart-rate monitor;; paragraph 0310, nonvisible light used to capture image of heat distribution, providing indication of blood flow, used to infer condition of user such as forces being exerted by certain fingers; paragraph 0729, PPG sensors detecting vascular changes; ), Osborn does not explicitly disclose wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor.
However, Chung teaches wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor (e.g. paragraph 0093, sensor module sensing information for movement of electronic device, and may include photoplethysmogram (PPG) sensor; paragraph 0098, recognizing OCO and shake gestures, gestures of turning hand, moving hand with palm open, lifting arm, moving thumb, moving up/down hand, etc.; gesture may be recognized based on user’s biometric signal (e.g. a signal indicating relaxation/contraction of blood vessels or a heartbeat signal); paragraph 0104, gesture application invoking sensor API to activate sensors, such as PPG sensor; gesture application identifying information for gestures; paragraph 0136, OCO gesture, PPG sensor detecting change due to relaxation and contraction of blood vessels when user opens and closes her hand; ).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Chung in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), to incorporate the teachings of Chung (directed to recognizing gestures) to include the capability to detect the hand gesture based on heart rate data detected via the optical sensor, such as by detecting a variety of gestures using a PPG sensor, based on a user biometric signal indicating relaxation/contraction of blood vessels or heartbeat (as taught by Chung).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability to recognize a gesture when there is a gesture recognition request from a specific application, eliminating the need for sensors to stay on and the gesture recognition process to keep on running, reducing power consumption and resource waste, and in order to make it possible to immediately recognize relatively small gestures, as described in Chung (paragraphs 0014-0015).
Claims 6, 7, 9-11, and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Kim et al. (US 20160029899 A1).
With respect to claim 13, Osborn teaches all of the limitations of claim 4 as previously discussed, and further teaches wherein the one or more programs further include instructions for:
in response to detecting, via at least the optical sensor, the second hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected):
in accordance with a determination that the second gesture is the fourth type of gesture, performing an action (e.g. paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user); and
in accordance with a determination that a notification was received and in accordance with a determination that the second hand gesture is a type of gesture, performing an action related to the notification (e.g. paragraph 1649 and corresponding table, using swipe to engage/ignore notification; paragraphs 1651-1653, flick gesture may be used as dismissal mechanism for notifications; notifying user that next meeting is coming up, flick dismissing notification outright, index pinch tap engaging timer function, rolling wrist to select timer duration, etc.).
Osborn does not explicitly disclose that the determinations (for performing the action related to the notification) are that the notification was received within a threshold period of time and that the second hand gesture is the fourth type of gesture.  However, Kim teaches in accordance with a determination that a notification was received within a threshold period of time and in accordance with a determination that the second hand gesture is the fourth type of gesture, performing an action related to the notification (e.g. paragraph 0014, determining level of folding fingers of the user as being in various states (i.e. gestures) based on measured blood flow rate; paragraph 0186, Fig. 5A, user determined in sleep state, generated notifications controlled to not be outputted to user; when user wakes up, user can be informed of saved notifications; paragraph 0187, after detecting user has got out of sleep state, if prescribed command (such as making a fist twice) is received, outputting weather information through the display; paragraph 0190, if user wakes from sleep state, outputting notification screen 501 for cumulative notifications (notification list of plurality of notifications) to user, if prescribed command is received, such as input of making a fist twice, as shown in Figs. 5A-C; i.e. where a notification is received within a time period during which the user is sleeping (analogous to a threshold period of time) and then an input of making a fist gesture is received, an action related to the notification is performed, such as displaying the notification; paragraph 0227, Fig. 11A, call signal currently received, displaying call screen 1101 indicating call signal is currently being received; paragraph 0229, Figs. 11A-C, when call signal currently received, if first input maintained over prescribed time, making response to call signal, and outputting call connected screen 1103; i.e. as shown in Figs. 11A-C, if, during the time period that a call signal is being currently received and a corresponding notification screen is displayed (e.g., during the time period beginning when the call signal is first received and ending when the signal is no longer received, analogous to a threshold period of time within which a notification is received), if a user makes an input such as holding a fist for a time period, the call is answered, analogous to performing an action (answering the call) related to the notification (the screen indicating the call is being received)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to detect the fourth type of gesture (i.e. of Osborn, such as a fist gesture for causing a menu to be displayed) and, based on determining that a received gesture is the fourth type of gesture/fist gesture and further determining that a notification has been received within a threshold time period (such as during a time period in which the user was sleeping, or during a time period in which a call signal is currently received and a corresponding notification is displayed), performing a related action, such as displaying the notification, or answering a corresponding phone call, etc., as taught by Kim.  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claim 6, Osborn teaches all of the limitations of claim 5 as previously discussed, and further teaches wherein the one or more programs further include instructions for:
detecting, via at least the optical sensor, a second hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that the second hand gesture is one type of gesture, toggling a first automatic scrolling operation between an active state and an inactive state (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling up a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll up a GUI; paragraph 1484, ring finger pinch mapped to scroll up so that when user makes ring finger pinch pose, the page/browser displayed scrolls up); and
in accordance with a determination that the second hand gesture is another type of gesture, reversing a first direction of the first automatic scrolling operation  (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling down a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll down a GUI; paragraph 1484, pinky finger pinch mapped to scroll down so that when user makes pinky finger pinch pose, the page/browser displayed scrolls down).
Although Osborn further teaches that user hand gestures may be mapped to different commands in different states/modes, and further teaches that a wrist roll gesture may be mapped to a scroll command (e.g. paragraph 0648, switching between modes; paragraph 0666-0667, selection of mode, such as from list or using gesture; paragraph 1380, transitioning between different mappings of body part states and responsive actions; paragraph 1408-1410, activating first mapping and/or deactivating second mapping between possible body part states and responsive actions; user switching between mappings; transition/switch between mappings; paragraph 1486, user modifying mappings between body states and actions, user configuring/mapping scroll controls to different poses; table accompanying paragraph 1649, indicating that an interaction with a list feature may include a “scroll through” interaction with a “wrist roll” control scheme), Osborn does not explicitly disclose while the computer system is in the second operation mode, detecting the second hand gesture, or the determinations that the second hand gesture is the first type of gesture or the second type of gesture.
However, Kim teaches while the computer system is in the second operation mode, detecting the second hand gesture, or the determinations that the second hand gesture is the first type of gesture or the second type of gesture (e.g. paragraphs 0251-254, Figs. 16A-C, outputting notification screen of received text message, receiving message check command such as folding input and outputting message check screen 1601 for displaying details of received text message; scroll action on message check screen performed by rotation of wrist; wrist rotated in first direction, scrolling to move outputted message check screen; direction for rotating wrist includes scroll direction; if scroll in bottom direction desired, rotating wrist in bottom direction; i.e. where a user selection of an indicator regarding a received message may transition the system to a second operation mode, such as a mode in which the corresponding functionality is executed and corresponding content is displayed, and in this second mode, the user may provide a wrist rotation input in a first or second direction in order to perform a scroll operation in a designated direction, as shown between Fig. 16B and Fig. 16C).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to place the system in a second mode (i.e. such as a mode of executing a corresponding functionality following a user’s selection of a displayed item, as taught by Osborn, and a mode of executing received message display and interaction following a user’s selection of a displayed message notification, as taught by Kim), and while in this mode receive a hand gesture to initiate automatic scrolling in a direction, and a hand gesture to reverse the scrolling direction (i.e. such as using different fingers to perform pinch gestures to scroll up/down as taught by Kim), where the hand gesture may be a wrist rotation gesture in different directions corresponding to the desired scroll direction (i.e. as taught by Kim, where Osborn also teaches the use of wrist rotation for at least scrolling a wrist, and further teaches that the wrist rotation may be performed in two different directions).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claim 7, Osborn in view of Kim teaches all of the limitations of claim 6 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
in response to detecting, via at least the optical sensor, the second hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected):
in accordance with a determination that the second hand gesture is the third type of gesture, performing a second operation (e.g. paragraph 1450, pinch pose 35700 mapped to activation action such as action 351900 in Fig. 36M; when user executes pinch pose for duration, directing/causing cursor 35902 to move toward element 351906 in GUI of corresponding application and possibly activate the element; paragraph 1451, providing feedback indicators, such as informing user that system has detected the pinch pose; paragraph 1455, pinch pose mapped to scrolling action such that when user performs pinch pose, the corresponding application scrolls GUI; i.e. if the system receives a second gesture which is a pinch pose, the system may perform some second operation, such as moving a cursor to a particular object, providing feedback indicators, scrolling a GUI etc.); and
in accordance with a determination that the second hand gesture is the fourth type of gesture, performing a third operation that is different from the second operation (e.g. paragraph 1270, switching between different types of control schemes/input modes, using fist to control virtual pointer in 2D; paragraph 1281, fist squeeze used for discrete event such as to engage or disengage a particular feature within a given application or a continuous event, such as to maintain activation of a particular feature within a given application; paragraph 1283, user selecting from options using hand gesture such as clenching fist; paragraph 1443, determining fist formed/held by user hand, controlling speed of cursor based on amount of force applied; paragraph 1539, using fist pose to delete characters, etc.; i.e. if the system receives a second gesture which is a fist pose, the system may perform some third operation such as controlling a virtual pointer/cursor and/or altering its speed, engaging or disengaging a feature, maintaining activation of a feature, selecting an option, deleting characters, etc.).
With respect to claim 9, Osborn in view of Kim teaches all of the limitations of claim 6 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
in response to detecting, via at least the optical sensor, the second hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected):
in accordance with a determination that the second hand gesture is a type of gesture, transitioning the computer system from the second operation mode to a third operation mode (e.g. paragraph 0648, switching between modes; paragraph 0666-0667, user selection of mode, such as from list or using gesture; paragraph 1380, transitioning between different mappings of body part states and responsive actions, such as according to neuromuscular signals/gestures; paragraph 1408-1410, activating first mapping and/or deactivating second mapping between possible body part states and responsive actions; user switching between mappings; transition/switch between mappings, such as based on detected states of body/gestures).
Osborn does not explicitly disclose in accordance with a determination that the second hand gesture is the fourth type of gesture, transitioning the computer system from the second operation mode to a third operation mode.  However, Kim teaches in accordance with a determination that the second hand gesture is the fourth type of gesture, transitioning the computer system from the second operation mode to a third operation mode (e.g. paragraph 0256, Figs. 17A-B, while message check screen outputted through display, if command for switching to lock screen is received, stopping output of message check screen and outputting/switching to lock screen; command may include first input (which is shown as a clenched fist gesture in Figs. 17A-B)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to place the system in a second mode (i.e. such as a mode of executing a corresponding functionality following a user’s selection of a displayed item, as taught by Osborn, and a mode of executing received message display and interaction following a user’s selection of a displayed message notification, as taught by Kim), and while in this mode receive a hand gesture to transition to a lock screen state/mode, such as in response to a clenched fist hand gesture (i.e. as taught by Kim).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claim 10, Osborn in view of Kim teaches all of the limitations of claim 6 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
in accordance with a determination that the second hand gesture is one type of gesture, displaying a notification that indicates a state of the automatic scrolling operation (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling up a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll up a GUI; paragraph 1484, ring finger pinch mapped to scroll up so that when user makes ring finger pinch pose, the page/browser displayed scrolls up; paragraph 1459, providing visual feedback of both identified pose and identified force; when action includes scrolling GUI, displaying cursor in connection with scrolling, which may be presented as a horizontal line with a bar extending above or below the line, depending on the scrolling direction; paragraph 1480, providing current state/setting of interface, current body state, and current action corresponding to current body state; paragraph 1481, graphical elements displaying current interface state, current body state, and responsive action; i.e. based on determining that the user has provided a gesture for scrolling, the system may initiate the scroll operation and may also provide a notification/indicator regarding the state of the operation, such as a cursor indicating a scrolling direction and magnitude, etc.).
Kim teaches that the one type of gesture is the first type of gesture (e.g. paragraphs 0251-254, Figs. 16A-C, outputting notification screen of received text message, receiving message check command such as folding input and outputting message check screen 1601 for displaying details of received text message; scroll action on message check screen performed by rotation of wrist; wrist rotated in first direction, scrolling to move outputted message check screen; direction for rotating wrist includes scroll direction; if scroll in bottom direction desired, rotating wrist in bottom direction; i.e. the user may provide a wrist rotation input in a first or second direction in order to perform a scroll operation in a designated direction, as shown between Fig. 16B and Fig. 16C).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to receive a hand gesture to initiate automatic scrolling in a direction, and a hand gesture to reverse the scrolling direction (i.e. such as using different fingers to perform pinch gestures to scroll up/down as taught by Kim), where the hand gesture may be a wrist rotation gesture in different directions corresponding to the desired scroll direction (i.e. as taught by Kim, where Osborn also teaches the use of wrist rotation for at least scrolling a wrist, and further teaches that the wrist rotation may be performed in two different directions, and further teaches, based on detecting the scroll gesture, outputting a notification/indicator of the scrolling operation state).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claim 11, Osborn teaches all of the limitations of claim 4 as previously discussed, and further teaches wherein the third type of gesture is a type of gesture that is an instance of the fourth type of gesture (e.g. paragraph 1281, fist squeeze used for discrete event such as to engage or disengage a particular feature within a given application or a continuous event, such as to maintain activation of a particular feature within a given application; paragraph 1283, selecting from presented options by performing hand gesture such as clenching the fist).
Osborn does not teach that the third type of gesture is multiple instances of the fourth type of gesture.  However, Kim teaches the third type of gesture is multiple instances of the fourth type of gesture (e.g. paragraph 0187, user command of making fist twice to cause output of weather information; i.e. as shown in Fig. 5A, where a UI element associated with a weather information check is currently displayed, the user may perform a gesture which is multiple instances of a fist clenching gesture in order to select/display content corresponding to the UI element).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to receive a hand gesture to select a displayed item on the screen(i.e. such as a fist clench gesture as taught by Osborn) as multiple instances of the hand gesture (i.e. such as receiving multiple fist clench gestures as taught by Kim).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Kim, further in view of Platzer et al. (US 20130239046 A1).
With respect to claim 8, Osborn teaches all of the limitations of claim 6 as previously discussed, and further teaches wherein the one or more programs further include instructions for:
performing a second automatic scrolling operation that scrolls a sequence of a plurality of interface objects in a second direction (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling down a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll down a GUI; paragraph 1484, pinky finger pinch mapped to scroll down so that when user makes pinky finger pinch pose, the page/browser displayed scrolls down).
Although Osborn further teaches that user hand gestures may be mapped to different commands in different states/modes, and further teaches that a wrist roll gesture may be mapped to a scroll command (e.g. paragraph 0648, switching between modes; paragraph 0666-0667, selection of mode, such as from list or using gesture; paragraph 1380, transitioning between different mappings of body part states and responsive actions; paragraph 1408-1410, activating first mapping and/or deactivating second mapping between possible body part states and responsive actions; user switching between mappings; transition/switch between mappings; paragraph 1486, user modifying mappings between body states and actions, user configuring/mapping scroll controls to different poses; table accompanying paragraph 1649, indicating that an interaction with a list feature may include a “scroll through” interaction with a “wrist roll” control scheme), Osborn does not explicitly disclose that the second automatic scrolling operation is performed while the computer system is in the second operation mode.
However, Kim teaches that the second automatic scrolling operation is performed while the computer system is in the second operation mode (e.g. paragraphs 0251-254, Figs. 16A-C, outputting notification screen of received text message, receiving message check command such as folding input and outputting message check screen 1601 for displaying details of received text message; scroll action on message check screen performed by rotation of wrist; wrist rotated in first direction, scrolling to move outputted message check screen; direction for rotating wrist includes scroll direction; if scroll in bottom direction desired, rotating wrist in bottom direction; i.e. where a user selection of an indicator regarding a received message may transition the system to a second operation mode, such as a mode in which the corresponding functionality is executed and corresponding content is displayed, and in this second mode, the user may provide a wrist rotation input in a first or second direction in order to perform a scroll operation in a designated direction, as shown between Fig. 16B and Fig. 16C).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to place the system in a second mode (i.e. such as a mode of executing a corresponding functionality following a user’s selection of a displayed item, as taught by Osborn, and a mode of executing received message display and interaction following a user’s selection of a displayed message notification, as taught by Kim), and while in this mode receive a hand gesture to initiate automatic scrolling in a direction, and a hand gesture to reverse the scrolling direction (i.e. such as using different fingers to perform pinch gestures to scroll up/down as taught by Kim), where the hand gesture may be a wrist rotation gesture in different directions corresponding to the desired scroll direction (i.e. as taught by Kim, where Osborn also teaches the use of wrist rotation for at least scrolling a wrist, and further teaches that the wrist rotation may be performed in two different directions).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
Osborn and Kim do not explicitly disclose:
while performing the second automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects; and
in response to detecting the end of the sequence of the plurality of user interface objects, performing a third automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction.
However, Platzer teaches:
while performing the second automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects (e.g. paragraph 0077, Figs. 5A-C, scrolling list of items to terminus of the list; paragraph 0078, terminus is edge of scrolled region; while list is being scrolled; terminus is virtual boundary associated with displayed objects; displayed objects reach or intersect with the terminus 514; i.e. the system detects the end of a list of displayed UI objects, as indicated by a terminus at the edge of the scrolled list); and
in response to detecting the end of the sequence of the plurality of user interface objects, performing a third automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction (e.g. paragraph 0077, Figs. 5A-C, reaching terminus of list, smoothly bouncing displayed items at end of list, reversing direction; paragraph 0078, as illustrated in Fig. 5B, when objects reach or intersect with the terminus 514, the movement corresponding to scrolling may stop and, as illustrated in Fig. 5C, subsequently reverse direction; i.e. scrolling list reaches and bounces at its terminus; i.e. in response to reaching/detecting the terminus while scrolling in a first direction, the system automatically performs a “bounce” by scrolling the list of displayed objects in the opposite direction).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Kim, and Platzer in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects, such as scrolling menus/lists of UI objects, webpages, etc.) and Kim (directed to controlling a mobile terminal, such as using hand gestures, including for scrolling displayed content), to incorporate the teachings of Platzer (directed to scrolling operations) to include the capability to, while scrolling user interface content, including a plurality of objects, in a given direction (i.e. as taught by Osborn, Kim, and Platzer), detect a terminus indicating the end of the plurality of objects and, in response automatically scroll the content including the plurality of objects in a reverse direction (as taught by Platzer).  One of ordinary skill would have been motivated to perform such a modification in order to provide operations desired by users, such as scrolling operations on a device, including on device having limited display size, user interface, etc. which limits the ease of use of the device, as described in Platzer (paragraphs 0006, 0008).
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Saurabh et al. (US 20180181261 A1).
With respect to claim 17, Osborn teaches all of the limitations of claim 4 as previously discussed.  Osborn does not explicitly disclose wherein the menu is displayed:
in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location; and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location.
However, Saurabh teaches wherein the menu is displayed:
in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location (e.g. abstract, positioning popup interface within display frame in relation to corresponding target element; relative positioning of popup interface to associated target element dynamically selected and determined based on location of corresponding target element within quadrants/subquandrants of display frame; paragraph 0001, presenting additional information/menu in the form of a popup interface; paragraph 0004, rendering popup interface at position determined to be best fit position with regard to target element’s position; paragraph 0005, determining dynamically made where to display the popup interface by determining quadrant the target element is actually displayed within; paragraph 0027, pop interface may refer to a context menu; paragraph 0028, target element linked to popup interface; target element may be word or icon, and popup interface may be a selection menu related to the word or icon; paragraph 0077, Fig. 8, target element 802 residing within subquadrant 5/quadrant 1, resulting popup interface 804 displayed at bottom position relative to target element; i.e. where a respective user interface element (target element) is displayed at a position within a first quadrant (analogous to a first location), the corresponding popup menu is also displayed at a position within the same quadrant/location); and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location (e.g. abstract, positioning popup interface within display frame in relation to corresponding target element; relative positioning of popup interface to associated target element dynamically selected and determined based on location of corresponding target element within quadrants/subquandrants of display frame; location of popup interface is dynamically adjustable as opposed to being positioned based on a predetermined location; paragraph 0004, rendering popup interface at position determined to be best fit position with regard to target element’s position, event when that target element changes position or otherwise moves; paragraph 0005, determining dynamically made where to display the popup interface by determining quadrant the target element is actually displayed within; as the target element moves within the user interface (due to scrolling etc.) the relative position of the popup interface is dynamically adjusted so that the popup interface is continuously displayed in a position identified as best fit relative to target element; paragraph 0083, Fig. 10, target element 1002 displayed spanning quadrant1 and quadrant2, resulting popup interface 1004 displayed in bottom position relative to target element 1002; i.e., where a respective user interface element is displayed at a different location such as a location spanning multiple quadrants (analogous to not being displayed at a first location), the corresponding popup menu may also be moved/repositioned on the display so that it is also displayed at a second location which is different from the first location).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Saurabh in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects, including displaying associated menus), to incorporate the teachings of Saurabh (directed to a positioning mechanism for bubble as a custom tooltip, such as positioning popup interfaces/menus in relation to a corresponding target element) to include the capability to dynamically display the menu (i.e. of Osborn) in a position relative to the position of a corresponding UI object such that when the UI object is displayed at a first location, the menu is also displayed at the first location and when the UI object is not displayed at the first location, such as because it is moved to a different location, the menu is displayed at a second location different from the first location, such as at the different location that the UI object has been moved to (as taught by Saurabh).  One of ordinary skill would have been motivated to perform such a modification in order to solve disadvantages found in existing popup interfaces by providing the capability to dynamically render a popup interface at a position that is determined to be a best fit position with regard to a target elements position, event when that target element changes position or otherwise moves in some manner, as described in Saurabh (paragraphs 0002-0004).
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain,” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting in re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (GCPA 1968)). Further, a reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill the art, including nonpreferred embodiments. Merck & Co, v. Biocraft Laboratories, 874 F.2d 804, 10 USPQ2d 1843 (Fed. Cir.), cert, denied, 493 U.S. 975 (1989). See also Upsher-Smith Labs. v. Pamlab, LLC, 412 F,3d 1319, 1323, 75 USPQ2d 1213, 1215 (Fed. Cir, 2005): Celeritas Technologies Ltd. v. Rockwell International Corp., 150 F.3d 1354, 1361, 47 USPQ2d 1516, 1522-23 (Fed. Cir. 1998).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Yoon et al. (US 20150061842 A1) teaches various gestures for controlling a mobile device such as a wrist worn device (Figs. 7-33).
Offenberg et al (US 20150248235 A1) teaches various gestures for controlling a mobile device such as a wrist worn device (Figs. 27-92), as well as for selecting among displayed items with corresponding visual indications, scrolling, etc. (Figs. 102A-D).
Hazra et al. (US 20170123487 A1) teaches various gestures for controlling a mobile device including a wrist worn device (Figs. 5, 8-15).

APPLICANT REBUTTAL: REMARKS
Claims 1-23 were previously pending. By virtue of this amendment, paragraphs [0085] and [0222] of the Specification have been amended, claims 1, 4, 7-10, 12-14, and 20-23 have been amended, claims 5 and 6 have been canceled, and new claims 24-59 have been added. The amendments to the Specification are for correcting minor typographical errors and are fully supported by the Application as-filed. The amendments to independent claims 1, 22, and 23 are supported by at least claims 5 and 6, by Figures 11B-11D, and by paragraphs [0191] - [0192] of the Application as-filed. The amendments to the remaining claims are for correcting typographical errors and for providing consistency across the claims due to the amendments to claim 1 and are fully supported by the Application as-filed. New claims 24-59 are counterpart dependent method and CRM claims to dependent system claims 2-4 and 7-21 and are fully supported by the Application as-filed. Thus, no new matter has been added. Claims 1-4 and 7-59 are presently pending.
Applicant respectfully requests reconsideration of this application based on the foregoing amendments and the following remarks.
Examiner Interview
Applicant thanks Examiner Stanley for the telephonic interview on September 12, 2023 with Applicant's representative Horace Zhang. During the September 12 interview, Applicant's Representative noted that Osborn (US 20230072423) is a continuation-in-part application with a filing date of April 15, 2022, which is after Applicant's priority date of May 19, 2021. Applicant's representative asked the Examiner to point to the priority application(s) of Osborn that have an effective filing date before Applicant's filing date and that include the subject matter of Osborn used to reject Applicant's claims. In response, the Examiner indicated that Anderson (U.S. Pat. No. 11,150,730, with a filing date of April 30, 2020) is a priority document of Osborn that includes the subject matter of Osborn used to reject the independent claims.
Applicant also thanks Examiner Stanley for the telephonic interview on September 20, 2023 with Applicant's representatives Horace Zhang and Christopher Eide. During the September 20 interview, Applicant's representatives requested that if the Examiner issues a new Office Action that maintains the rejection of any claim over Osborn, the Examiner point to the priority application(s) of Osborn that have an effective filing date before Applicant's filing date and that include the subject matter of Osborn used to reject the claim. The Examiner indicated that he would do so.
During the September 20 interview, the attendees also discussed the 102 rejection of claim 1 over Osborn/Anderson. In particular, Applicant's representatives proposed amending the independent claims as indicated above. The Examiner indicated that the claim amendments would likely overcome the present 102 rejection, subject to further review and consideration of the cited references. However, no agreement as to the ultimate patentability of the claims was reached.
Claim O
bj
ections
The Office Action rejects claim 14 because of an informality. As indicated above, claim 14 has been amended to correct the informality. Applicant thus respectfully requests the objection to claim 14 be withdrawn and the claim be allowed.
Claim Rejections - 35 USC 112
Claims 13 and 20 stand rejected under 35 USC 112 for allegedly being indefinite.
Applicant has amended claims 13 and 20 to correct the issues raised in the Office Action. Specifically:
13. (Currently Amended) The computer system of claim 4, wherein the one or more programs further include instructions for:
detecting, via at least th
al sensor, a third hand gesture
;
and
in response to detecting
least the optical sensor, the seeend
third
hand gesture:
in accordance with a deternination that a notification was received within a
threshold period of time and in accordance with a determination that the
second
third hand
gesture is the fourth type of gesture, performing an action related to the notification.
20. (Currently Amended) The computer system of claim 1, wherein the one or more programs further include instructions for:
while displaying, via the display generation component, the indication that the second user interface object is selected, detecting, via at least the optical sensor, a fifth hand gesture; and
in response to detecting, via at least the optical sensor, the
fourth
fifth
hand
gesture:
in accordance with a determination that the fifth hand gesture is the first type of gesture, displaying, via the display generation component, an indication that a fourth
first
user interface object is selected; and
in accordance a determination that the fifth hand gesture is the second type of
gesture, displaying, via the display generation component, the indication that the first user interface object
is selected.
Applicant thus respectfully requests the 112 rejection of claims 13 and 20 be withdrawn and the claims be allowed.
Claim Re
l
ections 35 U
SC
Claims 1, 3-5, 12, 14-16, and 18-23 are rejected under 35 U.S.C. 102(a)(2) as allegedly being anticipated by Osborn et al. (US 20230072423 A1).
Applicant has amended independent claims 1, 22, and 23 based on the subject matter of
previous claim 6, Figures 11B-11D of the Application as-filed, and paragraphs [0191]-[0192] of the Application as-filed.
Amended claims 1, 22, and 23 (and all claims depending therefrom) are allowable for at least the reasons discussed below.
Claim 6 stands rejected under 35 USC 103 over Osborn/Anderson and Kim (US
2016/0029899). The cited portions of Osborn/Anderson teach that the user can scroll up with one type of hand gesture (e.g., ring finger pinch) and the user can scroll down with another type of hand gesture (e.g., pinky finger pinch). The cited portions of Kim teach scrolling though a text message via a wrist rotation gesture.
However, Osborn/Anderson and Kim fail to teach or suggest any "automatic scrolling
operation in which the computer system scrolls through content without detecting user input...," as recited in the amended claims. Rather, Osborn/Anderson and Kim teach scrolling with user input, e.g., pinch or rotation gestures.
Further, much less do Osborn/Anderson and Kim teach or suggest "transitioning the computer system ... to a second operation mode" in which the computer system performs the "automatic scrolling operation" responsive to a hand gesture input that selects the "first user interface object."
Further still, Osborn/Anderson and Kim fail to teach or suggest detecting different hand gestures to control the "automatic scrolling operation" while performing "the automatic scrolling operation," as required by the amended limitations:
while the computer system is in the second operation mode, detecting, via at least the optical sensor, a second hand gesture; and
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that the second hand gesture is the first type of gesture, toggling the automatic scrolling operation between an active state and an inactive state; and
in accordance with a determination that the second hand gesture is the second type of gesture, reversing a first direction of the automatic scrolling operation.
For at least the reasons discussed above, Applicant respectfully requests the 102 rejections of claims 1, 22, and 23 (and all claims depending therefrom) be withdrawn and the claims be allowed.
Claim Re
j
ections 35 U
SC
Claim 2 is rejected under 35 U.S.C. 103 as allegedly being unpatentable over Osbom in view of Chung et al. (US 20220326779 Al).
Claim 2 depends from claim 1, which is allowable for at least the reasons discussed above. Thus, claim 2 is allowable for at least the reason that it depends from an allowable independent claim. The addition of Chung fails to cure the deficiencies of Anderson/Osbom discussed above, nor is it alleged to. Further, because Osborn is a continuation-in-part application with a filing date after Applicant's priority date, it is unclear whether Osborn is prior art for claim 2. In view of the above, Applicant respectfully requests the 103 rejection of claim 2 be withdrawn and the claim be allowed.
Claims 6, 7, 9-11, and 13 are rejected under 35 U.S.C. 103 as allegedly being unpatentable over Osborn in view of Kim et al. (US 20160029899 Al).
Claim 6 has been canceled, rendering the rejection moot.
Claims 7, 9-11, and 13 depend from claim 1, which is allowable for at least the reasons discussed above. Thus, claims 7, 9-11, and 13 are allowable for at least the reason that they each depend from an allowable independent claim. The addition of Kim fails to cure the deficiencies of Anderson/Osborn discussed above, nor is it alleged to. Further, because Osborn is a continuation- in-part application with a filing date after Applicant's priority date, it is unclear whether Osborn is prior art for claims 7, 9-11, and 13. In view of the above, Applicant respectfully requests the 103 rejection of claims 7, 9-11, and 13 be withdrawn and the claims be allowed.
Claim 8 is rejected under 35 U.S.C. 103 as allegedly being unpatentable over Osborn, in view of Kim, and further in view of Platzer et al. (US 20130239046 A1).
Claim 8 depends from claim 1, which is allowable for at least the reasons discussed above. Thus, claim 8 is allowable for at least the reason that it depends from an allowable independent claim. The addition of Kim and Platzer fails to cure the deficiencies of Anderson/Osborn discussed above, nor is it alleged to. Further, because Osborn is a continuation-in-part application with a filing date after Applicant's priority date, it is unclear whether Osborn is prior art for claim 8. In view of the above, Applicant respectfully requests the 103 rejection of claim 8 be withdrawn and the claim be allowed.
Claim 17 is rejected under 35 U.S.C. 103 as allegedly being unpatentable over Osborn in view of Saurabh et al. (US 20180181261 Al).
Claim 17 depends from claim 1, which is allowable for at least the reasons discussed above. Thus, claim 17 is allowable for at least the reason that it depends from an allowable independent claim. The addition of Saurabh fails to cure the deficiencies of Anderson/Osborn discussed above, nor is it alleged to. Further, because Osborn is a continuation-in-part application with a filing date after Applicant's priority date, it is unclear whether Osborn is prior art for claim 17. In view of the above, Applicant respectfully requests the 103 rejection of claim 17 be withdrawn and the claim be allowed.
New Claims
New claims 24-59 have been added. New claims 24-59 do not raise any new issues. In particular, new claims 24-41 are counterpart dependent CRM claims to (and contain substantially identical subject matter to) dependent system claims 2-4 and 7-21. Similarly, new claims 42-59 are counterpart dependent method claims to (and contain substantially identical subject matter to) dependent system claims 2-4 and 7-21.
New claims 24-59 depend variously from independent claims 22 and 23, which are allowable for at least the reasons discussed above. Thus, claims 24-59 are allowable.
CONCLUSION

NEXT OFFICE ACTION: DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the Amendment filed on September 26, 2023.  Claims 1, 4, 7-10, 12-14, and 20-23 are amended.  Claims 24-59 are new.  Claims 5 and 6 are cancelled.  Therefore, Claims 1-4 and 7-59 are pending in the case.  Claims 1, 22, and 23 are the independent claims.
This action is final.
Applicant’s Response
In the Amendment filed on September 26, 2023, Applicant amended the claims and provided arguments in response to the objections to the claims and rejections of the claims under 35 USC 112, 102, and 103 in the previous office action.
Response to Argument/Amendment
Applicant’s amendments to the claims in response to the objection to the claims in the previous office action are acknowledged, and the objection is withdrawn in view of the amendments, which correct the basis for the objection.
Applicant’s amendments to the claims in response to the rejection of the claims under 35 USC 112 in the previous office action are acknowledged, and Applicant’s associated arguments have been fully considered.  The rejections are withdrawn in view of Applicant’s amendments, which correct and clarify the identified issues and therefore render the rejections moot.
Applicant’s amendments to the claims in response to the rejections of the claims under 35 USC 102 and 103 in the previous office action are acknowledged, and Applicant’s associated arguments have been fully considered.  Applicant argues that Osborn and Kim “fail to teach or suggest any ‘automatic scrolling operation in which the computer system scrolls through content without detecting user input…,’ as recited in the amended claims,” and “’transitioning the computer system…to a second operation mode’ responsive to a hand gesture input that selects the ‘first user interface object,’” and “detecting different hand gestures to control the ‘automatic scrolling operation’ while performing ‘the automatic scrolling operation,’ as required by the amended limitations.”
Applicant’s arguments are persuasive, and the rejections are withdrawn.  However, new grounds of rejection are provided below.
Regarding the dependent claims, Applicant argues that “[b]ecause Osborn is a continuation-in-part application with a filing date after Applicant’s priority date, it is unclear whether Osborn is prior art” for the dependent claims.  In response to this argument and as requested by Applicant, the following tables show the priority applications of Osborn which have an effective filing date before the earliest priority date of the instant application (May 19, 2021), and include citations to the portions of the corresponding publications of these priority applications which include the subject matter of Osborn cited in the previous and current office actions.
Osborn et al. (US 20230072423 A1)
(portions cited in previous and current office actions)
Anderson et al. (US 11150730 B1)
(publication of Application Number 16/863,098, filed April 30, 2020 (prior to earliest priority date of instant application), which is the parent application of Application number 17/469,537, of which Osborn is a continuation-in-part)
(portions corresponding to, and therefore supporting cited portions of Osborn)
Figures
Fig. 35A
Fig. 1
Fig. 35B
Fig. 2
Figs. 9A-B and 36O-P
Figs. 21A-B
Fig. 35I
Fig. 7
Figs. 35J-L
Figs. 8A-C
Figs. 35O-V
Fig. 10A-H
Figs. 36A-E
Fig. 11A-E
Fig. 36F
Fig. 12
Fig. 36G
Fig. 13
Fig. 36H
Fig. 14
Fig. 36I
Fig. 15
Fig. 36J
Fig. 16
Fig. 36K
Fig. 17
Fig. 36L
Fig. 18
Fig. 36M
Fig. 19
Fig. 36N
Fig. 20
Fig. 36Q
Fig. 22
Fig. 36R
Fig. 23
Fig. 36S
Fig. 24
Paragraphs
paragraph 0352
col. 31 lines 14-32
paragraph 1379
col. 4 lines 36-51
paragraph 1380
col. 4 line 52 - col. 5 line 2
paragraph 1381
col. 5 lines 3-15
paragraph 1384
col. 5 line 48 - col. 6 line 2
paragraph 1385-1387
col. 6 lines 3-32
paragraph 1394
col. 7 lines 36-49
paragraph 1395
col. 7 lines 50-59
paragraph 1396-1398
col. 7 line 60 - col. 8 line 36
paragraph 1400
col. 8 lines 52-67
paragraph 1401
col. 9 lines 1-18
paragraph 1408-1410
col. 10 lines 15-51
paragraph 1411
col. 10 lines 52-64
paragraph 1443
col. 16 line 63 - col. 17 line 17
paragraph 1444
col. 17 lines 18-31
paragraph 1450
col. 18 lines 47-55
paragraph 1451
col. 18 lines 56-64
paragraph 1455
col. 19 lines 37-45
paragraph 1459
col. 20 lines 30-45
paragraph 1460-1461
col. 20 line 46 - col. 21 line 3
paragraph 1463
col. 21 lines 17-25
paragraph 1467
col. 21 line 59-col. 22 line 8
paragraph 1468-1469
col. 22 lines 9-31
paragraph 1470
col. 22 lines 32-41
paragraph 1471
col. 22 lines 42-57
paragraph 1473
col. 23 lines 4-17
paragraph 1474
col. 23 lines 18-26
paragraph 1475
col. 23 lines 27-41
paragraph 1480-1481
col. 24 lines 30-54
paragraph 1483
col. 24 line 62 - col. 25 line 9
paragraph 1484
col. 25 lines 10-23
paragraph 1485
col. 25 lines 24-37
paragraph 1486-1488
col. 25 line 38 - col. 26 line 13
Osborn et al. (US 20230072423 A1)
(portions cited in previous and current office actions)
Osborn et al. (US 20200111260 A1)
(published version of Application Number 16/593,446, filed October 4, 2019 (prior to earliest priority date of instant application), which is the parent application of Application Number 17/213,686, of which Osborn is a continuation-in-part)
(portions corresponding to, and therefore supporting cited portions of Osborn)
Figures
Fig. 1
Fig. 1
Figs. 9A-B
Figs. 9A-B
Paragraphs
paragraph 0257
paragraph 0109
paragraph 0272
paragraph 0124
paragraph 0281
paragraph 0133
paragraph 0286
paragraph 0138
paragraph 0288
paragraph 0140
paragraph 0289
paragraph 0141
paragraph 0292
paragraph 0144
paragraphs 0301-0304
paragraphs 0153-0156
paragraph 0306
paragraph 0158
paragraph 0310
paragraph 0162
paragraph 0311
paragraph 0163
paragraph 0312
paragraph 0164
paragraph 0323
paragraph 0175
paragraph 0344
paragraph 0196
paragraph 0345
paragraph 0197
Osborn et al. (US 20230072423 A1)
(portions cited in previous and current office actions)
Moschella et al. (US 11331045 B1)
(published version of Application Number 16/862,050, of which Osborn is a continuation-in-part, filed April 29, 2020 (prior to earliest priority date of instant application))
(portions corresponding to, and therefore supporting cited portions of Osborn)
paragraph 0442
col. 6 lines 41-51
paragraph 0445
col. 7 lines 25-36
Osborn et al. (US 20230072423 A1)
(portions cited in previous and current office actions)
Berenzweig et al. (US 20200073483  A1)
(published version of Application Number 16/557,342, filed August 30, 2019 (prior to earliest priority date of instant application), which is the parent application of Application Number 17/487,695, of which Osborn is a continuation-in-part)
(portions corresponding to, and therefore supporting cited portions of Osborn)
paragraph 0610
paragraph 0215
paragraph 0611
paragraph 0216
paragraph 0618
paragraph 0223
paragraph 0620
paragraph 0225
Osborn et al. (US 20230072423 A1)
(portions cited in previous and current office actions)
Berenzweig et al. (US 20200097082 A1)
(published version of Application Number 16/577,352, of which Osborn is a continuation-in-part, filed September 20, 2019 (prior to earliest priority date of instant application))
(portions corresponding to, and therefore supporting cited portions of Osborn)
paragraph 0648
paragraph 0113
paragraph 0666-0667
paragraph 0166-0167
Osborn et al. (US 20230072423 A1)
(portions cited in previous and current office actions)
Stone et al. (US 20200097081  A1)
(published version of Application Number 16/577,207, filed September 20, 2019 (prior to earliest priority date of instant application), which is the parent application of Application Number 17/569,836, of which Osborn is a continuation-in-part)
(portions corresponding to, and therefore supporting cited portions of Osborn)
paragraph 0703
paragraph 0194
paragraph 0706
paragraph 0197
Osborn et al. (US 20230072423 A1)
(portions cited in previous and current office actions)
Kaifosh et al. (US 20220019284 A1)
(published version of Application Number 17/293,472, of which Osborn is a continuation-in-part, with a PCT filing date of November 15, 2019 and 371 date of May 12, 2021 (prior to earliest priority date of instant application))
(portions corresponding to, and therefore supporting cited portions of Osborn)
paragraph 0729
paragraph 0095
Osborn et al. (US 20230072423 A1)
(portions cited in previous and current office actions)
Reisman et al. (US 20200310541 A1)
(published version of Application Number 16/833,626, of which Osborn is a continuation-in-part, filed March 29, 2020 (prior to earliest priority date of instant application))
(portions corresponding to, and therefore supporting cited portions of Osborn)
paragraph 1270
paragraph 0039
paragraph 1271
paragraph 0040
paragraph 1281
paragraph 0050
paragraph 1283
paragraph 0052
Osborn et al. (US 20230072423 A1)
(portions cited in previous and current office actions)
Barachant et al. (US 20190357787 A1)
(published version of Application Number 16/389,419, filed April 19, 2019 (prior to earliest priority date of instant application), which is the parent application of Application Number 16/995,859, of which Osborn is a continuation-in-part)
(portions corresponding to, and therefore supporting cited portions of Osborn)
paragraph 0509
paragraph 0056
Osborn et al. (US 20230072423 A1)
(portions cited in previous and current office actions)
Rubin et al. (US 20210064132 A1)
(published version of Application Number 17/010,689, of which Osborn is a continuation-in-part, filed September 2, 2020 (prior to earliest priority date of instant application))
(portions corresponding to, and therefore supporting cited portions of Osborn)
paragraph 1539
paragraph 0077
paragraph 1649 (and table)
paragraph 0187 (and table)
paragraph 1651-1653
paragraph 0189-0191
Osborn et al. (US 20230072423 A1)
(portions cited in previous and current office actions)
Hussami et al. (US 20200310540 A1)
(published version of Application Number 16/833,309, of which Osborn is a continuation-in-part, filed March 27, 2020 (prior to earliest priority date of instant application))
(portions corresponding to, and therefore supporting cited portions of Osborn)
Fig. 13
Fig. 12
paragraph 0385
paragraph 0136
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims under pre-AIA  35 U.S.C. 103(a), the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were made absent any evidence to the contrary.  Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and invention dates of each claim that was not commonly owned at the time a later invention was made in order for the examiner to consider the applicability of pre-AIA  35 U.S.C. 103(c) and potential pre-AIA  35 U.S.C. 102€, (f) or (g) prior art under pre-AIA  35 U.S.C. 103(a).
Claims 1, 3, 4, 9-16, 18-23, 25-27, 29-36, 38-41, 43-45, 47-54, and 56-58 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn et al. (US 20230072423 A1) in view of Jose (US 20220044309 A1), further in view of Kim et al. (US 20160029899 A1).
With respect to claims 1, 22, and 23, Osborn teaches a computer system that is in communication with a display generation component and an optical sensor, the computer system comprising: one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method; a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system that is in communication with a display generation component and an optical sensor, the one or more programs including instructions for performing the method (e.g. paragraph 0257, supplementing sensor data with images of user captured by camera; paragraph 0286, auxiliary devices/sensors including imaging devices/cameras, heart rate monitors, etc.; paragraph 0288, Figs. 9A-B, wearable system with sensors such as EMG sensors, sensors 810; computer processor such as microcontroller 934 receiving inputs from sensors; communicating with dongle portion 920 and outputting processing output to dongle portion; paragraph 0289, dongle portion 920 receiving signals from wearable portion 910, and providing them to host computer for further processing, display, effecting control of objects, etc.; paragraph 0292, describing Fig. 1, system including processors 112 programmed to communicate with sensors 110 and executing algorithms; paragraph 0301, system including display controller configured to display visual representations; computer processors implementing trained inference models that receive sensor signals as input and provide information to generate control signals for system as output; paragraph 0302, user interface; paragraph 0303, various types of devices in which processors may be included; paragraph 0304, multiple processors; paragraph 0306, system including processors, camera, and display; paragraph 310, camera information/cameras; paragraph 0311, camera mounted on user; paragraph 0344, embodiments implemented in software executed on processors; paragraph 0345, non-transitory computer-readable storage medium/memory encoded with computer program/instructions executed on processor to perform described functions; paragraph 0352, sensor including light emitter/detector; paragraph 0385, Fig. 13, vibrotactile system 1300 may be standalone system, or may be configured with interaction with various types of other devices, and includes processor 1360; paragraph 0610, cameras; paragraph 0729, photoplethysmography/PPG sensors detecting vascular changes/changes in blood volume; paragraph 1384, Fig. 35B, wearable device 35102 capable of controlling computer devices via neuromuscular signals of users, including set of sensors 35104 that detect/sense neuromuscular signals traversing body of user; processing device 35106 communicatively coupled to sensors, and memory 35108 including models and computer-executable instructions that when executed by processor cause it to identify force generated, generate commands for HCI system, etc.; i.e. the device includes at least one processor and memory storing instructions for execution by the processor, and is further in communication with at least one display/display controller/generation component, and at least one optical sensor, such as a camera (which may be mounted on the user)); and the method, comprising:
displaying, via the display generation component, a user interface that includes a first user interface object, a second user interface object, a third user interface object, and an indication that the first user interface object is selected (e.g. paragraph 1460, describing radial menu 351000 in Figs. 35O-V, where the menu may include visual indicators including a selection indicator 351020; paragraph 1461, selection indicator 351020 may indicate a currently selected action indicator; paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual, auditory, or haptic feedback; visual feedback representing change of a characteristic of visual element within radial menu in response to selection of the visual indicator, such as position change, orientation change, color change, size change, transparency change, fill change, emphasis change, shadow change, animation change, font change, line type change, line width change, combinations, etc.; paragraph 1473, describing Figs. 36B-C, radial menu 351110 including a set of action indicators including indicators 351111 and 351113 displayed in GUI along arc, where as shown in Fig. 36B includes an indication that indicator 35111 is selected; demonstrating/confirming selection of visual indicator using visual feedback; paragraph 1475, describing Figs. 36D-E, displaying sequential menu 351120 including a sequence of visual indicators depicted as A through F, where as shown in Fig. 36 D includes in indication that indicator D/351123 is selected; i.e. both of the menus/interfaces shown in the cited figures include at least first, second, and third UI objects along with an indication that an object of these is selected, such as a visual feedback including a shading/coloration of a selected indicator; paragraphs 1385-1387, determining based on neuromuscular signals state of body part of user and generating input commands accounting for state of user’s body part; sensors constitute any type of form of sensor capable of detecting/sensing neuromuscular signals in user’s body including EMG sensors, MMG sensors, SMG sensors; sensors may also include auxiliary sensors such as IMU, position tracking, imaging (camera), and radiation based sensors, microphones, heart rate monitors, etc.; paragraph 1394, wearable device 35102 coupled with computing system/display; user placing body in state/condition to control/modify presentation/performance of computing system, generating neuromuscular signals representative/indicative/suggestive of state/action);
while displaying the user interface that includes the first user interface object, the second user interface object, the third user interface object, and the indication that the first user interface object is selected, detecting, via at least the optical sensor, a hand gesture (e.g. paragraph 0310, camera information used to improve interpretation of neuromuscular signals; camera used to record image of user, may be within particular range/wavelengths of light, images produced by visible/non-visible light, including showing heat distribution in user’s body, providing indication of blood flow within user and used to infer condition of user such as forces being exerted by fingers; paragraph 0311, camera mounted on user; paragraph 0312, camera data in addition to neuromuscular activity signals used to reconstruct positions, geometries, and forces being applied by user’s arm/hand; paragraph 0323, auxiliary signals from auxiliary devices such as camera supplementing neuromuscular signals; paragraph 0610, cameras used to improve interpretation of signals; paragraph 0611, camera data used to provide real time determination of musculoskeletal representations of gestures; paragraph 0729, PPG sensors detecting vascular changes; paragraph 1271, camera/imaging data; paragraph 1387, sensors including auxiliary sensors such as cameras; paragraph 1395, user pose/gesture generating neuromuscular signals as they traverse down arm toward hand; sensors 35104 detecting/sensing neuromuscular signals and generating data representative of the signals; paragraph 1396-1398, sensor data undergoing processing, analyzing data to determine state of body parts of user, producing/outputting classification indicating state of body part; paragraph 1398, gestures; paragraph 1400, causing computing system to perform actions mapped to state of body parts; paragraph 1401, describing various actions mapped to different states of user’s body parts; paragraph 1444, capturing gestures with camera; photographic data of hand gestures; paragraph 1473, body state mapped to selection of next indicator in a given sequence; identification of body state; paragraph 1474, body states including wrist extension, flexion, rotation, radial deviation, ulnar deviation, etc.; paragraph 1475, body states mapped to selection of visual indicators in sequence; identification of body states; i.e. detecting a body state of the user, such as a hand gesture, including by using an optical sensor such as a camera,  PPG, etc., possibly in combination with other sensors),
wherein before detecting the hand gesture, the computer system is in a first operation mode (e.g. paragraph 1463, Fig. 35O, fist pose mapped to command to display, enter, and activate radial menu; i.e. where in the first state, at the time the radial menu is displayed, the user has provided a previous input causing the radial menu to be activated, and therefore in an operation mode in which the menu is activated, analogous to a first operation mode);
in response to detecting, via at least the optical sensor, the hand gesture:
in accordance with a determination that the hand gesture is a first type of gesture, displaying, via the display generation component, an indication that the second user interface object is selected (e.g. paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual feedback; paragraph 1473, Figs. 36B-C, first body state mapped to selection of the next indicator in a given sequence; in response to identification of first body state, deselecting indicator 35111 and selecting indicator 351113; demonstrating/confirming selection of visual indicator using visual feedback; paragraph 1475, Figs. 36D-E, first body state mapped to selection of next visual indicator in the sequence; i.e. in response to detecting a first type of gesture, as indicated by a first body state, the indication that a first UI object is selected may be changed to an indication that a second UI object is selected, such as by moving the indicator of the selection to the second UI object, where the second UI object is a next UI object in a sequence with respect to the first UI object); and
in accordance with a determination that the hand gesture is a second type of gesture that is different from the first type of gesture, displaying, via the display generation component, an indication that the third user interface object is selected (e.g. paragraph 1467, selection of a visual indicator may be demonstrated or confirmed using visual feedback; paragraph 1473-1474, Figs. 36B-C, first body state mapped to selection of next indicator in given sequence; demonstrating/confirming selection of visual indicator using visual feedback; first and second body states may represent counterparts of one another; 1475, Figs. 36D-E, second body state mapped to selection of previous visual indicator in the sequence; in response to identification of second body state, deselecting subsequent indicator 351123 and selecting prior indicator 351121; i.e. in response to detecting a second/different type of gesture, as indicated by a second body state, the indication that a first UI object is selected may be changed to an indication that a third UI object is selected, such as by moving the indicator of the selection to the third UI object, where the third UI object is a previous UI object in a sequence with respect to the first UI object; in addition, because these gestures cause selection to move to a next/previous object in sequence, receipt of multiple of either of these gestures may also cause a different/third UI object to be selected; for example, where multiple instances/two instances of the first type of gesture are received, the object selected as indicated may be first moved from the first UI object to the second/next UI object in the sequence, and then moved again from the second/next UI object to a third/subsequent UI object in the sequence); and
in accordance with a determination that the hand gesture is a third type of gesture that is different from the first type of gesture and the second type of gesture, performing an operation corresponding to selection of the first user interface object, wherein performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode (e.g. paragraph 1468, Fig. 35Q, illustrating finger pinch pose 351006, which may be mapped to command/instruction to click visual indicator in the radial menu, analogous to mouse button click or keypress; alternatively combination of finger pinch pose and certain degree of force mapped to click command; paragraph 1469, Fig. 35 U, clicking/engaging with or activating function of the currently selected action indicator; clicking action indicator in response to identified click gesture; upon performing the click action, causing visual indicator to appear to depress and/or release, similar to a physical button; i.e. clicking the currently selected action indicator causes the corresponding function to be activated, and therefore causes the system to transition to an operation mode in which the corresponding function has been activated, analogous to a second operation mode);
detecting, via at least the optical sensor, a second hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the second hand gesture:
in accordance with a determination that the second hand gesture is one type of gesture, toggling a first automatic scrolling operation between an active state and an inactive state (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling up a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll up a GUI; paragraph 1484, ring finger pinch mapped to scroll up so that when user makes ring finger pinch pose, the page/browser displayed scrolls up); and
in accordance with a determination that the second hand gesture is another type of gesture, reversing a first direction of the first automatic scrolling operation  (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling down a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll down a GUI; paragraph 1484, pinky finger pinch mapped to scroll down so that when user makes pinky finger pinch pose, the page/browser displayed scrolls down).
Although Osborn further teaches that user hand gestures may be mapped to different commands in different states/modes, and further teaches that a wrist roll gesture may be mapped to a scroll command (e.g. paragraph 0648, switching between modes; paragraph 0666-0667, selection of mode, such as from list or using gesture; paragraph 1380, transitioning between different mappings of body part states and responsive actions; paragraph 1408-1410, activating first mapping and/or deactivating second mapping between possible body part states and responsive actions; user switching between mappings; transition/switch between mappings; paragraph 1486, user modifying mappings between body states and actions, user configuring/mapping scroll controls to different poses; table accompanying paragraph 1649, indicating that an interaction with a list feature may include a “scroll through” interaction with a “wrist roll” control scheme), Osborn does not explicitly disclose:
wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture,
while the computer system is in the second operation mode, detecting the second hand gesture.
However, Jose teaches:
in accordance with a determination that the hand gesture is a third type of gesture that is different from the first type of gesture and the second type of gesture, performing an operation corresponding to selection of the first user interface object, wherein performing the operation corresponding to selection of the first user interface object includes transitioning the computer system from the first operation mode to a second operation mode wherein when the computer system is in the second operation mode, the computer system performs an automatic scrolling operation in which the computer system scrolls through content without detecting user input further to the hand gesture (e.g. paragraph 0018-0019, UI control element allowing user to select to invoke content carousel functionality, according to user preferences/configuration settings in a wide variety of settings/applications; paragraph 0049-0050, Fig. 13, touch screen; user selecting content carousel button 1302 in interface, invoking content carousel such that user interface will begin an automatic and continuous scroll, such that content items not currently presented will reveal themselves as the user interface scrolls in a direction from bottom to top; speed and direction of scroll may be configured by user via settings; paragraph 0051, Fig. 14, group of product listings presented in single scrollable user interface, content carousel button 1402 allowing user to invoke content carousel to have entire user interface continuously scroll in one direction at a constant speed; paragraph 0055, touch input device; paragraph 0056, detecting biometric/biosignals, body gestures, etc.; i.e. detecting a user selection gesture/input to a user interface component, such as a touch input/gesture, where this selection gesture/input activates a corresponding function of the component for activating an automatic and continuous scroll functionality/mode which scrolls through content without detecting further user input to this selection gesture/input; see also Figs. 2, 5, 8, and 10-12, and their corresponding descriptions, similarly describing invocation of content carousel functionality in response to user selection of a corresponding user interface component),
while the computer system is in the second operation mode, detecting a second gesture; and in response to detecting the second hand gesture:
in accordance with a determination that the second gesture is one type of gesture, toggling the automatic scrolling operation between an active state and an inactive state (e.g. paragraph 0025, Fig. 2, as content items are presented via the content carousel, content carousel playback controls 216 are displayed, enabling the user to manually pause the content carousel to statically show the current content item; i.e. while the content carousel is automatically/continuously scrolling the content, a second user gesture may be received; where this user gesture is of a type which selects a control for pausing the content carousel, this will cause the scrolling to pause, and is therefore analogous to a second gesture of one type which toggles the scrolling operation of the content carousel between an active/inactive state; see also Fig. 9, showing similar playback controls); and
in accordance with a determination that the second gesture is another type of gesture, reversing a first direction of the automatic scrolling operation (e.g. paragraph 0025, Fig. 2, as content items are presented via the content carousel, content carousel playback controls 216 are displayed, enabling the user to manually pause the content carousel to manually skip backwards; paragraph 0041, noting that the user may specify, via configuration settings, the transition direction for presentation of content items, such as a user selection input for left to right, right to left, top to bottom, or bottom to top transition directions; i.e. while the content carousel is automatically/continuously scrolling the content, a second user gesture may be received; where this user gesture is of a type which selects a control for skipping backwards in the content carousel, this will cause the scrolling to reverse, and is therefore analogous to a second gesture of another type which reverses the scrolling direction of the content carousel; further the user may provide a selection input to change/reverse the direction of the scrolling operation, such as via a corresponding configuration settings interface; see also Fig. 9, showing similar playback controls).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn and Jose in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), to incorporate the teachings of Jose (directed to providing content carousel functionality for presenting user interface contents according to configuration settings and user inputs/selections) to include the capability to place the system in second mode in response to a third type of user input (i.e. such as a mode of executing a corresponding functionality following a user’s selection activating a displayed item, as taught by Osborn, and a mode of initiating automatic and continuous content scrolling functionality following a user’s selection of a user interface component for activating a content carousel function, as taught by Jose), where the second mode includes an automatic scrolling function and, while in this automatic scrolling mode, receive an input/gesture to toggle between an active and inactive state of the automatic scrolling, and an input/gesture to reverse the scrolling direction (i.e. such as a selection input/gesture selecting a UI component for pausing the content carousel function and its corresponding automatic scrolling, an a selection input/gesture selecting a UI component for skipping backwards in the content carousel, or altering a configuration setting of the scrolling direction as taught by Jose).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability for the user to view a large amount of content with only a single interaction with the user interface, as described in Jose (paragraphs 0049-0050).
Osborn and Jose do not explicitly disclose the determinations that the second hand gesture is the first type of gesture or the second type of gesture.  However, Kim teaches while the computer system is in an operation mode which includes scrolling functionality, detecting the second hand gesture, and the determinations that the second hand gesture is the first type of gesture or the second type of gesture (e.g. paragraphs 0251-254, Figs. 16A-C, outputting notification screen of received text message, receiving message check command such as folding input and outputting message check screen 1601 for displaying details of received text message; scroll action on message check screen performed by rotation of wrist; wrist rotated in first direction, scrolling to move outputted message check screen; direction for rotating wrist includes scroll direction; if scroll in bottom direction desired, rotating wrist in bottom direction; i.e. where a user selection of an indicator regarding a received message may transition the system to a second operation mode, such as a mode in which the corresponding functionality is executed and corresponding content is displayed, and in this second mode, the user may provide a wrist rotation input in a first or second direction in order to perform a scroll operation in a designated direction, as shown between Fig. 16B and Fig. 16C).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Jose, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) and Jose (directed to providing content carousel functionality for presenting user interface contents according to configuration settings and user inputs/selections), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability, while the system is in a second mode which includes scrolling functionality (i.e. as taught by Osborn and Kim, including a mode which includes automatic scrolling functionality as taught by Jose), and while in this mode receive a second hand gesture to activate scrolling in a direction, and a second hand gesture to reverse the scrolling direction (i.e. such as using different fingers to perform pinch gestures to scroll up/down as taught by Osborn), where the hand gesture may be a wrist rotation gesture in different directions corresponding to the desired scroll direction (i.e. as taught by Kim, where Osborn also teaches the use of wrist rotation for at least scrolling a wrist, further teaches that the wrist rotation may be performed in two different directions, and further teaches that various body part states may be mapped to different UI controls in different modes, where Jose further teaches that user inputs may be provided to pause/deactivate/activate the scrolling, and to reverse the direction of the scrolling).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claims 3, 25, and 43, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the second type of gesture is a type of gesture that is multiple instances of the first type of gesture (e.g. paragraph 1473-1474, Figs. 36B-C, first body state mapped to selection of next indicator in given sequence; in response to identification of first body state, deselecting indicator 35111 and selecting indicator 351113; first and second body states may represent counterparts of one another; 1475, Figs. 36D-E, first body state mapped to selection of next visual indicator in the sequence; i.e. because first body state/gesture causes selection to move to a next object in sequence, receipt of multiple instances of this gesture may also cause a different/third UI object to be selected; for example, where multiple instances/two instances of the first type of gesture are received, the object selected as indicated may be first moved from the first UI object to the second/next UI object in the sequence, and then moved again from the second/next UI object to a third/subsequent UI object in the sequence).
With respect to claims 4, 26, and 44, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
in response to detecting, via at least the optical sensor, the hand gesture (as cited above, using camera/sensors to detect hand gesture):
in accordance with a determination that the hand gesture is a fourth type of gesture that is different from the first type of gesture, the second type of gesture, and the third type of gesture, displaying, via the display generation component, a menu that includes one or more selectable options (e.g. paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user (as compared to other movements/poses causing item selection/clicking); paragraph 1485, displaying interface 351300 of Fig. 36G, which may be a popup box/dialog that appears in response to user input; i.e. where a some suitable user input, such as a fist clenching gesture, may be used to open various menus, such as a menu including selectable options).
With respect to claims 12, 32, and 50, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the fourth type of gesture does not include multiple instance of the first type of gesture and the third type of gesture does not include multiple instances of the second type of gesture (e.g. paragraph 0620, finger/hand gesture combined with force, user clenching first, etc., to open menu/dialog; paragraph 1468, finger pinch pose used to click selected item; paragraph 1381, first pose to display menu; paragraph 1474, first and second body states may be counterparts of one another; body states including wrist extension, flexion, rotation, radial deviation, ulnar deviation, etc.; i.e. where a fist clenching or other user input used for opening a menu, analogous to a fourth type of gesture does not include multiple instances of a first body state such as wrist extension, flexion, rotation, deviation, etc., analogous to a first type of gesture, and where a finger pinch pose for clicking a selected item, analogous to a third type of gesture, does not include multiple instances of a second body state such as wrist extension, flexion, rotation, deviation, etc., counterpart to a first body state, analogous to a second type of gesture ).
With respect to claims 14, 34, and 52, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the one or more selectable options includes a first selectable user interface object for changing operations that one or more hand gestures can cause to the computer system to perform, wherein selection of the first selectable user interface object causes the computer system to display a plurality of settings, wherein each setting controls an operation that the one or more hand gestures can cause the computer system to perform when the one or more hand gestures are detected by the computer system (e.g. paragraph 1483, Fig. 36S, multi-state user interface configurable/programmable by user; user able to specify mappings between body states and responsive actions; such as mapping body part sates of ring/middle/pinky/index finger pinches, first pose, and open hand pose to scroll up/down, activate/deactivate link, increment/decrement selector actions; paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H; paragraph 1486, Fig. 36I, user modifying mappings between body states and actions; configuring web navigation settings by selecting setting options shown in drop-down menu of web browser; user clicking and selecting “scroll:options” to configure type of pose that initiates scrolling control or feature; paragraph 1487, Fig. 36J, link activate setting used to highlight links included in web page; user clicking on “linksActivate:options” to configure type of pose and predetermined pose duration that initiates the activation and rendering of link displayed in webpage; paragraph 1488, Fig. 36K, user clicking on “click:options” to configure type of pose and predetermined duration that initiates a click action in an application; i.e. selection of the “Control Options” dropdown object of Fig. 36G causes the plurality of settings to be shown in Fig. 36H, where these settings allow control of which gestures/poses are mapped to which operations).
With respect to claims 15, 35, and 53, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein:
the one or more selectable options includes a second selectable option for transitioning the computer system to a fourth operation mode and a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode (e.g. paragraph 1485, Fig. 36G, multi-state user interface facilitates enabling and disabling body state control using “enabled” button; drop down menu control from which user is able to select certain display settings and map body states to responsive actions; see also “reset settings” button in Fig. 36G;  i.e. where an “enabled” button can be selected to transition the system to a state/mode in which the body state controls are enabled, and to a state/mode in which body state controls are disabled, analogous to at least a second selectable option for transitioning the computer system to a fourth operation mode (i.e. a body state controls disabled mode); further where the user selects the drop down menu and modifies the mappings of body states and responsive actions, this causes the system to be transitioned from a first mode (i.e. a mode in which a current set of body state/action mappings are utilized) to a different/fifth mode (i.e. a mode in which a modified/custom set of body state/action mappings are utilized), analogous to a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode; in addition, where body state/action mappings are already customized/modified, and the user selects the option to “reset settings” this would cause the system to be transitioned from a first mode (i.e. a mode in which a current set of modified/customized body state/action mappings are utilized) to a different/fifth mode (i.e. a mode in which a default/reset set of body state/action mappings are utilized), analogous to a third selectable option for transitioning the computer system to a fifth operation mode that is different from the fourth operation mode);
selection of the second selectable option causes the computer system to transition to the fourth operation mode (e.g. paragraph 1485, Fig. 36G, enabling/disabling body state control in using “enabled” button, as discussed above); and
selection of the third selectable option causes the computer system to transition to the fifth operation mode (e.g. paragraph 1485, Fig. 36G, user selecting drop down menu control to map body states to responsive actions; user selecting to “reset settings”, as discussed above).
With respect to claims 16, 36, and 54, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the one or more selectable options include a fourth selectable option for displaying one or more additional selectable options, and wherein selection of the fourth selectable option causes the computer system to display the one or more additional selectable options that were not previously displayed before selection of the fourth selectable option (e.g. paragraph 1483, Fig. 36S, multi-state user interface configurable/programmable by user; user able to specify mappings between body states and responsive actions; such as mapping body part sates of ring/middle/pinky/index finger pinches, first pose, and open hand pose to scroll up/down, activate/deactivate link, increment/decrement selector actions; paragraph 1485, describing Figs. 36G-K, popup/dialog box including options to enable/disable body state control using the “enabled” button, and drop-down menu controls from which user is able to select certain display settings and map body states to responsive actions, as shown in Fig. 36H; paragraph 1486, Fig. 36I, user modifying mappings between body states and actions; configuring web navigation settings by selecting setting options shown in drop-down menu of web browser; user clicking and selecting “scroll:options” to configure type of pose that initiates scrolling control or feature; paragraph 1487, Fig. 36J, link activate setting used to highlight links included in web page; user clicking on “linksActivate:options” to configure type of pose and predetermined pose duration that initiates the activation and rendering of link displayed in webpage; paragraph 1488, Fig. 36K, user clicking on “click:options” to configure type of pose and predetermined duration that initiates a click action in an application; i.e. selection of the “Control Options” dropdown object of Fig. 36G, analogous to a fourth selectable option, causes additional selectable options, not previously displayed, to be displayed, as shown in Fig. 36H).
With respect to claims 19, 39, and 57, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
after displaying the user interface that includes the first user interface object, detecting, via at least the optical sensor, a fourth hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, fist clenching; paragraph 1463, Fig. 35O, fist pose; i.e. while/after the menu including the first user interface object is being/has been previously displayed, or at some later time, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the fourth hand gesture:
in accordance with a determination that the fourth hand gesture is a fifth type of gesture that is different from the first type of gesture and the second type of gesture, transitioning the computer system from an inactive state to an active state (e.g. paragraph 0509, mapping neuromuscular signals to control signals to control device based on type of gesture; paragraph 0618, determining the gesture being performed; paragraph 0620, fist clenching by user results in wake signal being sent to the system; paragraph 0703, first muscular activation state detected used to determine that a wake up mode is to be controlled, and second muscular activation state used to second activation state used to control initialization operation of the wake up mode; paragraph 0706, wake up/initialization mode; paragraph 1380, mappings causing device to perform action in response to detection of specific body part states; paragraph 1463, Fig. 35O, fist pose mapped to command to display, enter, and activate radial menu; i.e. when the system determines, based on a mapping body part states/gestures to actions, that the user is performing a fist pose hand gesture or other suitable gesture, the system may perform a corresponding action of activating a menu, or waking the system from an inactive/shut down state, analogous to transitioning the system from an inactive state to an active state).
With respect to claims 20, 40, and 58, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying, via the display generation component, the indication that the second user interface object is selected, detecting, via at least the optical sensor, a fifth hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraphs 1473-1475, first body state mapped to selection of next indicator in given sequence; first and second body states may represent counterparts; first body state mapped to selection of next visual indicator in sequence, second body state mapped to selection of previous visual indicator in sequence; i.e. while the interface/menu including the first, second, and third objects is being displayed, including after the user has performed a gesture causing a second user interface object to be selected and while the second user interface object is displayed with an indication that it is selected, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the fifth hand gesture:
in accordance with a determination that the fifth hand gesture is the first type of gesture, displaying, via the display generation component, an indication that a fourth first user interface object is selected (e.g. paragraphs 1473-1475, first body state mapped to selection of next indicator in given sequence; in response to identification of first body state, deselecting indicator 351111 and selecting indicator 351113, providing visual feedback; Figs. 36B-E; i.e., where the user has already performed the first type of gesture once, causing a second user interface object to be selected as shown in Fig. 36C, the user may perform another instance of the first type of gesture, which is mapped to selection of a next indicator, such that after this second instance of the first type of gesture is performed, a next/subsequent indicator (with respect to currently selected indicator 351113, such as the next indicator below it) will be selected, and indicated as selected via visual feedback, analogous to displaying an indication that a fourth UI object is selected in accordance with a determination that the fifth hand gesture is the first type of gesture (Examiner notes that a similar rationale would be applicable in the interfaces of Figs. 36D-E, such that if a user performs a first instance of the first type of gesture to cause item “E” to be selected as the next in sequence, and then performs a second instance of the first type of gesture, this will cause the item “F” to be selected as the next item in the sequence, and indicated as such)); and
in accordance a determination that the fifth hand gesture is the second type of gesture, displaying, via the display generation component, the indication that the first user interface object is selected (paragraphs 1473-1475, first and second body states may represent counterparts; second body state mapped to selection of previous visual indicator in sequence; i.e., where the user has already performed the first type of gesture, causing a second user interface object to be selected as shown in Fig. 36C, the user may perform the second type of gesture, which is mapped to selection of a previous indicator, such that after this second type of gesture is performed, a previous indicator (with respect to currently selected indicator 351113, such as the previous indicator above it) will be selected, and indicated as selected via visual feedback (i.e. the selection will be reverted back to the state shown in Fig. 36B), analogous to displaying an indication that the first UI object is selected in accordance with a determination that the fifth hand gesture is the second type of gesture (Examiner notes that a similar rationale would be applicable in the interfaces of Figs. 36D-E, such that if a user performs a first instance of the first type of gesture to cause item “E” to be selected as the next in sequence, and then performs the second type of gesture, this will cause the item “D” to be selected as the previous item in the sequence, and indicated as such, similar to what is shown in Fig. 36D)).
With respect to claims 21, 41, and 59, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying the user interface that includes the first user interface object, detecting a request to transition the computer system from the first operation mode to a fourth operation mode (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 1470, Fig. 35 R, open hand pose, mapped to command to deactivate the menu; i.e. while the menu is being displayed, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected, including a gesture to request transitioning from a first operation mode (such as one in which a menu is activated) to a fourth operation mode (such as one in which the menu is deactivated/disabled));
in response to detecting the request to transition the computer system from the first operation mode to the fourth operation mode, transitioning the computer system from the first operation mode to the fourth operation mode (e.g. paragraph 1461, disabled radial menu; paragraph 1471, deactivating radial menu; i.e. in response to detecting the user’s open hand gesture, the radial menu is disabled/deactivated, transitioning from an operation mode in which the radial menu is activated (analogous to a first operation mode) to an operation mode in which the radial menu is deactivated (analogous to a fourth operation mode));
while the computer system is in the fourth operation mode and while displaying the user interface that includes the first user interface object, the second user interface object, and the third user interface object, and the indication that the first user interface object is selected (e.g. paragraph 1471, state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; i.e. even after the radial menu is deactivated, all user interface objects, and selection indicator (indicating which object is selected) may continue to be displayed on the display), detecting, via at least the optical sensor, a sixth hand gesture (e.g. continuously receiving neuromuscular/musculoskeletal signals and continuously performing corresponding gesture recognition as cited above); and
in response detecting, via least the optical sensor, the sixth hand gesture:
in accordance with a determination that the sixth hand gesture is the first type of gesture, continuing to display the indication that the first user interface object is selected (e.g. paragraph 1411, user rotating arm without making fist pose has no effect on the radial menu; paragraph 1461, disable radial menu is a mode in which the user is not able to interact with the radial menu; paragraph 1470, deactivated radial menu; paragraph 1471 state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; radial menu will not recognize/respond to subsequent selection (wrist gesture) or click command until receiving a subsequent activation command); i.e. after the radial menu is deactivated, all user interface objects including the selection indicator are continuously displayed and, while in the deactivated state, the user’s hand gestures, such as a first type of gesture to select a next object, will have no effect, and will not be recognized/responded to, such that the menu will continue to be displayed including the selection indicator (indicating a current action indicator as selected) in an unchanged state); and
in accordance with a determination that the sixth hand gesture is the second type of gesture, continuing to display the indication that the first user interface object is selected (e.g. paragraph 1411, user rotating arm without making fist pose has not effect on the radial menu; paragraph 1461, disable radial menu is a mode in which the user is not able to interact with the radial menu; paragraph 1470, deactivated radial menu; paragraph 1471 state indicator, selection indicator, and action indicators may remain displayed within the GUI following the deactivation command; radial menu will not recognize/respond to subsequent selection (wrist gesture) or click command until receiving a subsequent activation command); i.e. after the radial menu is deactivated, all user interface objects including the selection indicator are continuously displayed and, while in the deactivated state, the user’s hand gestures, such as a second type of gesture to select a previous object, will have no effect, and will not be recognized/responded to, such that the menu will continue to be displayed including the selection indicator (indicating a current action indicator as selected) in an unchanged state).
With respect to claims 18, 38, and 56, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
while displaying the menu that includes one or more selectable options, detecting, via at least the optical sensor, a third hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 1470, Fig. 35 R, open hand pose; i.e. while a menu is being displayed, the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected); and
in response to detecting, via at least the optical sensor, the third hand gesture:
in accordance with a determination that the third hand gesture is a type of gesture, ceasing to display the menu that includes the one or more selectable options (e.g. paragraph 0509, mapping neuromuscular signals to control signals to control device based on type of gesture; paragraph 0618, determining the gesture being performed; paragraph 1381, open hand pose may cause the wearable device to direct the computing device to close the menu; paragraph 1380, mappings causing device to perform action in response to detection of specific body part states; paragraph 1470, open hand pose mapped to command to hide, exit, or deactivate menu; following identification of open hand pose, some or all of menu may be obscured or hidden from view within the GUI; i.e. when the system determines, based on a mapping body part states/gestures to actions, that the user is performing an open hand gesture or other suitable gesture, the system may perform a corresponding action of ceasing to display a currently displayed menu).
Osborn does not explicitly disclose that the type of gesture is a fourth type of gesture.  However, Kim teaches that the type of gesture is a fourth type of gesture (e.g. paragraph 0256, Figs. 17A-B, while message check screen outputted through display, if command for switching to lock screen is received, stopping output of message check screen and outputting/switching to lock screen; command may include first input (which is shown as a clenched fist gesture in Figs. 17A-B); i.e. where a user performing a fist gesture to initiate/switch to a lock screen display would cause the system to cease displaying all currently displayed content, including menus).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Jose, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) and Jose (directed to providing content carousel functionality for presenting user interface contents according to configuration settings and user inputs/selections), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability, while displaying user interface content (i.e. such as a GUI including currently displayed menus, as taught by Osborn) to receive a hand gesture to transition to a lock screen state/mode, such as in response to a clenched fist hand gesture, and in response, cease displaying currently displayed content (such as the menu of Osborn) and display a lock screen (i.e. as taught by Kim).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claims 13, 33, and 51, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
detecting, via at least the optical sensor, a third hand gesture; and in response to detecting, via at least the optical sensor, the third hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected):
in accordance with a determination that the third gesture is the fourth type of gesture, performing an action (e.g. paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user); and
in accordance with a determination that a notification was received and in accordance with a determination that the third hand gesture is a type of gesture, performing an action related to the notification (e.g. paragraph 1649 and corresponding table, using swipe to engage/ignore notification; paragraphs 1651-1653, flick gesture may be used as dismissal mechanism for notifications; notifying user that next meeting is coming up, flick dismissing notification outright, index pinch tap engaging timer function, rolling wrist to select timer duration, etc.).
Osborn does not explicitly disclose that the determinations (for performing the action related to the notification) are that the notification was received within a threshold period of time and that the third hand gesture is the fourth type of gesture.  However, Kim teaches in accordance with a determination that a notification was received within a threshold period of time and in accordance with a determination that the third hand gesture is the fourth type of gesture, performing an action related to the notification (e.g. paragraph 0014, determining level of folding fingers of the user as being in various states (i.e. gestures) based on measured blood flow rate; paragraph 0186, Fig. 5A, user determined in sleep state, generated notifications controlled to not be outputted to user; when user wakes up, user can be informed of saved notifications; paragraph 0187, after detecting user has got out of sleep state, if prescribed command (such as making a fist twice) is received, outputting weather information through the display; paragraph 0190, if user wakes from sleep state, outputting notification screen 501 for cumulative notifications (notification list of plurality of notifications) to user, if prescribed command is received, such as input of making a fist twice, as shown in Figs. 5A-C; i.e. where a notification is received within a time period during which the user is sleeping (analogous to a threshold period of time) and then an input of making a fist gesture is received, an action related to the notification is performed, such as displaying the notification; paragraph 0227, Fig. 11A, call signal currently received, displaying call screen 1101 indicating call signal is currently being received; paragraph 0229, Figs. 11A-C, when call signal currently received, if first input maintained over prescribed time, making response to call signal, and outputting call connected screen 1103; i.e. as shown in Figs. 11A-C, if, during the time period that a call signal is being currently received and a corresponding notification screen is displayed (e.g., during the time period beginning when the call signal is first received and ending when the signal is no longer received, analogous to a threshold period of time within which a notification is received), if a user makes an input such as holding a fist for a time period, the call is answered, analogous to performing an action (answering the call) related to the notification (the screen indicating the call is being received)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Jose, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) and Jose (directed to providing content carousel functionality for presenting user interface contents according to configuration settings and user inputs/selections), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to detect the fourth type of gesture (i.e. of Osborn, such as a fist gesture for causing a menu to be displayed) and, based on determining that a received gesture is the fourth type of gesture/fist gesture and further determining that a notification has been received within a threshold time period (such as during a time period in which the user was sleeping, or during a time period in which a call signal is currently received and a corresponding notification is displayed), performing a related action, such as displaying the notification, or answering a corresponding phone call, etc., as taught by Kim.  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claims 7, 27, and 45, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
in response to detecting, via at least the optical sensor, the second hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected):
in accordance with a determination that the second hand gesture is the third type of gesture, performing a second operation (e.g. paragraph 1450, pinch pose 35700 mapped to activation action such as action 351900 in Fig. 36M; when user executes pinch pose for duration, directing/causing cursor 35902 to move toward element 351906 in GUI of corresponding application and possibly activate the element; paragraph 1451, providing feedback indicators, such as informing user that system has detected the pinch pose; paragraph 1455, pinch pose mapped to scrolling action such that when user performs pinch pose, the corresponding application scrolls GUI; i.e. if the system receives a second gesture which is a pinch pose, the system may perform some second operation, such as moving a cursor to a particular object, providing feedback indicators, scrolling a GUI etc.); and
in accordance with a determination that the second hand gesture is the fourth type of gesture, performing a third operation that is different from the second operation (e.g. paragraph 1270, switching between different types of control schemes/input modes, using fist to control virtual pointer in 2D; paragraph 1281, fist squeeze used for discrete event such as to engage or disengage a particular feature within a given application or a continuous event, such as to maintain activation of a particular feature within a given application; paragraph 1283, user selecting from options using hand gesture such as clenching fist; paragraph 1443, determining fist formed/held by user hand, controlling speed of cursor based on amount of force applied; paragraph 1539, using fist pose to delete characters, etc.; i.e. if the system receives a second gesture which is a fist pose, the system may perform some third operation such as controlling a virtual pointer/cursor and/or altering its speed, engaging or disengaging a feature, maintaining activation of a feature, selecting an option, deleting characters, etc.).
With respect to claims 9, 29, and 47, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
in response to detecting, via at least the optical sensor, the second hand gesture (e.g. paragraph 0272, interpreting sensed neuromuscular signals; continuously sensing and recording neuromuscular signals; estimating handstate; paragraph 0286, continuously sensing and recording auxiliary signals including from imaging devices, heart rate monitors etc.; determining gesture being performed by user based on estimate of handstate; paragraph 0442, sensors 19102 including plurality of neuromuscular sensors and also auxiliary sensors configured to continuously record auxiliary signals; paragraph 0445, signals continuously recorded by sensors 19102 and processed to provide had movement in real-time; paragraph 0509, neuromuscular signals continuously recorded, musculoskeletal model updated, control signals generated based on updated musculoskeletal position information; paragraph 0618, signals continuously sensed and recorded, musculoskeletal representation updated in real time; current handstate estimates used to determine gesture being performed by user; paragraph 0620, detection of force applied by finger/hand used to control system such as to open a help dialog interface, a global system properties menu, etc.; position, force, gesture related event such as user clenching his or her fist; combinations of neuromuscular signals used in conjunction with camera data to control the system; paragraph 1381, states of user’s body parts corresponding to control actions; fist pose causing wearable device to direct computing device to display radial menu for selection by user; i.e. the system continuously updates sensed neuromuscular information, including using optical sensors, and continuously determines a current gesture being performed by the user, such that a subsequent gesture by the user may be detected):
in accordance with a determination that the second hand gesture is a type of gesture, transitioning the computer system from the second operation mode to a third operation mode (e.g. paragraph 0648, switching between modes; paragraph 0666-0667, user selection of mode, such as from list or using gesture; paragraph 1380, transitioning between different mappings of body part states and responsive actions, such as according to neuromuscular signals/gestures; paragraph 1408-1410, activating first mapping and/or deactivating second mapping between possible body part states and responsive actions; user switching between mappings; transition/switch between mappings, such as based on detected states of body/gestures).
Osborn does not explicitly disclose in accordance with a determination that the second hand gesture is the fourth type of gesture, transitioning the computer system from the second operation mode to a third operation mode.  However, Kim teaches in accordance with a determination that the second hand gesture is the fourth type of gesture, transitioning the computer system from the second operation mode to a third operation mode (e.g. paragraph 0256, Figs. 17A-B, while message check screen outputted through display, if command for switching to lock screen is received, stopping output of message check screen and outputting/switching to lock screen; command may include first input (which is shown as a clenched fist gesture in Figs. 17A-B)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Jose, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) and Jose (directed to providing content carousel functionality for presenting user interface contents according to configuration settings and user inputs/selections), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to place the system in a second mode (i.e. such as a mode of executing a corresponding functionality following a user’s selection of a displayed item, as taught by Osborn, and a mode of executing received message display and interaction following a user’s selection of a displayed message notification, as taught by Kim), and while in this mode receive a hand gesture to transition to a lock screen state/mode, such as in response to a clenched fist hand gesture (i.e. as taught by Kim).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claims 10, 30, and 48, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the one or more programs further include instructions for:
in accordance with a determination that the second hand gesture is one type of gesture, displaying a notification that indicates a state of the automatic scrolling operation (e.g. paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling up a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll up a GUI; paragraph 1484, ring finger pinch mapped to scroll up so that when user makes ring finger pinch pose, the page/browser displayed scrolls up; paragraph 1459, providing visual feedback of both identified pose and identified force; when action includes scrolling GUI, displaying cursor in connection with scrolling, which may be presented as a horizontal line with a bar extending above or below the line, depending on the scrolling direction; paragraph 1480, providing current state/setting of interface, current body state, and current action corresponding to current body state; paragraph 1481, graphical elements displaying current interface state, current body state, and responsive action; i.e. based on determining that the user has provided a gesture for scrolling, the system may initiate the scroll operation and may also provide a notification/indicator regarding the state of the operation, such as a cursor indicating a scrolling direction and magnitude, etc.).
Kim teaches that the one type of gesture is the first type of gesture (e.g. paragraphs 0251-254, Figs. 16A-C, outputting notification screen of received text message, receiving message check command such as folding input and outputting message check screen 1601 for displaying details of received text message; scroll action on message check screen performed by rotation of wrist; wrist rotated in first direction, scrolling to move outputted message check screen; direction for rotating wrist includes scroll direction; if scroll in bottom direction desired, rotating wrist in bottom direction; i.e. the user may provide a wrist rotation input in a first or second direction in order to perform a scroll operation in a designated direction, as shown between Fig. 16B and Fig. 16C).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Jose, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) and Jose (directed to providing content carousel functionality for presenting user interface contents according to configuration settings and user inputs/selections), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to receive a hand gesture to initiate automatic scrolling in a direction, and a hand gesture to reverse the scrolling direction (i.e. such as using different fingers to perform pinch gestures to scroll up/down as taught by Kim), where the hand gesture may be a wrist rotation gesture in different directions corresponding to the desired scroll direction (i.e. as taught by Kim, where Osborn also teaches the use of wrist rotation for at least scrolling a wrist, and further teaches that the wrist rotation may be performed in two different directions, and further teaches, based on detecting the scroll gesture, outputting a notification/indicator of the scrolling operation state).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
With respect to claims 11, 31, and 49, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 4, 26, and 44 as previously discussed, and Osborn further teaches wherein the third type of gesture is a type of gesture that is an instance of the fourth type of gesture (e.g. paragraph 1281, fist squeeze used for discrete event such as to engage or disengage a particular feature within a given application or a continuous event, such as to maintain activation of a particular feature within a given application; paragraph 1283, selecting from presented options by performing hand gesture such as clenching the fist).
Osborn does not teach that the third type of gesture is multiple instances of the fourth type of gesture.  However, Kim teaches the third type of gesture is multiple instances of the fourth type of gesture (e.g. paragraph 0187, user command of making fist twice to cause output of weather information; i.e. as shown in Fig. 5A, where a UI element associated with a weather information check is currently displayed, the user may perform a gesture which is multiple instances of a fist clenching gesture in order to select/display content corresponding to the UI element).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Jose, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects) and Jose (directed to providing content carousel functionality for presenting user interface contents according to configuration settings and user inputs/selections), to incorporate the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to receive a hand gesture to select a displayed item on the screen(i.e. such as a fist clench gesture as taught by Osborn) as multiple instances of the hand gesture (i.e. such as receiving multiple fist clench gestures as taught by Kim).  One of ordinary skill would have been motivated to perform such a modification in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
Claims 2, 24, and 42 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Jose, further in view of Kim, further in view of Chung et al. (US 20220326779 A1).
With respect to claims 2, 24, and 42, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 1, 22, and 23 as previously discussed.  Although Osborn teaches that auxiliary sensors which may contribute to gesture detection may include heart rate sensors, PPG sensors, and cameras used to detect changes in blood flow within the user (e.g. paragraph 0281, sensors including cardio sensor to measure user’s pulse, heart rate; paragraph 0286, auxiliary device/sensor including heart-rate monitor; paragraph 0310, nonvisible light used to capture image of heat distribution, providing indication of blood flow, used to infer condition of user such as forces being exerted by certain fingers; paragraph 0729, PPG sensors detecting vascular changes), Osborn does not explicitly disclose wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor.
However, Chung teaches wherein the hand gesture is detected based on heart rate data that is determined using data detected via the optical sensor (e.g. paragraph 0093, sensor module sensing information for movement of electronic device, and may include photoplethysmogram (PPG) sensor; paragraph 0098, recognizing OCO and shake gestures, gestures of turning hand, moving hand with palm open, lifting arm, moving thumb, moving up/down hand, etc.; gesture may be recognized based on user’s biometric signal (e.g. a signal indicating relaxation/contraction of blood vessels or a heartbeat signal); paragraph 0104, gesture application invoking sensor API to activate sensors, such as PPG sensor; gesture application identifying information for gestures; paragraph 0136, OCO gesture, PPG sensor detecting change due to relaxation and contraction of blood vessels when user opens and closes her hand; ).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Jose, Kim, and Chung in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), Jose (directed to providing content carousel functionality for presenting user interface contents according to configuration settings and user inputs/selections), and Kim (directed to controlling a mobile terminal, such as using hand gestures), to incorporate the teachings of Chung (directed to recognizing gestures) to include the capability to detect the hand gesture based on heart rate data detected via the optical sensor, such as by detecting a variety of gestures using a PPG sensor, based on a user biometric signal indicating relaxation/contraction of blood vessels or heartbeat (as taught by Chung).  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability to recognize a gesture when there is a gesture recognition request from a specific application, eliminating the need for sensors to stay on and the gesture recognition process to keep on running, reducing power consumption and resource waste, and in order to make it possible to immediately recognize relatively small gestures, as described in Chung (paragraphs 0014-0015).
Claims 8, 28, and 46 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Jose, further in view of Kim, further in view of Platzer et al. (US 20130239046 A1).
With respect to claims 8, 28, and 46, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 1, 22, and 23 as previously discussed, and Osborn, Jose, and Kim further teach wherein:
the automatic scrolling operation scrolls a sequence of a plurality of interface objects in a second direction (e.g. Osborn paragraph 1379, wearable device facilitating browsing based on controls/commands, including scrolling down a webpage; paragraph 1401, actions mapped to body part states including scrolling through GUI; paragraph 1455, pinch pose mapped to scrolling action such that when user executes/performs pinch pose, application system directing/causing application to scroll down a GUI; paragraph 1484, pinky finger pinch mapped to scroll down so that when user makes pinky finger pinch pose, the page/browser displayed scrolls down; Jose paragraph 0025, Fig. 2, as content items are presented via the content carousel, content carousel playback controls 216 are displayed, enabling the user to manually pause the content carousel to manually skip backwards; paragraph 0041, noting that the user may specify, via configuration settings, the transition direction for presentation of content items, such as a user selection input for left to right, right to left, top to bottom, or bottom to top transition directions; i.e. see also Fig. 9, showing similar playback controls; i.e. the automatic scrolling operation may be performed in a second direction; Kim paragraphs 0251-254, Figs. 16A-C, outputting notification screen of received text message, receiving message check command such as folding input and outputting message check screen 1601 for displaying details of received text message; scroll action on message check screen performed by rotation of wrist; wrist rotated in first direction, scrolling to move outputted message check screen; direction for rotating wrist includes scroll direction; if scroll in bottom direction desired, rotating wrist in bottom direction; i.e. the user may provide a wrist rotation input in a first or second direction in order to perform a scroll operation in a designated direction, as shown between Fig. 16B and Fig. 16C).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Jose, and Kim in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects), to incorporate the teachings of Jose (directed to providing content carousel functionality for presenting user interface contents according to configuration settings and user inputs/selections) and the teachings of Kim (directed to controlling a mobile terminal, such as using hand gestures) to include the capability to perform the automatic scrolling operation in a second direction.  One of ordinary skill would have been motivated to perform such a modification in order to provide the capability for the user to view a large amount of content with only a single interaction with the user interface, as described in Jose (paragraphs 0049-0050), and in order to provide totally new types of input required for controlling highly portable mobile terminals, such as watch type terminals, obviating problems due to limitations and disadvantages of the related art, as described in Kim (paragraphs 0008-0009).
Osborn and Kim do not explicitly disclose wherein the one or more programs further includes instructions for:
while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects; and
in response to detecting the end of the sequence of the plurality of user interface objects, performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction.
However, Platzer teaches wherein the one or more programs further includes instructions for:
while performing the automatic scrolling operation, detecting an end of the sequence of the plurality of user interface objects (e.g. paragraph 0077, Figs. 5A-C, scrolling list of items to terminus of the list; paragraph 0078, terminus is edge of scrolled region; while list is being scrolled; terminus is virtual boundary associated with displayed objects; displayed objects reach or intersect with the terminus 514; i.e. the system detects the end of a list of displayed UI objects, as indicated by a terminus at the edge of the scrolled list); and
in response to detecting the end of the sequence of the plurality of user interface objects, performing a second automatic scrolling operation that scrolls the sequence of a plurality interface objects in a third direction that is different from the second direction (e.g. paragraph 0077, Figs. 5A-C, reaching terminus of list, smoothly bouncing displayed items at end of list, reversing direction; paragraph 0078, as illustrated in Fig. 5B, when objects reach or intersect with the terminus 514, the movement corresponding to scrolling may stop and, as illustrated in Fig. 5C, subsequently reverse direction; i.e. scrolling list reaches and bounces at its terminus; i.e. in response to reaching/detecting the terminus while scrolling in a second direction (such as opposite of a first direction), the system automatically performs a “bounce” by scrolling the list of displayed objects in the opposite direction (which may be a third direction, which may be the same as a first direction)).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Jose, Kim, and Platzer in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects, such as scrolling menus/lists of UI objects, webpages, etc.), Jose (directed to providing content carousel functionality for presenting user interface contents according to configuration settings and user inputs/selections), and Kim (directed to controlling a mobile terminal, such as using hand gestures), to incorporate the teachings of Platzer (directed to scrolling operations) to include the capability to, while scrolling user interface content, including a plurality of objects, in a given direction (i.e. as taught by Osborn, Kim, and Platzer, such as in a second direction which is opposite of a first scroll direction, or some other direction, where at least Jose teaches a scrolling direction may be set as left-right, right-left, up-down, or down-up), detect a terminus indicating the end of the plurality of objects and, in response automatically scroll the content including the plurality of objects in a reverse direction (as taught by Platzer, where this reverse direction may be a third direction which is different from the second direction, but potentially the same as a first direction).  One of ordinary skill would have been motivated to perform such a modification in order to provide operations desired by users, such as scrolling operations on a device, including on device having limited display size, user interface, etc. which limits the ease of use of the device, as described in Platzer (paragraphs 0006, 0008).
Claims 17, 37, and 55 are rejected under 35 U.S.C. 103 as being unpatentable over Osborn in view of Jose, further in view of Kim, further in view of Saurabh et al. (US 20180181261 A1).
With respect to claims 17, 37, and 55, Osborn in view of Jose, further in view of Kim teaches all of the limitations of claims 4, 26, and 44 as previously discussed.  Osborn does not explicitly disclose wherein the menu is displayed:
in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location; and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location.
However, Saurabh teaches wherein the menu is displayed:
in accordance with a determination that a respective user interface object is at a first location on the user interface, at the first location (e.g. abstract, positioning popup interface within display frame in relation to corresponding target element; relative positioning of popup interface to associated target element dynamically selected and determined based on location of corresponding target element within quadrants/subquandrants of display frame; paragraph 0001, presenting additional information/menu in the form of a popup interface; paragraph 0004, rendering popup interface at position determined to be best fit position with regard to target element’s position; paragraph 0005, determining dynamically made where to display the popup interface by determining quadrant the target element is actually displayed within; paragraph 0027, pop interface may refer to a context menu; paragraph 0028, target element linked to popup interface; target element may be word or icon, and popup interface may be a selection menu related to the word or icon; paragraph 0077, Fig. 8, target element 802 residing within subquadrant 5/quadrant 1, resulting popup interface 804 displayed at bottom position relative to target element; i.e. where a respective user interface element (target element) is displayed at a position within a first quadrant (analogous to a first location), the corresponding popup menu is also displayed at a position within the same quadrant/location); and
in accordance with a determination that the respective user interface object is not at the first location on the user interface, at a second location that is different from the first location (e.g. abstract, positioning popup interface within display frame in relation to corresponding target element; relative positioning of popup interface to associated target element dynamically selected and determined based on location of corresponding target element within quadrants/subquandrants of display frame; location of popup interface is dynamically adjustable as opposed to being positioned based on a predetermined location; paragraph 0004, rendering popup interface at position determined to be best fit position with regard to target element’s position, event when that target element changes position or otherwise moves; paragraph 0005, determining dynamically made where to display the popup interface by determining quadrant the target element is actually displayed within; as the target element moves within the user interface (due to scrolling etc.) the relative position of the popup interface is dynamically adjusted so that the popup interface is continuously displayed in a position identified as best fit relative to target element; paragraph 0083, Fig. 10, target element 1002 displayed spanning quadrant1 and quadrant2, resulting popup interface 1004 displayed in bottom position relative to target element 1002; i.e., where a respective user interface element is displayed at a different location such as a location spanning multiple quadrants (analogous to not being displayed at a first location), the corresponding popup menu may also be moved/repositioned on the display so that it is also displayed at a second location which is different from the first location).
Accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention having the teachings of Osborn, Jose, Kim, and Saurabh in front of him to have modified the teachings of Osborn (directed to wearable devices including neuromuscular sensors capable of detecting user gestures for controlling various interface objects, including displaying associated menus), Jose (directed to providing content carousel functionality for presenting user interface contents according to configuration settings and user inputs/selections), and Kim (directed to controlling a mobile terminal, such as using hand gestures), to incorporate the teachings of Saurabh (directed to a positioning mechanism for bubble as a custom tooltip, such as positioning popup interfaces/menus in relation to a corresponding target element) to include the capability to dynamically display the menu (i.e. of Osborn) in a position relative to the position of a corresponding UI object such that when the UI object is displayed at a first location, the menu is also displayed at the first location and when the UI object is not displayed at the first location, such as because it is moved to a different location, the menu is displayed at a second location different from the first location, such as at the different location that the UI object has been moved to (as taught by Saurabh).  One of ordinary skill would have been motivated to perform such a modification in order to solve disadvantages found in existing popup interfaces by providing the capability to dynamically render a popup interface at a position that is determined to be a best fit position with regard to a target elements position, event when that target element changes position or otherwise moves in some manner, as described in Saurabh (paragraphs 0002-0004).
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain,” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting in re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (GCPA 1968)). Further, a reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill the art, including nonpreferred embodiments. Merck & Co, v. Biocraft Laboratories, 874 F.2d 804, 10 USPQ2d 1843 (Fed. Cir.), cert, denied, 493 U.S. 975 (1989). See also Upsher-Smith Labs. v. Pamlab, LLC, 412 F,3d 1319, 1323, 75 USPQ2d 1213, 1215 (Fed. Cir, 2005): Celeritas Technologies Ltd. v. Rockwell International Corp., 150 F.3d 1354, 1361, 47 USPQ2d 1516, 1522-23 (Fed. Cir. 1998).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Yoon et al. (US 20150061842 A1) teaches various gestures for controlling a mobile device such as a wrist worn device (Figs. 7-33).
Offenberg et al (US 20150248235 A1) teaches various gestures for controlling a mobile device such as a wrist worn device (Figs. 27-92), as well as for selecting among displayed items with corresponding visual indications, scrolling, etc. (Figs. 102A-D).
Hazra et al. (US 20170123487 A1) teaches various gestures for controlling a mobile device including a wrist worn device (Figs. 5, 8-15).

CITED REFERENCE SUMMARIES: Ref US20220374085A1: Claims:
A computer system that is in communication with a display generation component, comprising: one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: displaying, via the display generation component, a user interface that includes a selectable user interface object and a cursor that is displayed at a first location on the user interface; while displaying the selectable user interface object and the cursor at the first location on the user interface, detecting a request to move the cursor from the first location to a second location on the user interface; and in response to detecting the request to move the cursor from the first location to the second location: displaying the cursor at the second location; in accordance with a determination that the second location corresponds to a location of the selectable user interface object, displaying an animation that provides a visual indication of how long the cursor needs to be located at the second location to perform an operation, wherein the visual indication is updated over a period of time; and in accordance with a determination that the second location does not correspond to the location of the selectable user interface object, forgoing displaying the animation.
one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: displaying, via the display generation component, a user interface that includes a selectable user interface object and a cursor that is displayed at a first location on the user interface; while displaying the selectable user interface object and the cursor at the first location on the user interface, detecting a request to move the cursor from the first location to a second location on the user interface; and in response to detecting the request to move the cursor from the first location to the second location: displaying the cursor at the second location; in accordance with a determination that the second location corresponds to a location of the selectable user interface object, displaying an animation that provides a visual indication of how long the cursor needs to be located at the second location to perform an operation, wherein the visual indication is updated over a period of time; and in accordance with a determination that the second location does not correspond to the location of the selectable user interface object, forgoing displaying the animation.
displaying, via the display generation component, a user interface that includes a selectable user interface object and a cursor that is displayed at a first location on the user interface;
while displaying the selectable user interface object and the cursor at the first location on the user interface, detecting a request to move the cursor from the first location to a second location on the user interface; and
in response to detecting the request to move the cursor from the first location to the second location: displaying the cursor at the second location; in accordance with a determination that the second location corresponds to a location of the selectable user interface object, displaying an animation that provides a visual indication of how long the cursor needs to be located at the second location to perform an operation, wherein the visual indication is updated over a period of time; and in accordance with a determination that the second location does not correspond to the location of the selectable user interface object, forgoing displaying the animation.
displaying the cursor at the second location;
in accordance with a determination that the second location corresponds to a location of the selectable user interface object, displaying an animation that provides a visual indication of how long the cursor needs to be located at the second location to perform an operation, wherein the visual indication is updated over a period of time; and
in accordance with a determination that the second location does not correspond to the location of the selectable user interface object, forgoing displaying the animation.
The computer system of claim 1 , wherein the one or more programs further include instructions for: in response to detecting the request to move the cursor from the first location to the second location: in accordance with a determination that the second location corresponds to the location of the selectable user interface object and the cursor is displayed at the second location for more than a first threshold period of time, performing the operation, wherein performing the operation includes selecting the selectable user interface object.
in response to detecting the request to move the cursor from the first location to the second location: in accordance with a determination that the second location corresponds to the location of the selectable user interface object and the cursor is displayed at the second location for more than a first threshold period of time, performing the operation, wherein performing the operation includes selecting the selectable user interface object.
in accordance with a determination that the second location corresponds to the location of the selectable user interface object and the cursor is displayed at the second location for more than a first threshold period of time, performing the operation, wherein performing the operation includes selecting the selectable user interface object.
The computer system of claim 1 , wherein the one or more programs further include instructions for: in response to detecting the request to move the cursor from the first location to the second location: in accordance with a determination that the second location corresponds to the location of the selectable user interface object and the cursor is displayed at the second location for more than a second threshold period of time, performing the operation, wherein performing the operation includes launching an application that corresponds to the selectable user interface object.
in response to detecting the request to move the cursor from the first location to the second location: in accordance with a determination that the second location corresponds to the location of the selectable user interface object and the cursor is displayed at the second location for more than a second threshold period of time, performing the operation, wherein performing the operation includes launching an application that corresponds to the selectable user interface object.
in accordance with a determination that the second location corresponds to the location of the selectable user interface object and the cursor is displayed at the second location for more than a second threshold period of time, performing the operation, wherein performing the operation includes launching an application that corresponds to the selectable user interface object.
The computer system of claim 1 , wherein the one or more programs further include instructions for: in response to detecting the request to move the cursor from the first location to the second location: in accordance with a determination that the second location does not correspond to the location of the selectable user interface object, forgoing performing the operation.
in response to detecting the request to move the cursor from the first location to the second location: in accordance with a determination that the second location does not correspond to the location of the selectable user interface object, forgoing performing the operation.
in accordance with a determination that the second location does not correspond to the location of the selectable user interface object, forgoing performing the operation.
The computer system of claim 1 , wherein the one or more programs further include instructions for: in response to detecting the request to move the cursor from the first location to the second location: in accordance with a determination that the cursor is not displayed at the second location for more than a third threshold period of time, forgoing performing the operation.
in response to detecting the request to move the cursor from the first location to the second location: in accordance with a determination that the cursor is not displayed at the second location for more than a third threshold period of time, forgoing performing the operation.
in accordance with a determination that the cursor is not displayed at the second location for more than a third threshold period of time, forgoing performing the operation.
The computer system of claim 1 , wherein the one or more programs further include instructions for: in response to detecting the request to move the cursor from the first location to the second location: in accordance with a determination that the second location corresponds to a location that is in an area of the display generated component and the cursor is displayed at the second location for more than a fourth threshold period of time, performing a second operation that includes displaying a second user interface that is different from the user interface.
in response to detecting the request to move the cursor from the first location to the second location: in accordance with a determination that the second location corresponds to a location that is in an area of the display generated component and the cursor is displayed at the second location for more than a fourth threshold period of time, performing a second operation that includes displaying a second user interface that is different from the user interface.
in accordance with a determination that the second location corresponds to a location that is in an area of the display generated component and the cursor is displayed at the second location for more than a fourth threshold period of time, performing a second operation that includes displaying a second user interface that is different from the user interface.
The computer system of claim 1 , wherein the one or more programs further include instructions for: in accordance with a determination that the second location corresponds to a location of the selectable user interface object and the cursor is displayed at the second location for more than a fifth threshold period of time, performing the operation, wherein performing the operation includes displaying, via the display generation component, a menu that includes one or more selectable options to preform one or more operations at the second location.
in accordance with a determination that the second location corresponds to a location of the selectable user interface object and the cursor is displayed at the second location for more than a fifth threshold period of time, performing the operation, wherein performing the operation includes displaying, via the display generation component, a menu that includes one or more selectable options to preform one or more operations at the second location.
The computer system of claim 7 , wherein the one or more selectable options include a first selectable option for performing a selection operation, and wherein selection of the first selectable option causes the computer system to perform the selection operation at the second location.
The computer system of claim 7 , wherein the one or more selectable options include a second selectable option for performing an operation that corresponds to an operation that is performed in response to an input being detected at the second location, and wherein selection of second selectable option causes the computer system to perform the operation that corresponds to the operation that is performed in response to the input being detected at the second location.
The computer system of claim 7 , wherein the one or more selectable options include a third selectable option for performing an automatic scroll operation, and wherein selection of the third selectable option causes the computer system to automatically scroll through a plurality of selectable user interface objects that include the selectable user interface object.
The computer system of claim 7 , wherein the one or more selectable options include a fourth selectable option for displaying one or more additional options, and wherein selection of the fourth selectable option causes the computer system to display the one or more additional options that were not previously displayed before selection of the fourth selectable option was detected.
The computer system of claim 11 , wherein the one or more additional options that were not previously displayed before selection of the fourth selectable option was detected includes a respective additional option, wherein selection of the respective additional option does not cause the computer system to perform an operation that is not performed at the second location.
The computer system of claim 7 , wherein: the computer system is in communication with one or more input devices; the one or more selectable options include a fifth selectable option for performing an operation that corresponds to an operation that is performed in response to detecting an input via the one or more input devices; and selection of the fifth selectable option causes the computer system to perform the operation that corresponds to the operation that is performed in response to detecting the input via the one or more input devices.
the computer system is in communication with one or more input devices;
the one or more selectable options include a fifth selectable option for performing an operation that corresponds to an operation that is performed in response to detecting an input via the one or more input devices; and
selection of the fifth selectable option causes the computer system to perform the operation that corresponds to the operation that is performed in response to detecting the input via the one or more input devices.
The computer system of claim 7 , wherein the one or more selectable options include a sixth selectable option and a seventh selectable option, and wherein the one or more programs further include instructions for: while displaying the menu, detecting a request to move the cursor from the second location to a third location on the user interface; in response to detecting the request to move the cursor from the second location to the third location on the user interface: in accordance with a determination that the third location corresponds to a location of the sixth selectable option and the cursor is displayed at the third location for more than a sixth threshold period of time, performing a first operation that corresponds to the sixth selectable option; and in accordance with a determination that the third location corresponds to a location of the seventh selectable option and the cursor is displayed at the third location for more than the sixth threshold period of time, performing a second operation that is different from the first operation.
while displaying the menu, detecting a request to move the cursor from the second location to a third location on the user interface;
in response to detecting the request to move the cursor from the second location to the third location on the user interface: in accordance with a determination that the third location corresponds to a location of the sixth selectable option and the cursor is displayed at the third location for more than a sixth threshold period of time, performing a first operation that corresponds to the sixth selectable option; and in accordance with a determination that the third location corresponds to a location of the seventh selectable option and the cursor is displayed at the third location for more than the sixth threshold period of time, performing a second operation that is different from the first operation.
in accordance with a determination that the third location corresponds to a location of the sixth selectable option and the cursor is displayed at the third location for more than a sixth threshold period of time, performing a first operation that corresponds to the sixth selectable option; and
in accordance with a determination that the third location corresponds to a location of the seventh selectable option and the cursor is displayed at the third location for more than the sixth threshold period of time, performing a second operation that is different from the first operation.
The computer system of claim 7 , wherein the one or operations include an operation that causes the computer system to display a user interface for an application that corresponds to the selectable user interface object.
The computer system of claim 7 , wherein the one or more operations include an operation that causes the computer system to transitioned from a first operation mode to a second operation mode.
The computer system of claim 7 , wherein the one or more programs further include instructions for: after displaying the menu that includes the one or more selectable options, detecting a request to display a second user interface that is different from the user interface; in response to detecting the request to display the second user interface, displaying the second user interface, wherein the second user interface does not include the menu and includes a second selectable user interface object and the cursor; and while displaying the second selectable user interface object and the cursor, detecting a request to move the cursor from a fourth location on the second user interface to a fifth location on the second user interface; and in response to detecting the request to move the cursor from the fourth location on the second user interface to a fifth location on the second user interface: displaying the cursor at the fifth location on the second user interface; and in accordance with a determination that the fifth location corresponds to a location of the second selectable user interface object and the cursor is displayed at the second location for more than a seventh threshold period of time, re-displaying, via the display generation component, the menu that includes the one or more selectable options.
after displaying the menu that includes the one or more selectable options, detecting a request to display a second user interface that is different from the user interface;
in response to detecting the request to display the second user interface, displaying the second user interface, wherein the second user interface does not include the menu and includes a second selectable user interface object and the cursor; and
while displaying the second selectable user interface object and the cursor, detecting a request to move the cursor from a fourth location on the second user interface to a fifth location on the second user interface; and
in response to detecting the request to move the cursor from the fourth location on the second user interface to a fifth location on the second user interface: displaying the cursor at the fifth location on the second user interface; and in accordance with a determination that the fifth location corresponds to a location of the second selectable user interface object and the cursor is displayed at the second location for more than a seventh threshold period of time, re-displaying, via the display generation component, the menu that includes the one or more selectable options.
displaying the cursor at the fifth location on the second user interface; and
in accordance with a determination that the fifth location corresponds to a location of the second selectable user interface object and the cursor is displayed at the second location for more than a seventh threshold period of time, re-displaying, via the display generation component, the menu that includes the one or more selectable options.
The computer system of claim 1 , wherein the request to move
[Truncated to input limit]