Office Action 1: Non-Final Rejection (2024-07-18)
Rejection types: 103 - Obviousness

CLAIMS:
CLAIMS

What is claimed is:

1\. A gesture recognition method, comprising:
obtaining a first feature map that is a feature map corresponding to a target region in a depth image, and determining a target key point from the first feature map;
establishing a three-dimensional coordinate system in the first feature map by using the target key point, and predicting other three-dimensional coordinates of other key points in the three- dimensional coordinate system, the other key points being key points other than the target key point in the first feature map; and
determining, based on the target key point and the other three-dimensional coordinates, a plurality of absolute coordinates of a plurality of key points, to determine a gesture of a target recognition object in the target region based on the plurality of absolute coordinates, the plurality of key points comprising the target key point and the other key points.

2\. The method according to claim 1, wherein said predicting the other three-dimensional coordinate of the other key points in the three-dimensional coordinate system comprises:
inputting the first feature map into an XYZ encoder and a fully connected layer sequentially, to obtain the other three-dimensional coordinates corresponding to the other key points based on the three-dimensional coordinate system.

3\. The method according to claim 1, wherein said determining, based on the target key point and the other three-dimensional coordinates, the plurality of absolute coordinates of the plurality of key points comprises:
determining target absolute coordinates of the target key point; and
determining, based on the target absolute coordinates and the other three-dimensional coordinates, the plurality of absolute coordinates.

4\. The method according to claim 3, wherein said determining the target absolute coordinates of the target key point comprises: obtaining a two-dimensional position of the target key point;
determining, based on the two-dimensional position, a target absolute depth of the target key point; and
determining, based on the target absolute depth of the target key point, the target absolute coordinates of the target key point.

5\. The method according to claim 4, wherein said obtaining the two-dimensional position of the target key point comprises:
inputting the first feature map into a UV encoder to obtain target UV coordinates of the target key point; and
determining the target UV coordinates as the two-dimensional position.

6\. The method according to claim 4, wherein said determining, based on the two-dimensional position, the target absolute depth of the target key point comprises:
in response to determining that the two-dimensional position is located in a background of the depth image, determining depth values of pixels within a predetermined range having the two- dimensional position as a center, and determining a closest depth from the depth values of the pixels within the predetermined range; and
determining the target absolute depth by using the closest depth.

7\. The method according to claim 1, wherein the target recognition object is a hand, and the target key point is a key point of a thumb root.

8\. A gesture recognition apparatus, comprising:
a processor; and
a memory, wherein the processor, when executing an operating program stored on the memory, implements a gesture recognition method comprising:
obtaining a first feature map that is a feature map corresponding to a target region in a depth image, and determining a target key point from the first feature map;
establishing a three-dimensional coordinate system in the first feature map by using the target key point, and predicting other three-dimensional coordinates of other key points in the three- dimensional coordinate system, the other key points being key points other than the target key point in the first feature map; and
determining, based on the target key point and the other three-dimensional coordinates, a plurality of absolute coordinates of a plurality of key points, to determine a gesture of a target recognition object in the target region based on the plurality of absolute coordinates, the plurality of key points comprising the target key point and the other key points.

9\. The apparatus according to claim 8, wherein the processor, when executing the operating program stored on the memory, implements:
inputting the first feature map into an XYZ encoder and a fully connected layer sequentially, to obtain the other three-dimensional coordinates corresponding to the other key points based on the three-dimensional coordinate system.

10\. The apparatus according to claim 8, wherein the processor, when executing the operating program stored on the memory, implements:
determining target absolute coordinates of the target key point; and
determining, based on the target absolute coordinates and the other three-dimensional coordinates, the plurality of absolute coordinates.

11\. The apparatus according to claim 10, wherein the processor, when executing the operating program stored on the memory, implements:
obtaining a two-dimensional position of the target key point;
determining, based on the two-dimensional position, a target absolute depth of the target key point; and
determining, based on the target absolute depth of the target key point, the target absolute coordinates of the target key point.

12\. The apparatus according to claim 11, wherein the processor, when executing the operating program stored on the memory, implements:
inputting the first feature map into a UV encoder to obtain target UV coordinates of the target key point; and determining the target UV coordinates as the two-dimensional position.

13\. The apparatus according to claim 11, wherein the processor, when executing the operating program stored on the memory, implements:
in response to determining that the two-dimensional position is located in a background of the depth image, determining depth values of pixels within a predetermined range having the two- dimensional position as a center. and determining a closest depth from the depth values of the pixels within the predetermined range; and
determining the target absolute depth by using the closest depth.

14\. The apparatus according to claim 8, wherein the target recognition object is a hand, and
the target key point is a key point of a thumb root.

15\. A storage medium, having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method according to claim 1. SMRH:4855-1532-6752.218 SMRH:4855-1532-6752.219 SMRH:4855-1532-6752.220 SMRH:4855-1532-6752.221

OFFICE ACTION:
DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA  35 U.S.C. 102 and 103 (or as subject to pre-AIA  35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA  to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-15 are rejected under 35 U.S.C. 103 as being unpatentable over CN109308459 (hereinafter, Tong et al.) in view of CN108549489 (hereinafter, Chunyan et al.).
Regarding claims 1, 8 and 15, Tong discloses a gesture recognition method and apparatus, comprising: a processor; and a memory, wherein the processor, when executing an operating program stored on the memory (paragraph 2, lines 1-2: “The invention relates to the field of computer vision applications, in particular to a gesture estimation method”), implements a gesture recognition method comprising, comprising:
obtaining a first feature map that is a feature map corresponding to a target region in a depth image, and determining a target key point from the first feature map (paragraph 16: “Step 3-1 … obtain a feature map F” Paragraph 30, lines 1-3: “using the existing public gesture data set or collecting the gesture depth image according to the application requirement, manually marking the required key points … performing target model training on the ASNet network”;
establishing a three-dimensional coordinate system in the first feature map, and predicting other three-dimensional coordinates of other key points in the three-dimensional coordinate system, the other key points being key points other than the target key point in the first feature map (paragraph 30, lines 5-7: “It is necessary to perform an inverse process of the preprocessing to obtain the actual three-dimensional coordinates of the key points, and the three-dimensional coordinates of all the key points.”); and
determining, based on the target key point and the other three-dimensional coordinates, a plurality of absolute coordinates of a plurality of key points, to determine a gesture of a target recognition object in the target region based on the plurality of absolute coordinates, the plurality of key points comprising the target key point and the other key points (paragraph 12, lines 1-2: “Step 4: Target Model Training: Perform target model training on the ASNet network, and then use the trained model to output the spatial coordinates of the required key points. The coordinates of these key points constitute a gesture representation estimate.”).
Tong fails to disclose establishing a coordinate system in the feature map by using the target key point.
However, in an analogous art, Chunyan discloses establishing a coordinate system in a feature map by using a target key point (paragraph 13: Computing coordinates in a relative coordinate system with origin in one of the key points).  It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Tong by incorporating this feature taught in Chunyan for the purpose of obviating the need for a user to view a gesture device in real time.
Regarding claims 2 and 9, Tong discloses that said predicting the other three-dimensional coordinate of the other key points in the three-dimensional coordinate system comprises: inputting the first feature map into an XYZ encoder and a fully connected layer sequentially, to obtain the other three-dimensional coordinates corresponding to the other key points based on the three-dimensional coordinate system (paragraph 54, line 10).
Regarding claims 3 and 10, Tong discloses that said determining, based on the target key point and the other three-dimensional coordinates, the plurality of absolute coordinates of the plurality of key points comprises: determining target absolute coordinates of the target key point; and determining, based on the target absolute coordinates and the other three-dimensional coordinates, the plurality of absolute coordinates (paragraph 64, lines 16-22).
Regarding claims 4 and 11, Tong discloses that said determining the target absolute coordinates of the target key point comprises: obtaining a two-dimensional position of the target key point; determining, based on the two-dimensional position, a target absolute depth of the target key point; and determining, based on the target absolute depth of the target key point, the target absolute coordinates of the target key point (paragraph 47, lines 1-5).
Regarding claims 5 and 12, Tong discloses that said obtaining the two-dimensional position of the target key point comprises: inputting the first feature map into a UV encoder to obtain target UV coordinates of the target key point; and determining the target UV coordinates as the two-dimensional position (paragraph 54, line 10).
Regarding claims 6 and 13, Tong discloses that said determining, based on the two-dimensional position, the target absolute depth of the target key point comprises: in response to determining that the two-dimensional position is located in a background of the depth image, determining depth values of pixels within a predetermined range having the two-dimensional position as a center, and determining a closest depth from the depth values of the pixels within the predetermined range; and determining the target absolute depth by using the closest depth (paragraph 51, lines 1-7).
Regarding claims 7 and 14, Tong discloses that the target recognition object is a hand, and the target key point is a key point of a thumb root (paragraph 46, line 17).
Conclusion

APPLICANT REBUTTAL: Status of the Claims

Upon entry of the present amendment, claims 1-6, 8-13 and 15 will be pending in this application. Claim 1 has been amended to incorporate the limitations recited in claim 7 and limitations of "by taking the target key point as the coordinate origin" recited in paragraph [0053] of the specification. Accordingly, claim 7 has been cancelled. Claim 8 has been amended to incorporate the limitations recited in claim 14 and limitations of "by taking the target key point as the coordinate origin" recited in paragraph [0053] of the specification. Accordingly, claim 14 has been cancelled. No new matter has been introduced by the above amendments. Based on the following remarks, Applicant respectfully requests reconsideration and allowance of the pending claims.

Claim Rejections under 35 U.S.C. 103

I. REJECTED CLAIMS 1-15 UNDER 35 U.S.C. 103 AS BEING UNPATENTABLE OVER TONG ET

AL. (CN 109308459, HEREINAFTER AS "TONG") IN VIEW OF CHUNYAN ET AL. (CN

108549489, HEREINAFTER AS "CHUNYAN")

Applicant submits that claim 1 as amended is patentable under 35 U.S.C. 103 over Tong in view of Chunyan because none of Tong and Chunyan disclose at least the following limitations in claim 1 as amended:

\- Feature (A): the target key point being a key point of a thumb root;

\- Feature (B): establishing a three-dimensional coordinate system in the first feature map by taking the target key point as a coordinate origin.

For feature (A), claim 1 as amended is not taught or suggested in Tong. Tong discloses, in combination of paragraph [0041] and FIG. 2a of Tong as reproduced below, label 16 and 21 key points are labelled in FIG. 2a and FIG. 2c, respectively. As shown in FIG. 2a below, 16 key points are divided into 5 groups to represent 5 fingers. Each of 5 groups owns a same key point at palm center, which may be considered as the target key point. That is, the target key point in Tong is located at palm center of the hand.

 1.4931.43FIG. 2a of Tong

In addition, Chunyan discloses that when the hand coordinate system o'- x'y'z' is extended and opened, the intersection point between the line where the thumb is located and the line where the middle finger is located is the origin, the line where the middle finger is located is the x 'axis, and the direction pointed by the fingertip is positive; the vertical palm direction is the y 'axis direction.

In contrast, the thumb root is used as the target key point in claim 1 as amended.

For feature (B), as described above, in paragraph [0027], Tong discloses the output value of the model is the coordinates of all key points in the input image obtained by preprocessing the original depth image in step 2. A preprocessing inverse process is required to obtain the actual three-dimensional coordinates of the key points, which are used to represent the estimation of the current gesture using the three-dimensional coordinates of all key points. That is to say, actual three-dimensional coordinates are used to estimate the current gesture instead of three-dimensional coordinates in local hand three-dimensional system. When there is no local three-dimensional coordinate system, it is impossible for Tong to disclose the above feature (B).

Moreover, as described above, Chunyan discloses taking the intersection point between the line where the thumb is located and the line where the middle finger is located as the origin instead of thumb root in claim 1 as amended.

In contrast, the thumb root is the target key point, which is used as an origin to establish a three-dimensional coordinate system in the first feature map.

Therefore, claim 1 as amended is patentable over the cited references, individually or in combination.

Claims 2-6 and 15 depend on claim 1 as amended and include further limitations in addition to the limitation in claim 1 as amended. Therefore, claims 2-6 and 15 are patentable for at least similar reasons as claim 1 as amended is patentable.

The arguments regarding claim 1 apply also to independent claim 8 as amended. Therefore, independent claim 8 as amended is also patentable over the cited documents.

Claims 9-13 depend on claim 8 as amended and include further limitations in addition to the limitation in claim 8 as amended. Therefore, claims 9-13 are patentable for at least similar reasons as claim 8 as amended is patentable.

CONCLUSION

In view of the foregoing amendments and remarks, Applicant respectfully requests reconsideration of this application and the timely allowance of the pending claims.

If the Examiner believes, for any reason, that personal communication will expedite prosecution of this application, the Examiner is invited to telephone the undersigned at the number provided. Please grant any extensions of time required to enter this response and charge any additional required fees to Deposit Account No. 50-4561.

Respectfully Submitted,

SHEPPARD MULLIN RICHTER & HAMPTON LLP Dated: September 24, 2024 By: /Weiguo Chen/

Weiguo Chen

Reg. No. 61878

Phone: 650-815-2692

NEXT OFFICE ACTION: DETAILED ACTION
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
EXAMINER’S AMENDMENT
An examiner’s amendment to the record appears below. Should the changes and/or additions be unacceptable to applicant, an amendment may be filed as provided by 37 CFR 1.312. To ensure consideration of such an amendment, it MUST be submitted no later than the payment of the issue fee.
The application has been amended as follows:  in claim 15, line 1, “non-transitory” is inserted between “A” and “storage.”
Allowable Subject Matter
Claims 1-6, 8-13 and 15 are allowed.
The following is an examiner’s statement of reasons for allowance: the prior art fails to disclose the recited combinations of elements in gesture recognition, including obtaining a first feature map that is a feature map corresponding to a target region in a depth image, and determining a target key point from the first feature map, the target recognition object being a hand, and the target key point being a key point of a thumb root; establishing a three-dimensional coordinate system in the first feature map by taking the target key point as a coordinate origin, and predicting other three- dimensional coordinates of other key points in the three-dimensional coordinate system, the other key points being key points other than the target key point in the first feature map, as in claims 1 and 8.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee.  Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”

CITED REFERENCE SUMMARIES: Ref CN109308459: Claims:
Mappings , called A 0 , A 1 , . A 1x1 convolutional layer using the sigmoid excitation function S(x), ie Set the size of F as (W, H, C), where W, H, and C are the width, height, and number of channels, respectively. After convolution, an image Mi of size (W, H) is obtained, and where Each pixel value of is used as the weight of the corresponding pixel in the feature map F, where M i is called a weighted image; the feature map F is given corresponding weights through Mi to obtain the corresponding attention feature A i ; i obtained by each FAM through an AP layer and an FC layer, and then send it into the JSM structure. JSM pays attention to the correlation between the key points in the same group, so as to obtain the feature B i , A total of n features B 0 , B 1 , ..., B n -1 are obtained; 3-4 , using B 0 , B 1 , . The output size of the fully connected layer is m × 3, and all the outputs are formed into a vector of N × 3 This vector is the coordinate representation of the key points predicted by the model in the input image, where N refers to the total number of all key points;
i is calculated by the following formula: i , so that the independence of n groups of key points is modeled, where the and F x, y, c correspond to the value at coordinates (x, y, c) in feature map A and the value at coordinates (x, y, c) in feature map F, respectively, and corresponds to the value at coordinates (x, y) in the image Mi.
i , first Through a 16 × 16 AP layer and a FC layer containing 64 × m neurons, and then resizing the output of the FC layer to (m, 64), at this time such an output is regarded as m small blocks, the The m small blocks are strung together in spatial order by using GRU to obtain m optimized small blocks. Finally, the m optimized small blocks are stored as a vector, that is, an optimized feature, called B i , with a length of 64×m.
Ref CN108549489: Claims:
1. a kind of gestural control method based on hand form, posture, position and motion feature, which is characterized in that the method
Including：
Step 1: reading the image data of input picture；
Step 2: the hand-characteristic in one described image data of extraction step, and obtain hand-characteristic result；The hand-characteristic
Including hand morphological feature, hand gestures feature, hand position feature and hand motion feature；The hand-characteristic result includes
Hand morphological feature result, hand gestures characteristic results, hand position characteristic results and hand motion characteristic results；
Step 3: being merged, being analyzed and gesture identification to the hand-characteristic data described in step 2, gesture identification knot is obtained
Fruit；
Step 4: gesture identification result described in storing step three；
Step 5: whether the gesture identification result of judgment step four is complete gesture, it, then will transmission and institute if it is complete gesture
State the corresponding gesture command of gesture identification result；If not complete gesture, then iteration execute step 1 to step 4 until
Until recognizing complete gesture.
2. gestural control method according to claim 1, which is characterized in that the hand-characteristic is in world coordinate system and hand
It is described in portion's coordinate system；Absolute coordinate systems of the world coordinate system o-xyz as whole system；The hand coordinate
When being that o '-x ' y ' z ' are stretched flat opening with palm, straight line where thumb is origin, straight line where middle finger with straight-line intersection where middle finger
For x ' axis, finger tip direction is forward direction；Vertical volar direction is y ' axis directions, when using the right hand as control hand, palm
Direction is forward direction, the use of the back of the hand direction when left hand is forward direction；The plane that z ' axis is formed perpendicular to x ', y ' axis, thumb direction are just
To.
3. gestural control method according to claim 1, which is characterized in that hand morphological feature described in step 2 passes through image
Fractal method extracts, and the extraction process of described image fractal method includes：
Step 1, the image data for reading input picture；
Step 2, using the gradation of image or colouring information of step 1 described image data into row threshold division, obtain hand region；
The edge contour of hand region described in step 3, extraction step 2 obtains hand edge profile results；
Step 4 analyzes hand edge profile results described in step 3, bent by the edge contour in edge contour result
The protrusion of line judges finger number and direction with recess, obtains the hand morphological feature result；
Hand morphological feature result described in step 5, output step 4.
4. gestural control method according to claim 1, which is characterized in that hand morphological feature described in step 2 passes through module
Matching process extracts, and the extraction process of the module matching process includes：
The first step, the image data for reading input picture, and be loaded into and preset template data；
Second step uses correlation filtering method to the default template based on gray scale or color characteristic preset described in the first step in template
Carry out the full figure matching of described image；It is special to presetting the textural characteristics based on HOG, Haar or LPB in template described in the first step
The full figure that the default template of sign carries out described image using cascade classifier matches, and obtains matching result；
Third step determines hand morphological feature according to matching result described in second step, and exports the hand morphological feature.
5. gestural control method according to claim 1, which is characterized in that the extraction of hand gestures feature described in step 2
Journey includes：
Step a, the hand morphological feature is read as a result, and the hand morphological feature result is loaded into the default three-dimensional mould of hand
In type；
Step b, characteristic point is extracted using hand morphological feature result described in step a；
Step c, it is matched, is preset with the default threedimensional model according to characteristic point position described in step b and pixel value
Threedimensional model matching result；
Step d, hand coordinate system is established according to default threedimensional model matching result described in step c；
Step e, according to hand coordinate system described in step d, hand gestures parameter, the hand gestures ginseng are solved using PnP methods
Number is the hand gestures feature；
Step d, the characteristic results of hand gestures feature are exported.
6. gestural control method according to claim 1, which is characterized in that using hand gestures characteristic results to step 2 institute
It states hand position feature to extract, the extraction process of the hand position feature includes：
Step I, the hand gestures characteristic results are read
Step II, using hand gestures characteristic results described in step I, hand position is calculated by world coordinate system and hand coordinate system
Parameter is set, the hand position parameter is the hand position feature；
Step III, hand position characteristic results described in output step II.
7. gestural control method according to claim 1, which is characterized in that using hand morphological feature result to step 2 institute
It states hand position feature to extract, the extraction process of the hand position feature includes：
Step A, the hand morphological feature result is read；
Step B, using hand morphological feature described in step A as a result, calculating hand position by the hand centre of form and world coordinate system
Parameter, the hand position parameter are the hand position feature；
Step C, hand position characteristic results described in output step B.
8. gestural control method according to claim 1, which is characterized in that the extraction of hand motion feature described in step 2
Journey includes：
1st step, hand morphological feature result, hand gestures characteristic results and the hand position for reading preamble image and present image
Characteristic results；
2nd step carries out hand morphological feature result, hand gestures characteristic results described in the 1st step and hand position characteristic results
Variance analysis obtains hand motion after analysis；The hand motion is hand motion feature；
3rd step, output hand motion characteristic results；
Hand motion characteristic results described in 4th step, the 3rd step of storage, so as to the subsequent extracted invocation of procedure.
9. gestural control method according to claim 1, which is characterized in that using depth image feature extracting method to step
Two hand-characteristics extract, and the extraction process is：
Step1：The image data of input picture is read, default three-dimensional skeleton model is loaded into；
Step2：Divide the hand region in input described image using depth information；
Step3：Hand-characteristic point is extracted in hand region；
Step4：Model Matching is carried out using hand-characteristic point described in step3, obtains Model Matching result；
Step5：Establish hand coordinate system；
Step6：According to hand coordinate system extraction hand morphological feature, hand described in Model Matching result combination step5 described in step4
Portion's posture feature and hand position feature；
Step7：Hand exercise feature is calculated according to the difference of the front and back frame image of input picture.
10. a kind of gestural control system for realizing gestural control method described in claim 1, which is characterized in that the control system
System includes master controller, data acquisition module, data processing module, command output module and operational feedback module；The master control
The data acquisition signal control interaction end of device processed is connected with the data of data acquisition module control interaction end；The main control
The data processing signal control interaction end of device is connected with the data of data processing module control interaction end；The master controller
The data of instruction output signal control interaction end and described instruction output module control interaction end and be connected；The master controller
Operational feedback signal controls interaction end and is connected with the data of operational feedback module control interaction end；The data acquisition module
Data output end be connected with the data input pin of the data processing module；The data output end of the data processing module with
The data input pin of described instruction output module is connected.