Office Action 1: Notice of Allowance (2025-02-26)

CLAIMS:
What is claimed is:

1\. A system of automatic target image acquisition and calibration, comprising:a first robotic arm, being disposed near a first conveyor and a second conveyor, wherein the first conveyor transfers a plurality of articles, and at least one surface of said article being provided with a recognition structure thereon;
a first camera, being disposed near the first robotic arm;
at least one second robotic arm, being disposed near the second conveyor, and
grasping at least one second camera; andan electronic device, being coupled to the first robotic arm, the first camera,
the at least one second robotic arm, and the at least one second camera, and
comprising a processor and a memory;
wherein the memory stores an application program, and the processor accesses the memory so as to execute the application program;
wherein when the application program is executed, the processor is configured to:
control the first robotic arm to grasp one article from the first conveyor, move the article to be in a first photographing region of the first camera, and then turn over the article in a rotation sequence;
control the first camera to acquire a plurality of 2D images from the article during the article being turned over;
apply a feature recognition process to the plurality of 2D images, so as to find at least one 2D image including a recognition feature pattern;
control the first robotic arm to move the article to the second conveyor, and determine a relative position and a relative 3D coordinate of the article based on the recognition feature pattern and the plurality of 2D images;
conduct a camera moving operation, so as to control the robotic arm to move the second camera to make one of N inspection surfaces of the article be in a second photographing region of the second camera, where N is an integer;
conduct a photographing operation, so as to control the second camera to
photograph the inspection surface of the article, thereby obtaining one article image;repeat the foregoing camera moving operation and the foregoing photographing operation N-1 times, so as to obtain a total of N article images;
apply a feature extracting process to the N article images so as to generate N feature images; and
conduct a feature matching operation, so as to determine whether said feature image contains at least one defect feature that matches at least one of a plurality of reference defect features.

2\. The system of claim 1, further comprising:a first light source, being coupled to the electronic device, and being controlled by the electronic device so as to radiate a first illumination light to the first photographing region; and
a second light source, being coupled to the electronic device, and being controlled by the electronic device so as to radiate a second illumination light to
the second photographing region.

3\. The system of claim 1, wherein the recognition structure is a tag selected from the group consisting of RFID tag, 2D barcode tag, QR code tag, matrix barcode tag, patterns-containing tag, text-containing tag, alphabets-containing tags, and digits- containing tag.

4\. The system of claim 1, wherein the recognition structure is a structural feature of the article.

5\. The system of claim 3, wherein the tag is adopted for storing an article information comprising vendor ID, product ID, serial number, and batch/lot number.

6\. The system of claim 1, wherein the application program comprises instructions, andthe processor executes the instructions so as to be configured to establish a 3D image of the article based on the recognition feature pattern and the plurality of 2D
images, and then to determine the relative position and the relative 3D coordinate of the article.

7\. rhe system of claim 6, wherein the instructions include a mathematical algorithm selected from the group consisting of Delaunay triangulation algorithm, Crust triangulation algorithm, and triangulated irregular network (TIN) algorithm.

8\. The system of claim 1, wherein the application program includes a plurality of subprograms, the plurality of subprograms comprising:a first subprogram, being compiled to be integrated in the application program
by one type of programming language, and including instructions for configuring the processor to control the first robotic arm to grasp one article from the first conveyor, move the article to be in the first photographing region of the first camera,
and turn over the article in said rotation sequence, and configuring the processor to control the first robotic arm to move the article to the second conveyor;a second subprogram, being compiled to be integrated in the application program by one type of programming language, and including instructions for 15 configuring the processor to control the first camera to acquire the plurality of 2D images from the article during the article being turned over;
a third subprogram, being compiled to be integrated in the application program by one type of programming language, and including instructions for configuring
the processor to apply the feature recognition process to the plurality of 2D images,
so as to find at least one 2D image including the recognition feature pattern;a fourth subprogram, being compiled to be integrated in the application program by one type of programming language, and including instructions for configuring the processor to establish a 3D image of the article based on the
recognition feature pattern and the plurality of 2D images, and then to determine the relative position and the relative 3D coordinate of the article;a fifth subprogram, being compiled to be integrated in the application program by one type of programming language, and including instructions for configuring the processor to control the robotic arm to move the second camera, thereby making
one of N inspection surfaces of the article be in the second photographing region of the second camera;a six subprogram, being compiled to be integrated in the application program by one type of programming language, and including instructions for configuring the processor to control the second camera to photograph the inspection surface of
the article, thereby obtaining one article image; anda seventh subprogram, being compiled to be integrated in the application program by one type of programming language, and including instructions for configuring the processor to apply the feature extracting process to the N article images, and then to determine whether said feature image contains at least one
defect feature that matches at least one of a plurality of reference defect features.

9\. The system of claim 1, wherein the electronic device is further coupled to an apparatus controlling device that is used for controlling the first conveyor and the second conveyor.

10\. The system of claim 1, wherein the electronic device is integrated in an apparatus controlling device that is used for controlling the first conveyor and the second conveyor.5 10 15 20 25

OFFICE ACTION:
DETAILED ACTION
1.	This Office Action is sent in response to Applicant’s communication received on 08/17/2023 for application number 18/451,264. The Office herby acknowledges receipt of the following and placed of record in file: Specification, Drawings, Abstract, Oath/Declaration, and claims.
Notice of Pre-AIA  or AIA  Status
2.	The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
3.	Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed.
REASONS FOR ALLOWANCE
4.	The following is an examiner's statement of reasons for allowance: The instant invention is related to technology field of machine vision, and more particularly to an automatic target image acquisition and calibration system for application in a defect inspection system.
Prior art was found for the claims as follows:
Stubbs et al., [US Pub. No.: 2017/0080566 A1] discloses a robotic manipulators may be used to manipulate objects. Manipulation data about manipulations performed on objects may be generated and accessed. This data may be analyzed to generate a profile indicating how an object may be manipulated. A portion of the profile may be transmitted to a particular robotic manipulator. For example, the portion may be based on a manipulation capability of the robotic manipulator. In turn, the robotic manipulator may use the portion of the profile to manipulate the object.
5.	Applicant uniquely claimed a distinct feature in the instant invention, which are not found in the prior art, either singularly or in combination. The claims are allowable because the prior art fails to disclose the following claim features “…conduct a camera moving operation, so as to control the robotic arm to move the second camera to make one of N inspection surfaces of the article be in a second photographing region of the second camera, where N is an integer; conduct a photographing operation, so as to control the second camera to photograph the inspection surface of the article, thereby obtaining one article image; repeat the foregoing camera moving operation and the foregoing photographing operation N-1 times, so as to obtain a total of N article images; apply a feature extracting process to the N article images so as to generate N feature images; and conduct a feature matching operation, so as to determine whether said feature image contains at least one defect feature that matches at least one of a plurality of reference defect features”.
This feature is not found or suggested in the prior art.
6.	Any comments considered necessary by applicant must be submitted no later
than the payment of the issue fee and, to avoid processing delays, should preferably
accompany the issue fee. Such submissions should be clearly labeled “Comments on
Statement of reasons for Allowance.”
7.	Claims 1-10 are allowed.
Conclusion



CITED REFERENCE SUMMARIES: Ref US20170080566A1: Claims:
An inventory system, comprising: a central station comprising: one or more processors, and one or more computer-readable media comprising instructions; a first robotic manipulator in communication with the central station over a data network, the first robotic manipulator configured to perform a first action and a second action associated with manipulating an inventory item; and a second robotic manipulator in communication with the central station over the data network, the second robotic manipulator configured to perform the second action and a third action, the third action different from the first action and associated with manipulating the inventory item, wherein, when the instructions are executed with the one or more processors, the instructions cause the central station to at least: receive, from the first robotic manipulator over the data network, first data about performing the first action and the second action to manipulate the inventory item; generate a manipulation profile based at least in part on the first data, the manipulation profile comprising instructions about manipulating the item in association with performing the first action and the second action; transmit a portion of the manipulation profile to the second robotic manipulator over the data network, the portion comprising a subset of the instructions associated with the second action; receive, from the second robotic manipulator over the data network, second data about performing the second action and the third action to manipulate the inventory item; and update the manipulation profile based at least in part on the second data, the update comprising updating the subset of the instructions associated with the second action and generating additional instructions about manipulating the item in association with performing the third action.
a central station comprising: one or more processors, and one or more computer-readable media comprising instructions;
one or more processors, and
one or more computer-readable media comprising instructions;
a first robotic manipulator in communication with the central station over a data network, the first robotic manipulator configured to perform a first action and a second action associated with manipulating an inventory item; and
a second robotic manipulator in communication with the central station over the data network, the second robotic manipulator configured to perform the second action and a third action, the third action different from the first action and associated with manipulating the inventory item,
wherein, when the instructions are executed with the one or more processors, the instructions cause the central station to at least: receive, from the first robotic manipulator over the data network, first data about performing the first action and the second action to manipulate the inventory item; generate a manipulation profile based at least in part on the first data, the manipulation profile comprising instructions about manipulating the item in association with performing the first action and the second action; transmit a portion of the manipulation profile to the second robotic manipulator over the data network, the portion comprising a subset of the instructions associated with the second action; receive, from the second robotic manipulator over the data network, second data about performing the second action and the third action to manipulate the inventory item; and update the manipulation profile based at least in part on the second data, the update comprising updating the subset of the instructions associated with the second action and generating additional instructions about manipulating the item in association with performing the third action.
receive, from the first robotic manipulator over the data network, first data about performing the first action and the second action to manipulate the inventory item;
generate a manipulation profile based at least in part on the first data, the manipulation profile comprising instructions about manipulating the item in association with performing the first action and the second action;
transmit a portion of the manipulation profile to the second robotic manipulator over the data network, the portion comprising a subset of the instructions associated with the second action;
receive, from the second robotic manipulator over the data network, second data about performing the second action and the third action to manipulate the inventory item; and
update the manipulation profile based at least in part on the second data, the update comprising updating the subset of the instructions associated with the second action and generating additional instructions about manipulating the item in association with performing the third action.
The inventory system of claim 1 , wherein transmitting the portion of the manipulation profile comprises: accessing a profile of the second robotic manipulator; generating a customized manipulation profile from the manipulation profile based at least in part on the profile of the second robotic manipulator; and transmitting the customized manipulation profile to the second robotic manipulator.
accessing a profile of the second robotic manipulator;
generating a customized manipulation profile from the manipulation profile based at least in part on the profile of the second robotic manipulator; and
transmitting the customized manipulation profile to the second robotic manipulator.
The inventory system of claim 1 , wherein the first robotic manipulator is configured to generate the first data about manipulating the inventory item based at least in part on one or more of: a multi-dimensional model generated by the first robotic manipulator and specifying manipulations of the item, a grasp and a position of the grasp generated by the first robotic manipulator to manipulate the item, or a feature of the item identified by the first robotic manipulator to manipulate the item.
The inventory system of claim 1 , wherein the second data about manipulating the item comprises a failure to manipulate the item based at least in part on the subset of the instructions, and wherein updating the manipulation profile comprises updating the subset of the instructions based at least in part on the failure.
A computer-implemented method, comprising: accessing, by a computer system, manipulation data of a plurality of robotic manipulators about manipulations of an object type; generating, by the computer system, a profile associated with manipulating the object type based at least in part on the manipulation data; selecting, by the computer system, a portion of the profile based at least in part on a robotic manipulator of the plurality of robotic manipulators; and transmitting, by the computer system, the portion of the profile to the robotic manipulator.
accessing, by a computer system, manipulation data of a plurality of robotic manipulators about manipulations of an object type;
generating, by the computer system, a profile associated with manipulating the object type based at least in part on the manipulation data;
selecting, by the computer system, a portion of the profile based at least in part on a robotic manipulator of the plurality of robotic manipulators; and
transmitting, by the computer system, the portion of the profile to the robotic manipulator.
The computer-implemented method of claim 5 , wherein the object type comprises a type of items, wherein the computer system is associated with a central station of an inventory of the items, wherein the items are offered from an electronic marketplace, and wherein the portion of the profile is transmitted over a network to the robotic manipulator and comprises instructions about manipulating the type of the items based at least in part on actions supported by the robotic manipulator.
The computer-implemented method of claim 5 , wherein the manipulation data comprises one or more of: a multi-dimensional model associated with the object type, grasps and positions of the grasps associated with manipulating the object type, or features of the object type, and wherein the manipulation data identifies successes and failures of manipulating the object type based at least in part on the one or more of: the multi-dimensional model, the grasps and the positions, or the features.
The computer-implemented method of claim 7 , wherein the one or more of: the multi-dimensional model, the grasps and the positions, or the features are generated based at least in part on data received from one or more of: optical sensors, force sensors, pressure sensors, weight sensors, or touch sensors of the plurality of robotic manipulators.
The computer-implemented method of claim 5 , wherein the profile comprises one or more of: instructions about grasping, moving, and releasing an object of the object type, features of the object type to orient the object, a multi-dimensional model of the object type to manipulate the object, information about an end effector to use in association with manipulating the object, a force to apply by the end effector, a sequence of actions to apply by the end effector, or instructions about manipulating a bundle of two or more objects of the object type.
The computer-implemented method of claim 5 , wherein selecting the portion of the profile comprises: identifying, by the computer system, a capability of the robotic manipulator; and customizing the profile to generate the portion of the profile based at least in part on the capability.
identifying, by the computer system, a capability of the robotic manipulator; and
customizing the profile to generate the portion of the profile based at least in part on the capability.
The computer-implemented method of claim 5 , wherein selecting the portion of the profile comprises: identifying, by the computer system, an action of the robotic manipulator to be performed on an object of the object type; and selecting, by the computer system, a subset of instructions about manipulating the object type from the profile based at least in part on the action.
identifying, by the computer system, an action of the robotic manipulator to be performed on an object of the object type; and
selecting, by the computer system, a subset of instructions about manipulating the object type from the profile based at least in part on the action.
The computer-implemented method of claim 5 , wherein the robotic manipulator comprises a robotic arm and an end effector configured to manipulate objects of the object type, and wherein the portion of the profile is selected based at least in part on one or more of the robotic arm or the end effector.
A system, comprising: one or more processors; and one or more computer-readable media comprising instructions that, when executed with the one or more processors, cause the system to at least: access first manipulation data of a first robotic manipulator about manipulations of an object type; generate a profile associated with manipulating the object type based at least in part on the first manipulation data; access second manipulation data of a second robotic manipulator about the manipulations of the object type; update the profile based at least in part on the second manipulation data; and transmit a portion of the profile to one or more of the first robotic manipulator or the second robotic manipulator based at least in part on the update.
one or more processors; and
one or more computer-readable media comprising instructions that, when executed with the one or more processors, cause the system to at least: access first manipulation data of a first robotic manipulator about manipulations of an object type; generate a profile associated with manipulating the object type based at least in part on the first manipulation data; access second manipulation data of a second robotic manipulator about the manipulations of the object type; update the profile based at least in part on the second manipulation data; and transmit a portion of the profile to one or more of the first robotic manipulator or the second robotic manipulator based at least in part on the update.
access first manipulation data of a first robotic manipulator about manipulations of an object type;
generate a profile associated with manipulating the object type based at least in part on the first manipulation data;
access second manipulation data of a second robotic manipulator about the manipulations of the object type;
update the profile based at least in part on the second manipulation data; and
transmit a portion of the profile to one or more of the first robotic manipulator or the second robotic manipulator based at least in part on the update.
The system of claim 13 , wherein the first robotic manipulator is configured to generate a manipulation model for the object type, wherein the profile comprises the manipulation model, and wherein the manipulation model is transmitted to the second robotic manipulator in the portion of the profile.
The system of claim 13 , wherein the first manipulation data comprises images of an object of the object type, and wherein generating the profile comprises generating a multi-dimensional model of the object type based at least in part on the images.
The system of claim 13 , wherein the first manipulation data comprises a multi-dimensional model of the object type based at least in part on images from the first robotic manipulator of an object of the object type, and wherein the profile is generated based at least in part on the multi-dimensional model.
The system of claim 13 , wherein the first manipulation data indicates a manipulation of the object type and a success or a failure of the manipulation, wherein the first robotic manipulator is configured to generate the first manipulation data based at least in part on multiple performances of the manipulation.
The system of claim 13 , wherein the object type is associated with an attribute that is common to objects of the object type and that impacts the manipulations of the object type, wherein, when the instructions are executed with the one or more processors, the instructions further cause the system to at least: receive, from the first robotic manipulator or the second robotic manipulator, a change to the attribute; and update the profile based at least in part on the change to the attribute.
receive, from the first robotic manipulator or the second robotic manipulator, a change to the attribute; and
update the profile based at least in part on the change to the attribute.
The system of claim 13 , wherein the object type is associated with an attribute that is common to objects of the object type and that impacts the manipulations of the object type, wherein, when the instructions are executed with the one or more processors, the instructions further cause the system to at least: receive, from the first robotic manipulator or the second robotic manipulator, a change to the attribute; request, from the first robotic manipulator or the second robotic manipulator, an indication whether the attribute has changed for the objects of the object type; receive, from the first robotic manipulator or the second robotic manipulator based at least in part on the request, information that a first set of the objects comprises the attribute unchanged and that a second set of the objects comprises the attribute changed; and generate a second profile associated with manipulating the second set of the objects based at least in part on the change to the attribute.
receive, from the first robotic manipulator or the second robotic manipulator, a change to the attribute;
request, from the first robotic manipulator or the second robotic manipulator, an indication whether the attribute has changed for the objects of the object type;
receive, from the first robotic manipulator or the second robotic manipulator based at least in part on the request, information that a first set of the objects comprises the attribute unchanged and that a second set of the objects comprises the attribute changed; and
generate a second profile associated with manipulating the second set of the objects based at least in part on the change to the attribute.
The system of claim 13 , wherein the portion of the profile is transmitted to the second robotic manipulator and comprises information about an action to manipulate the object type, wherein, when the instructions are executed with the one or more processors, the instructions further cause the system to at least: receive, from the second robotic manipulator, an indication of a failure of the action at the second robotic manipulator; send a request to the first robotic manipulator to confirm the failure of the action at the first robotic manipulator; and update the profile based at least in part on a confirmation of the failure from the first robotic manipulator.
receive, from the second robotic manipulator, an indication of a failure of the action at the second robotic manipulator;
send a request to the first robotic manipulator to confirm the failure of the action at the first robotic manipulator; and
update the profile based at least in part on a confirmation of the failure from the first robotic manipulator.
Ref US20110298939A1: Claims:
A method for generating a signal based on a visual image, the method comprising: photographing a target object with a digital camera to obtain a target image; receiving the target image into a processor that is in communication with the camera; cross-correlating the target image with a structure having a variety of scales across the target image; based on cross-correlating the target image, generating a signal for output on a device associated with the camera.
photographing a target object with a digital camera to obtain a target image;
receiving the target image into a processor that is in communication with the camera;
cross-correlating the target image with a structure having a variety of scales across the target image;
based on cross-correlating the target image, generating a signal for output on a device associated with the camera.
The method of claim 1 , further comprising: extracting icons from a coordinate generated by cross-correlating the target image; normalizing each icon to a predetermined constant size; comparing at least one normalized icon to stored icons associated with learned images to determine a match.
extracting icons from a coordinate generated by cross-correlating the target image;
normalizing each icon to a predetermined constant size;
comparing at least one normalized icon to stored icons associated with learned images to determine a match.
The method of claim 2 , wherein signal includes an audible signal indicating the learned image that matches the target image.
The method of claim 2 , wherein the camera includes a lens having a predetermined focal length and a sensor having a predetermined size, the method further comprising: computing a distance between the target object and the camera based on comparing the at least one normalized icon to at least one stored icon from a learned image that matches the target image and based on the focal length and size of the sensor, wherein the signal includes an audible signal indicating the distance between the target object and the camera or a direction that the camera is located with respect to the target object.
computing a distance between the target object and the camera based on comparing the at least one normalized icon to at least one stored icon from a learned image that matches the target image and based on the focal length and size of the sensor, wherein the signal includes an audible signal indicating the distance between the target object and the camera or a direction that the camera is located with respect to the target object.
The method of claim 1 , further comprising: identifying RGB data for at least one location within the
[Truncated to input limit]