Office Action 1: Notice of Allowance (2025-04-24)

CLAIMS:
WHAT IS CLAIMED IS:

1\. A method for improved hand-eye calibration based on a structured light camera, comprising:
step 1: establishing a pinhole camera model, and using a depth camera to detect a three- dimensional (3D) coordinate to obtain a physical coordinate of each point in an image coordinate system with a known depth relative to a camera coordinate system;
step 2: establishing a Denavit-Hartenberg (DH) model of a robotic arm, and moving the robotic arm to a determined coordinate using inverse kinematics;step 3: collecting n sets of point cloud data, applying depth scaling coefficients to the n sets of point cloud data to perform Singular Value Decomposition(SVD), and solving for an optimal depth scaling coefficient using a Nelder-Mead algorithm; wherein in the step 3, the robotic arm is driven using a given set of known joint angles, an end effector SVG {EMBED HERE} 0.1830.34 of the robotic arm is obtained using a kinematic model of the robotic arm, an axial difference Pt0a,of a checkerboard corner point at a specified position relative to the end effector of the robotic arm is manually measured, and SVG {EMBED HERE} 0.1870.337tol and Ptaol are combined to obtain PcaSe; the above operation is repeated to obtain a set of point cloud data relative to a base coordinate system of the robotic arm; the depth camera obtains a pixel coordinate of the checkerboard corner point at the specified position relative to the image coordinate system of the depth camera and a depth value of the checkerboard corner point relative to the camera coordinate system through corner detection, the pixel coordinate and the depth value are substituted into the pinhole camera model to obtain a 3D coordinate Pcaaera of the checkerboard corner point relative to the depth camera; the above operation is repeated to obtain a set of point clouds relative to the camera coordinate system, depth values of all point cloud data collected by the depth camera are processed through a first-order Kalman filter, and after the depth values measured by the depth camera are filtered and converged, complete point cloud data are calculated using an intrinsic parameter matrix of the depth camera;
in the step 3, the performing SVD includes: de-centering two sets of point cloud data, constructing a covariance matrix based on normalized two sets of point cloud data, decomposing the covariance matrix to obtain a matrix U, a matrix S, and a matrix V, and obtaining a calibration result by determining a rotation matrix R and a translation vector t based on a decomposition result;
analyzing and compensating for axial scaling of a depth of the depth camera; multiplying the depth value obtained by the depth camera by a scaling coefficient, such that the point cloud data of the checkerboard corner point is changed relative to the depth camera, multiplying the point cloud data collected by the depth camera and a mean value of the point cloud data by the scaling coefficient, and according to properties of the SVD, U and V remain unchanged, and S is scaled by a same factor as the covariance matrix:
\[U S \* z\_scale V\]= SVD(H')SVG {EMBED HERE} 0.3330.053RSVG {EMBED HERE} 0.371.043t'=Y in the step 3, the depth scaling coefficient, denoted as z\_scale, is optimized based on the calibration result, and an error function is defined using the scaling coefficient as an independent variable:e(zscale)=Paase(R \* SVG {EMBED HERE} 0.1770.463 + t') ;designating a coordinate of a calibration plate relative to the base coordinate system of the robotic arm as a true value, determining a calibration error by subtracting a calculation result of a measured coordinate of the depth camera and the calibration result from the true value, taking the independent variable of the error function as the depth scaling coefficient and combining the above equations to obtain an equation:e(zscale)=PAaseSVG {EMBED HERE} 0.1530.313\*pcalera\*SVG {EMBED HERE} 0.1872.147 determining a relationship between the error function and the depth scaling coefficient based on the combined equation, calculating a magnitude of an error, with an optimization objective being identifying a minimum value of the error; and using the Nelder-Mead algorithm to identify the minimum value and solving for the optimal depth scaling coefficient; and
step 4: completing the hand-eye calibration of the robotic arm based on the solved optimal depth scaling coefficient.

2\. The method for improved hand-eye calibration based on the structured light camera of claim 1, wherein the step 1 includes:
establishing a camera 3D coordinate system with a camera origin Oim as a coordinate origin to represent a 3D coordinate of a point in space, using a physical length as the unit;
establishing the image coordinate system with an image origin Opix as a coordinate origin to represent a pixel coordinate of a projection of the point in space on an imaging plane, using a pixel count as the unit; and
establishing an imaging coordinate system with an imaging origin Oxy as a coordinate origin to represent a two-dimensional (2D) coordinate of the projection of the point in space on the imaging plane, using the physical length as the unit.

3\. The method for improved hand-eye calibration based on the structured light camera of claim 2, wherein the camera 3D coordinate system and the imaging coordinate system are represented using the physical length as the unit, and the following equation is established basedSVG {EMBED HERE} 0.5231.043SVG {EMBED HERE} 0.5430.75
on the principle of similar triangles:SVG {EMBED HERE} 0.7170.19X = f SVG {EMBED HERE} 0.330.117= f SVG {EMBED HERE} 0.1670.107 wherein x and y collectively denote a coordinate of a spatial point P in the imaging coordinate system, and X, Y, Z collectively denote a coordinate of the spatial point P in the camera 3D coordinate system;in the image coordinate system and the imaging coordinate system, a projection of the spatial point P is converted using the following equation:u=ax+cxv=/yy + cSVG {EMBED HERE} 0.3330.053wherein u and v collectively denote a coordinate of the projection of the spatial point P in the image coordinate system, a and p denote a count of pixels corresponding to a unit of physical length in an x direction and a count of pixels corresponding to a unit of physical length in a y direction, respectively, and cx and cy collectively denote a coordinate of the imaging origin Oxy in the image coordinate system;combining the above two equations to obtain: SVG {EMBED HERE} 0.6131.053letaf=fx,SVG {EMBED HERE} 0.1530.073f=fy then the combined equation is written in a matrix multiplication form: 0 f, 0 wherein K denotes the intrinsic parameter matrix of the depth camera; and the above equation is used to determine the physical coordinate of the each point in the image coordinate system with the known depth relative to the camera coordinate system, given that the intrinsic parameter matrix of the depth camera is known.

4\. The method for improved hand-eye calibration based on the structured light camera of claim 3, wherein in the step 2, a transformation relationship between links is represented using the DH model; a motion relationship between neighboring links is expressed as rotation of a link i around a Z axis in a coordinate system n relative to a link i-1; the DH model includes four parameters: d, a, a, and 0, which are defined as follows d, denotes a link length, which is a distance between a joint m and a neighboring joint m-1 along the Z axis of the coordinate system n;a1SVG {EMBED HERE} 0.0730.107 denotes a link offset, which is a distance between the Z axis of the coordinate system n

and a Z axis of a coordinate system n-1 along an X axis of the coordinate system n-1;aci denotes a link twist angle, which is an angle between directions of the Z axes of the coordinate system n and the coordinate system n-1 along the X axis of the coordinate system n-1;
and 61 denotes a joint angle, which is an angle between a direction of an X axis of the coordinate system n and a direction of the X axis of the coordinate system n-1 along the Z axis of the coordinate system n.

5\. The method for improved hand-eye calibration based on the structured light camera of claim 4, wherein the transformation of neighboring links between coordinate systems is performed by:rotating the coordinate system n-1 around the X axis by SVG {EMBED HERE} 0.1170.337 translating the coordinate system n-1 along the X axis by SVG {EMBED HERE} 0.1170.33 rotating the coordinate system n-1 around the Z axis of the coordinate system n-1 by B; and translating the coordinate system n-1 around the Z axis of the coordinate system n-1 by di; and achieving complete overlap between the coordinate system n and the coordinate system n-1; wherein a homogeneous transformation matrix corresponding to a target pose is equal to a transformation matrix of the end effector of the robotic arm relative to the base coordinate system of the robotic arm; and
a set of equations is constructed, and the set of equations is solved to obtain a required joint angle to achieve the target pose.

OFFICE ACTION:
EXAMINER'S AMENDMENT
An examiner’s amendment to the record appears below. Should the changes and/or additions be unacceptable to applicant, an amendment may be filed as provided by 37 CFR 1.312. To ensure consideration of such an amendment, it MUST be submitted no later than the payment of the issue fee.
The application has been amended as follows:
Claim 1, lines 7-23:
step 3: collecting n sets of point cloud data, applying depth scaling coefficients to the n sets of point cloud data to perform Singular Value Decomposition (SVD), and solving for an optimal depth scaling coefficient using a Nelder-Mead algorithm; wherein in the step 3, the robotic arm is driven using a given set of known joint angles, an end effector position
P
b
a
s
e
t
o
o
l
of the robotic arm is obtained using a kinematic model of the robotic arm, an axial difference
P
t
o
o
l
c
a
l
of a checkerboard corner point at a specified position relative to the end effector of the robotic arm is manually measured, and
P
b
a
s
e
t
o
o
l
and
P
t
o
o
l
c
a
l
are combined to obtain
P
b
a
s
e
c
a
l
; the above operation is repeated to obtain a set of point cloud data relative to a base coordinate system of the robotic arm; the depth camera obtains a pixel coordinate of the checkerboard corner point at the specified position relative to the image coordinate system of the depth camera and a depth value of the checkerboard corner point relative to the camera coordinate system through corner detection, the pixel coordinate and the depth value are substituted into the pinhole camera model to obtain a 3D coordinate
P
c
a
m
e
r
a
c
a
l
of the checkerboard corner point relative to the depth camera; the above operation is repeated to obtain a set of point clouds relative to the camera coordinate system, depth values of all point cloud data collected by the depth camera are processed through a first-order Kalman filter, and after the depth values measured by the depth camera are filtered and converged, complete point cloud data are calculated using an intrinsic parameter matrix of the depth camera;
Claim 5, lines 4-8:
rotating the coordinate system n-1 around the X axis by
α
i
-
1
; translating the coordinate system n-1 along the X axis by
α
i
-
1
; rotating the coordinate system n-1 around the Z axis of the coordinate system n-1 by
θ
i
; and translating the coordinate system n-1 along the Z axis of the coordinate system n-1 by [[di]]
d
i
; and achieving complete overlap between the coordinate system n and the coordinate system n-1; wherein
Reasons for Allowance
The following is an examiner’s statement of reasons for allowance: the available prior art fails to teach or suggest using the claimed algorithms in combination to calibrate a 3D imaging system with a robotic manipulator.
Sivanath et al. (US Publication No. 2022/0402131) teaches calibrating a structured-light based 3D machine-vision system for use with a robotic manipulator using Singular Value Decomposition (SVD) at paragraphs [0039-0041], but fails to teach utilizing the Nelder-Mead algorithm to solve for an optimal depth scaling coefficient.
Kubota et al. (US Patent No. 5,280,436) teaches, at column 5, line 26 through column 6, line 46, depth scaling for such structured light systems, but fails to teach the specific algorithms and methods detailed in independent claim 1, such as using the Nelder-Mead algorithm.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee.  Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
Conclusion



CITED REFERENCE SUMMARIES: Ref US20220402131A1: Claims:
A robotic system, the system comprising: a machine-vision module; a robotic arm comprising an end-effector; a robotic controller configured to control movements of the robotic arm; and an error-compensation module configured to compensate for pose errors of the robotic arm by determining a controller-desired pose corresponding to a camera-instructed pose of the end-effector such that, when the robotic controller controls the movements of the robotic arm based on the controller-desired pose, the end-effector achieves, as observed by the machine-vision module, the camera-instructed pose; wherein the error-compensation module comprises a machine learning model configured to output an error matrix that correlates the camera-instructed pose to the controller-desired pose.
a machine-vision module;
a robotic arm comprising an end-effector;
a robotic controller configured to control movements of the robotic arm; and
an error-compensation module configured to compensate for pose errors of the robotic arm by determining a controller-desired pose corresponding to a camera-instructed pose of the end-effector such that, when the robotic controller controls the movements of the robotic arm based on the controller-desired pose, the end-effector achieves, as observed by the machine-vision module, the camera-instructed pose;
wherein the error-compensation module comprises a machine learning model configured to output an error matrix that correlates the camera-instructed pose to the controller-desired pose.
The robotic system of claim 1 , wherein the machine learning model comprises a neural network.
The robotic system of claim 2 , wherein the neural network comprises an embedding layer and a processing layer, and wherein each of the embedding and processing layers comprises a multi-layered perceptron.
The robotic system of claim 3 , wherein the embedding layer is configured to embed separate translational components and rotational components of a pose.
The robotic system of claim 3 , wherein the embedding layer uses Rectified Linear Unit (ReLU) as an activation function, and wherein the processing layer uses leaky ReLU as an activation function.
The robotic system of claim 2 , further comprising a model-training module configured to train the neural network by collecting training samples, wherein while training the neural network, the model-training module is configured to: cause the robotic controller to generate a controller-desired pose sample; control movements of the robotic arm based on the controller-desired pose sample; determine, using a machine-vision module, an actual pose of the end-effector; and compute an error matrix based on the controller-desired pose sample and the actual pose.
cause the robotic controller to generate a controller-desired pose sample;
control movements of the robotic arm based on the controller-desired pose sample;
determine, using a machine-vision module, an actual pose of the end-effector; and
compute an error matrix based on the controller-desired pose sample and the actual pose.
The robotic system of claim 6 , wherein the model-training module is configured to train the neural network until the error matrix produced by the machine learning model reaches a predetermined level of accuracy.
The robotic system of claim 1 , further comprising a coordinate-transformation module configured to transform a pose determined by the machine-vision module from a camera-centered coordinate system to a robot-centered coordinate system.
The robotic system of claim 8 , wherein the coordinate-transformation module is further configured to determine a transformation matrix based on a predetermined number of measured poses of a calibration target.
The robotic system of claim 8 , wherein the coordinate-transformation module is further configured to correlate a pose of a component held by the end-effector to a corresponding pose of the end-effector.
A computer-implemented method, the method comprising: determining, by a machine-vision module, a camera-instructed pose of an end-effector of a robotic arm for accomplishing an assembly task; determining a controller-desired pose corresponding to the camera-instructed pose of the end-effector, which comprises applying a machine learning model to obtain an error matrix that correlates the camera-instructed pose to the controller-desired pose; and controlling, by a robotic controller, movements of the robotic arm based on the controller-desired pose, thereby facilitating the end-effector to achieve the camera-instructed pose in order to accomplish the assembly task.
determining, by a machine-vision module, a camera-instructed pose of an end-effector of a robotic arm for accomplishing an assembly task;
determining a controller-desired pose corresponding to the camera-instructed pose of the end-effector, which comprises applying a machine learning model to obtain an error matrix that correlates the camera-instructed pose to the controller-desired pose; and
controlling, by a robotic controller, movements of the robotic arm based on the controller-desired pose, thereby facilitating the end-effector to achieve the camera-instructed pose in order to accomplish the assembly task.
The method of claim 11 , wherein the machine learning model comprises a neural network, wherein the neural network comprises an embedding layer and a processing layer, and wherein each of the embedding and processing layers comprises a multi-layered perceptron.
The method of claim 12 , wherein applying the machine learning model comprises embedding, by the embedding layer, translational components and rotational components of a pose separately.
The method of claim 12 , wherein applying the machine learning model further comprises: implementing, at the embedding layer, Rectified Linear Unit (ReLU) as an activation function, and implementing, at the processing layer, leaky ReLU as an activation function.
implementing, at the embedding layer, Rectified Linear Unit (ReLU) as an activation function, and
implementing, at the processing layer, leaky ReLU as an activation function.
The method of claim 12 , further comprising training the neural network by collecting training samples, wherein collecting a respective training sample comprises: causing the robotic controller to generate a controller-desired pose sample; controlling movements of the robotic arm based on the controller-desired pose sample; determining, using a machine-vision module, an actual pose of the end-effector; and computing an error matrix based on the controller-desired pose sample and the actual pose.
causing the robotic controller to generate a controller-desired pose sample;
controlling movements of the robotic arm based on the controller-desired pose sample;
determining, using a machine-vision module, an actual pose of the end-effector; and
computing an error matrix based on the controller-desired pose sample and the actual pose.
The method of claim 11 , further comprising determining a transformation matrix used for transforming a pose determined by a machine-vision module from a camera-centered coordinate system to a robot-centered coordinate system.
The method of claim 11 , further comprising determining a component-transformation matrix used for transforming a pose of a component held by the end-effector to a corresponding pose of the end-effector.
A computer-implemented method, the method comprising: modeling pose errors associated with an end-effector of a robotic arm using a neural network; training the neural network using a plurality of training samples, wherein a respective training sample comprises a camera-instructed pose of the end-effector and a corresponding error matrix that correlates the camera-instructed pose to a controller-desired pose of the end-effector; and applying the trained neural network to compensate for the pose errors during operations of the robotic arm.
modeling pose errors associated with an end-effector of a robotic arm using a neural network;
training the neural network using a plurality of training samples, wherein a respective training sample comprises a camera-instructed pose of the end-effector and a corresponding error matrix that correlates the camera-instructed pose to a controller-desired pose of the end-effector; and
applying the trained neural network to compensate for the pose errors during operations of the robotic arm.
The computer-implemented method of claim 18 , wherein the neural network comprises an embedding layer and a processing layer, and wherein each of the embedding and processing layers comprises a multi-layered perceptron.
The computer-implemented method of claim 19 , wherein modeling the pose errors comprises embedding, by the embedding layer, translational components and rotational components of a pose separately.
The computer-implemented method of claim 19 , wherein modeling the pose errors comprises: implementing, at the embedding layer, Rectified Linear Unit (ReLU) as an activation function, and implementing, at the processing layer, leaky ReLU as an activation function.
The computer-implemented method of claim 18 , further comprising collecting the training samples, wherein collecting a respective training sample comprises: causing the robotic controller to generate a controller-desired pose sample; controlling movements of the robotic arm based on the controller-desired pose sample; determining, using a machine-vision module, an actual pose of the end-effector; and computing an error matrix based on the controller-desired pose sample and the actual pose.
Ref US5280436A: Claims:
A method for measuring a three-dimensional position of an object to be captured comprising providing a laser measuring unit having an image recognition camera confronting the object and a first and a second laser emitting light source, emitting from the first laser emitting light source laser light for irradiating a lateral slit light onto the object from a slant direction relative to a vertical axis of a vision coordinate system, and emitting from the second laser emitting light source further laser light for irradiating a vertical slit light from a slant direction relative to a lateral axis of the vision coordinate system, one slit image on the vertical axis of the object and one slit image on the lateral axis thereof and one of the slit images on the vertical and lateral axes thereof formed before and after the laser measuring unit is moved a predetermined amount in a direction of a camera optical axis relative to the object are taken into the camera, and calculating the three-dimensional position of the object from the relationship between a predetermined moving amount of the laser measuring unit and deviation of the slit images on vision coordinates of the coordinate system.
A method for measuring a three-dimensional position of an electronic component to be captured by an electric adjustment member at an electric adjustment position of the electronic component, comprising the steps of: providing in a control of a servo-driven industrial robot an initial position of an electric adjustment member mounted on a hand portion of the industrial robot; calculating, as an electric adjustment position for the electronic component, the three-dimensional position of the electronic component to be captured by said electric adjustment member and mounted on a workpiece, said calculating step being constituted by providing a laser measuring unit having an image recognition camera confronting the electronic component and a first and second laser emitting light source, emitting from the first laser emitting light source laser light for irradiating a lateral slit light onto the electronic component from a slant direction relative to a vertical axis of a vision coordinate system, and emitting from the second laser emitting light source further laser light for irradiating a vertical slit light onto the electronic component from a slant direction relative to a lateral axis of the vision coordinate system, one slit image on the vertical axis of the electronic component and one slit image on the lateral axis thereof and one of the slit images on the vertical and lateral axes thereof formed before and after the laser measuring unit is moved a predetermined amount in a direction of a camera optical axis relative to the object are taken into the camera, and calculating the three-dimensional position of the electronic component from the relationship between a pre-taught moving distance of the laser measuring unit and deviation of the slit images on vision coordinates of the coordinate system; compensating the initial position of the electric adjustment member on the hand portion of the industrial robot by the amount of deviation of the electric adjustment position of the electronic component from the initial position so as to move the electric adjustment member to the electric adjustment position of the electronic component.
providing in a control of a servo-driven industrial robot an initial position of an electric adjustment member mounted on a hand portion of the industrial robot;
calculating, as an electric adjustment position for the electronic component, the three-dimensional position of the electronic component to be captured by said electric adjustment member and mounted on a workpiece, said calculating step being constituted by providing a laser measuring unit having an image recognition camera confronting the electronic component and a first and second laser emitting light source, emitting from the first laser emitting light source laser light for irradiating a lateral slit light onto the electronic component from a slant direction relative to a vertical axis of a vision coordinate system, and emitting from the second laser emitting light source further laser light for irradiating a vertical slit light onto the electronic component from a slant direction relative to a lateral axis of the vision coordinate system, one slit image on the vertical axis of the electronic component and one slit image on the lateral axis thereof and one of the slit images on the vertical and lateral axes thereof formed before and after the laser measuring unit is moved a predetermined amount in a direction of a camera optical axis relative to the object are taken into the camera, and calculating the three-dimensional position of the electronic component from the relationship between a pre-taught moving distance of the laser measuring unit and deviation of the slit images on vision coordinates of the coordinate system;
compensating the initial position of the electric adjustment member on the hand portion of the industrial robot by the amount of deviation of the electric adjustment position of the electronic component from the initial position so as to move the electric adjustment member to the electric adjustment position of the electronic component.
The method as claimed in claim 2 further comprising the steps of: providing at least two claws with tapered portions and movable between closed and open states for capturing an electronic component by holding an end face of the electronic component onto the electric adjustment member mounted on the hand portion of the industrial robot; moving the electric adjustment member according to the deviation of the electric adjustment position of the electronic component to the electric adjustment position of the electronic component with the claws normally closed; causing the claws to be opened as they are moved over end edges of the electronic component after contact of the tapered portions therewith and then to be closed at a final capturing position after the tapered portions have passed the end edges; detecting the closed state and open state of the capturing claws as the tapered portions thereof are moved over end edges of the electronic component; stopping movement of the industrial robot when all of the capturing claws have been detected as having been shifted from the closed state through the open state to the closed state again; and determining the position of the industrial robot when all of the capturing claws have reached the closed state again as the captured position of the electronic component.
providing at least two claws with tapered portions and movable between closed and open states for capturing an electronic component by holding an end face of the electronic component onto the electric adjustment member mounted on the hand portion of the industrial robot;
moving the electric adjustment member according to the deviation of the electric adjustment position of the electronic component to the electric adjustment position of the electronic component with the claws normally closed;
causing the claws to be opened as they are moved over end edges of the electronic component after contact of the tapered portions therewith and then to be closed at a final capturing position after the tapered portions have passed the end edges;
detecting the closed state and open state of the capturing claws as the tapered portions thereof are moved over end edges of the electronic component;
stopping movement of the industrial robot when all of the capturing claws have been detected as having been shifted from the closed state through the open state to the closed state again; and
determining the position of the industrial robot when all of the capturing claws have reached the closed state again as the captured position of the electronic component.
The method as claimed in claim 3 further comprising the steps of: cutting off a servo power source for only the axis of the industrial robot perpendicular to the end face of the electronic component when the industrial robot is determined to be at the captured position so as to release stress on the electronic adjustment member and the hand portion of the industrial robot at the captured position so that the industrial robot is in a state free from servo control as to said axis; and determining the position of the industrial robot in the servo-free state as the final captured position of the electronic component, and servo-locking the servo power source for said axis of the industrial robot.
cutting off a servo power source for only the axis of the industrial robot perpendicular to the end face of the electronic component when the industrial robot is determined to be at the captured position so as to release stress on the electronic adjustment member and the hand portion of the industrial robot at the captured position so that the industrial robot is in a state free from servo control as to said axis; and
determining the position of the industrial robot in the servo-free state as the final captured position of the electronic component, and servo-locking the servo power source for said axis of the industrial robot.
Ref US5280436: Claims:
1. A method for measuring a three-dimensional position of an object to be captured comprising providing a laser measuring unit having an image recognition camera confronting the object and a first and a second laser emitting light source, emitting from the first laser emitting light source laser light for irradiating a lateral slit light onto the object from a slant direction relative to a vertical axis of a vision coordinate system, and emitting from the second laser emitting light source further laser light for irradiating a vertical slit light from a slant direction relative to a lateral axis of the vision coordinate system, one slit image on the vertical axis of the object and one slit image on the lateral axis thereof and one of the slit images on the vertical and lateral axes thereof formed before and after the laser measuring unit is moved a predetermined amount in a direction of a camera optical axis relative to the object are taken into the camera, and calculating the three-dimensional position of the object from the relationship between a predetermined moving amount of the laser meas
[Truncated to input limit]