Automatically identifying locations to apply sealant and applying sealant to a target object

Aspects of the disclosure are directed towards generating a trajectory for use by a robot to perform an operation on a target object. A method includes a robot instructed to traverse a surface of the target object from the starting position and obtain sensor input from a sensor system of the robot using a first trajectory specified by a CAD file for the target object. Sensor input that includes a first height to the target object, a width of the seam, and a depth of the seam may be received. A second trajectory may be generated for traversing the surface of the target object and that modifies the first trajectory based on the sensor input. The robot may be instructed to move to the starting position and traverse the surface of the target object using the second trajectory and apply a sealant to the seam.

BACKGROUND

Industrial robotics is an expanding field for various industries that want to improve their internal and customer-facing processes. Industrial robots can be fabricated and programmed to perform various tasks for different applications. This customizability has led many enterprises to expand the incorporation of robots from manufacturing to other processes to improve the safety and the efficiency of the enterprise's workers.

DETAILED DESCRIPTION

Robots can be deployed to perform different operations on targets in an operational environment. For example, robots can be deployed in a facility to perform manufacturing and/or maintenance operations on parts or target objects, such as an airplane part. At least some of the operations can be autonomously performed, whereby a robot can move to a surface holding a target object and then perform operations on the target object. Thereafter, the robot can perform error checking or other operations on the target object before moving to another target object or having a target object being brought to the robot. To be able to do so, the robot can register or localize the target object in a coordinate system of the robot. Upon registering the target object in a coordinate system of the robot, the robot can identify the target object using markers or features of the target object and determine a computer aided design (CAD) file for the target object. Among other things, the CAD file for the target object can include a starting position to move the robot to and a trajectory to traverse the target object. Trajectory planning can be performed by using the robot using the coordinate system of the robot and the localization of the target object in the coordinate system of the robot, whereby the robot can position itself or the target object at such an orientation or angle such that the operations can be performed on the target object while avoiding collision and adhering to the trajectory.

Machine manufactured parts, such as airplane wings or fuselage compartments may include differences when compared to their CAD files. For example, slight alterations may be made to the dimensions of portions of machine manufactured parts due to several issues during manufacturing. This can result in errors when relying on CAD files which specify specific dimensions, locations, or structural properties of the machine manufactured parts. For example, suppose a CAD file for an airplane wing designated a channel of the wing at a first set of dimensions. Yet when the airplane wing is manufactured the first set of dimensions actually manufactured for the airplane wing is slightly different. This can result in errors when other processes are performed on the airplane wing which rely on the original specifications for the airplane wing. When dealing with certain machine manufactured parts accuracy is vitally important and processes or operations which are performed on said parts using inaccurate information can lead to delays in production, wasted resources, and inefficient use of time.

Embodiments described herein addressed the above-referenced issues via a system that can determine differences between dimensions or structural properties of a target object as defined in a CAD file or some other measurement and as-is or as detected, using sensor input, dimensions or structural properties of the target object. The differences detected or determined between the properties of target objects can be used to modify previously specified trajectories or generate new trajectories for traversing a target object and performing an operation on said target object. The modified trajectories can result in more accurate operations performed on the target object thereby reducing the likelihood of errors and repeating a process on the target object. A robot with a sensor system may be instructed to move to a starting position of a target object, traverse the target object and obtain sensor input using the sensor system, and compare the sensor input for the target object or a structural property of the target object to the information of a CAD file for the target object (including a trajectory). The comparison between the information of the CAD file and the sensor input can be used to identify or detect differences which can result in modifications to the trajectory specified in the CAD file and eventually used by the robot to traverse the target object and perform an operation on the structural property of the target object.

The embodiments described below are described with reference to performing operations on a target object and a structural property of the target object such as a seam, channel, or gap. It should be appreciated that the functionality described herein can be applied to various applications that involve a robot with an end effector dispensing system that traverses the surface of a target object using a generated trajectory, such as painting, washing, and welding.

FIG.1is an illustration of a robot and computer system generating updates to a trajectory for traversing a surface of a target object102, in accordance with one or more embodiments.FIG.1includes a robot104that includes a sensor system106, an end effector dispensing system108, and image sensor110. In embodiments, the robot104may be in communication with a computer system112which includes a user interface114for presenting information about the robot104, the sensor input obtained by the sensor system106and image sensor110, as well as determined differences between a specified trajectory for the target object102(represented as second surface116and seam118) and the sensor input. In embodiments, the sensor system106may be configured to obtain sensor input (information obtained from the sensors of the sensor system106) as the robot104traverses the target object102. For example, the sensor system106may be a laser sensor system such as a light detection and ranging (LiDAR) sensor system or other sensors configured to detect distance and rotational orientation of the target object102by generating point cloud data, for example.

FIG.1includes a representation of the robot104traversing a first surface120, holding the target object102, and using the sensor system106to obtain a sensor input represented as the laser sensor detection122. In embodiments, the trajectory utilized by robot104to traverse the second surface116and obtain sensor input for seam118, is specified by a CAD file associated with target object102. In accordance with at least one embodiment, the computer system112may determine the CAD file for the target object based on images captured by the image sensor110of the target object102, the first surface120, and/or markers (machine readable codes)124. For example, the computer system112may compare the images obtained by image sensor110which include a certain configuration and/or location of the markers124to compare to known locations and/or configurations of makers in maintained images to determine a type of the target object102and an associated CAD file. In embodiments, the computer system112may maintain a database that maps configurations and/or locations of markers in images or other representations (e.g., point cloud data, 3D models, 2D models, etc.) to CAD files. The CAD files may specify a type of target object, a starting position for the target object, and a trajectory for the target object that includes a height to maintain the robot104at when performing an operation on said target object, as well as a width and depth of a structural property of the target object, such as seam118.

In some embodiments, the computer system112may read or scan the information included in markers124to identify the CAD file for the target object102. The computer system112may be configured to compare the information specified in the CAD file for the target object102to the information detected or identified by the sensor input from sensor system106. For example, the depth or width of the seam118may be different at certain locations—due to a manufacturer error or other issue during manufacturing of the target object102. In embodiments, the computer system112may be configured to modify or update the trajectory specified in the CAD file for the target object102by adjusting the measurements, dimensions, or properties of the trajectory in the CAD file in accordance with the actual measurements, dimensions, or properties for the target object102, second surface116, and/or seam118as determined from the sensor input of sensor system106. The computer system112thereafter can generate instructions and transmit them to the robot104for traversing the second surface116of the target object102using the updated trajectory and performing an operation on seam118, such as applying a sealant. The sensor system106can continue to obtain sensor input of the target object102as the robot104uses the end effector dispensing system108to apply a sealant to the seam118using the updated trajectory. The image sensor110can also obtain images of the operation of applying the sealant to the seam118by the robot104and end effector dispensing system108to generate waypoints for manual review by an operator associated with computer system112. For example, the images of the waypoints of the operation performed on the target object102may be presented via UI114for manual review.

FIG.2is an illustration of a robot registering a target object in a local coordinate system, in accordance with one or more embodiments.FIG.2depicts a robot220and target object250. The robot220may be an example of the robot104ofFIG.1.FIG.2illustrates the results of performing a localization of the target object250in the coordinate system of the robot220as described herein. The localization or registration of the target object250to robot220results in the definition (226) of an X, Y, and Z coordinates and rotations along each of the X axis, Y axis, and Z axis of the target object250in the local coordinate system of the robot220. The local coordinate system of the robot220may indicate a position of the target object250as being placed or otherwise attached to surface222.

As depicted inFIG.2, the robot220may include sensor systems, such as an image capture sensor270and an end effector dispensing system224. The image capture sensor270may be configured to obtain images, video, or other media of the target object250and/or surface222to aid in registering or localizing the target object250in the coordinate system of the robot220as well as identifying a particular CAD file for the target object250. The end effector dispensing system224may be configured to perform an operation on target object250such as applying a sealant, welding gaps, or other applicable operations. Once the target object250is registered or localized in the coordinate system of the robot220the robot220can utilize the end effector dispensing system224to traverse a surface of the target object250and perform an operation using the X, Y, and Z coordinates and rotations of the definition (226) for the target object250. The robot220can utilize the definition (226) of the axis and rotations in the coordinate system to rotate the robot220and/or the end effector dispensing system224relative to the surface of the target object250while performing the operation on the target object such as applying the sealant.

The robot220may use the definition226of the axis and rotations in the coordinate system to adjust the height of the end effector dispensing system224as it performs an operation on target object250. In embodiments, the robot220may utilize a trajectory generated by the robot220and/or a computer system in communication with the robot220to traverse the surface of the target object250and perform an operation such as applying a sealant. The trajectory planning may involve inverse and forward kinematics to derive a set of transformations (which may be referred to as an n-frame mode) for controlling the movements of the end effector dispensing system224of the robot220(including the joints and wrist) such that the end effector dispensing system224can follow a trajectory in space to traverse the surface of the target object250and perform an operation such as applying a sealant or removing a sealant. In embodiments, the image capture sensor270may have a field of view that includes the robot220, the end effector dispensing system224and surface222. The image capture sensor270may generate images that can be processed locally at the robot220or remotely at a computer system (not pictured) to determine whether target object250is positioned properly on surface222. If not, an offset can be determined and this offset can be used to further modify a known trajectory for traversing the target object250or generate a new trajectory for traversing the target object250to perform an operation. In embodiments, the coordinate system of the robot220can have an origin, such as the base of the robot220or an arm (not specified) where the end effector dispensing system224is mounted.

In an example, the end effector dispensing system224is operated to move about each of the axes to enable multiple degrees of freedom (DoF) via inverse kinematics, such as to enable six DoF. Inverse kinematics includes a process of computing the variable joint parameters needed to place the end of a kinematic chain (e.g., the end effector dispensing system224of the kinematic chain that includes the robot220and an arm (not specified) and its components), a given position and orientation relative to the start of the chain (e.g., the origin of the local coordinate system). Forward kinematics involves using kinematic equations to compute the position of the end effector dispensing system224from specified values for the joint parameters. Whereas inverse kinematics takes as input the Cartesian end effector position and orientation (e.g., as defined in the local coordinate system of the robot220) and calculates joint angles, forward kinematics (for the arm and end effector dispensing system224) takes as input joint angles and calculates the Cartesian position and orientation of the end effector dispensing system224. Through inverse and forward kinematics, the robot220can determine a configuration of its base, arm components, and end effector dispensing system224to effectuate the desired pose of the end effector dispensing system224. The robot220can apply various approaches for inverse and forward kinematics, such as a closed-form solution approach or an optimization approach. Once the robot220determines the desired position for each of the above-referenced components, the robot220can determine a motion for each of the components to reach the desired position. The end effector dispensing system224can be actuated to move about each of their axes to enable multiple DoF, and multiple configurations can lead to the desired pose, a phenomenon known as kinematic redundancy. Therefore, the robot220can apply an optimization technique to calculate an optimal motion for each component to reach the desired position.

The robot220can further identify one or more continuous trajectories that are collision free. The robot220can employ various techniques to avoid collisions with obstacles and/or targets by optimizing certain criteria. The robot220can simulate candidate trajectories and based on the localization determine whether any component will collide with an obstacle (e.g., the robot220, the surface222, any other parts or components of the robot220such as arms, cables, etc.) and/or a target (e.g., the target object250). If a candidate trajectory will result in a collision, the robot220can move on to a next candidate trajectory until it identifies a trajectory without a collision. The robot220can further store collision data based on past collisions or past calculations. The robot220can further eliminate candidate trajectories based on the stored collision data. For example, the robot220can compare candidate trajectories to collision data to determine whether a collision is likely. If a collision has a probability less than the threshold probability, the candidate trajectory can be eliminated. If, however, a collision has a probability greater than a threshold probability, the candidate trajectory is not eliminated.

FIG.3is an illustration of a target object with a computer aided design (CAD) trajectory being updated to generate an updated trajectory for traversing a seam of the target object, in accordance with one or more embodiments.FIG.3depicts a target object302that includes a seam304and a surface306of the target object302. The target object302may represent a portion of an airplane wing or a fuselage. The seam304may be a channel or gap which requires an operation to be performed on it such as applying a sealant or welding the material near the seam304together. In embodiments, the surface306of the target object302may be curved, elliptical, or concave, such that the seam304does not conform to a standard straight trajectory. As described herein, a robot and associated sensor system, may obtain sensor input for the target object302and seam304to identify changes or modifications between an expected trajectory (e.g., CAD specified trajectory308) and the actual trajectory detected based on the sensor input upon scanning the target object302and seam304.

For example, target object302may be associated with a CAD file that was used when manufacturing the target object302and seam304. However, slight modifications or alterations can occur during manufacturing that can result in slight alterations between a trajectory, as defined in the CAD file, for traversing the seam304which could result in incorrect operations being performed on the seam304by a robot. As described herein, the sensor system of the robot may be configured to obtain sensor input which can be used to determine differences between the CAD specified trajectory308and the actual trajectory as determined by the sensor input. For example, modifications can occur in the location (X, Y, or Z axis location) of the seam304in surface306, and the width or the depth of the seam304during manufacture.

As illustrated inFIG.3, a computer system may update or modify (310) the CAD specified trajectory to generate an updated trajectory312.FIG.3depicts the same surface306, seam304, and target object302but with a representation of the updated trajectory312which includes adjustments to the original trajectory314. It should be noted that the adjustments to the original trajectory314(e.g., the CAD specified trajectory308) are visible viaFIG.3, the changes detected between the CAD specified trajectory308and the updated trajectory312may be in the range of nanometers. The adjustments to the original trajectory314ofFIG.3are exaggerated for clarity. In embodiments, the updated trajectory312may be used by the robot and end effector dispensing system to more accurately perform an operation on target object302and seam304.FIG.3also depicts one or more way points316which may represent markers for obtaining image data (images or video) using image capturing sensors of the robot as the robot traverses the seam304, surface306, and target object302using the CAD specified trajectory308or the updated trajectory312. The images or video of the way points316may be provided to a computer system for manual review of the seam304prior to and after an operation, such as applying a sealant to the seam304. Indication of an error based on the manual review of the images or video of the way points316after performing an operation can result in further operations to correct any specified errors or mistakes during a previous operation. For example, the robot may be instructed to utilize a sealant wiping tool or some other removal tool to remove a recently applied sealant, traverse the surface306of the target object302, receive updated sensor input, further update the updated trajectory312, and traverse the seam304using the further updated trajectory312to reapply the sealant to seam304—thereby removing the previous detected errors. Way points316may be generated and recorded again during each operation to ensure accuracy of the operations performed.

FIG.4is a flow for updating trajectories and applying a sealant to a seam of a target object, in accordance with one or more embodiments. At402, a computer system may receive first input from one or more image sensors of a robot. The first input may include images of one or more markers attached to a first surface holding a target object. For example, the one or more markers may correspond to machine readable codes such as QR codes or bar codes. The one or more image sensors may include a camera(s) or other image capturing devices capable of obtaining images within detection range of the image sensors of a surface, target object, and/or structural property of the target object.

At404, the computer system may register a location of the target object in a coordinate system of the robot based at least in part on the first input. For example, the computer system can utilize a known location of the robot as well as images of the target object to localize the target object in the coordinate system of the robot.

At406, the computer system may determine a particular computer aided design (CAD) file from a plurality of CAD files based at least in part on the first input. The particular CAD file may identify a starting position to move the robot from the initial position associated with the robot (e.g., a current position of the robot). In embodiments, the computer system may use object detection algorithms or machine learning algorithms to compare the images of the first surface and/or target object to known images of first surfaces and/or target objects to identify a match between the images. Each known image may be associated with a CAD file from the plurality of CAD files and specify information for the target object such as a depth and width of a seam (structural property) of the target object, curvatures or rotation of the target object on the first surface, a starting position to move the robot to for traversing a trajectory, as well as the trajectory for traversing the structural property of the target object, etc.

At408, the computer system may instruct the robot to move to the starting position based on the particular CAD file. In embodiments, the robot may include a laser sensor system and an end effector dispensing system. The laser sensor system may include a LiDAR sensor system or other sensors configured to detect distance and rotational orientation of an object by generating point cloud data, for example. The end effector dispensing system may be configured to dispense a substance or otherwise perform an action to a structural property of a target object such as welding two separate objects with a gap in between them together.

At410, the computer system may instruct the robot to traverse a second surface of the target object and obtain sensor input from the laser sensor system starting from the starting position. As the robot traverses the second surface of the target object the laser sensor system obtains the sensor input from the target object as well as the structural property (e.g., seam). The robot may use the first trajectory to traverse the target object and structural property as specified in the particular CAD file. For example, the first trajectory may include locations to move the robot and laser sensory system as it traverses the target object and structural property to obtain the sensor input.

At412, the computer system may receive the sensor input from the laser sensor system that includes at least a height from the robot and/or laser sensor system to the target object, a width of the structural property, and a depth of the structural property. In embodiments, the sensor input is obtained in real time as the robot traverses the target object and structural property of the target object.

At414, the computer system generates a second trajectory for traversing the second surface of the target object and modifies the first trajectory based at least in part on the sensor input and identifying changes between at least one of the following: the first height to the target object, the width of the structural property, and/or the depth of the structural property as determined from the sensor input, compared to the first height to the target object, the width of the structural property, and/or the depth of the structural property specified in the particular CAD file. For example, the comparison may identify a difference between the width of a channel or seam at a particular location in the target object between what is specified in the CAD file for the width at that particular location and what is actually detected from the sensor input upon scanning the seam at the particular location by the robot. The differences detected between distances, depths, widths, etc., between what is specified in a CAD file and what is actually detected by sensors of the robot may be small such as in the range of nanometers. However, this difference can be critical when dealing with certain machine parts or components which correspond to the target object such as airplane parts, submarine parts, etc.

At416, the computer system may instruct the robot to move to the starting position, again, and traverse the second surface of the target object using the second trajectory and apply a sealing to the structural property using the end effector dispensing system of the robot. For example, in the case where the structural property corresponds to a seam in a machine part, the seam may be filled with a sealant to seal the machine part. In cases where the structural property corresponds to a gap between two objects, the end effector dispensing system may comprise a welding mechanism that welds the two objects together. In embodiments, the laser sensor system may include a light detection and ranging (LiDAR) sensors that are configured to generate point cloud data for the target object and/or the structural property of the target object. The robot may include a sealing wiping tool that is configured to wipe away or remove excess sealant or other materials after the sealant or other material has been applied to the structural property or seam of the target object.

FIG.5is a flow for updating trajectories and applying a sealant to a seam of a target object, in accordance with one or more embodiments. At502, a robot may obtain a CAD file for a target object that identifies a starting position to move a robot from an initial position associated with the robot.

At504, the robot may move from the initial position to the starting position based at least in part on the obtained CAD file. The robot may include an end effector dispensing system and a sensor system.

At506, the robot may traverse a surface of the target object and obtain sensor input from the sensor system starting from the starting position. The traversal of the surface of the target object may use a first trajectory that is specified or defined by the CAD file. The first trajectory may include instructions and locations to move the robot as it traverses the target object and correspond to a traversal path for a seam or structural property of the target object. In embodiments, the sensor system may be configured to obtain the sensor input for the seam and/or the target object.

At508, the robot may receive the sensor input from the sensor system that includes at least the first height to the target object, a width of the seam, and a depth of the seam. In embodiments, the first height to the target object may be compared to a height specified in the CAD file for the target object. The first height may define a height with which the robot must remain positioned from the target object to avoid collisions as well as to complete an operation with the target object, such as applying sealant, given known measurements or dimensions of the robot and the end effector dispensing system of the robot.

At510, the robot may generate a second trajectory for traversing the surface of the target object that modifies the first trajectory based at least in part on the sensor input and identifying changes between at least one of the first height to the target object, the width of the seam, or the depth of the seam as determined from the sensor input compared to the first height to the target object, the width of the seam, or the depth of the seam specified in the CAD file. The robot can determine differences between dimensions and measurements of the target object and seam as defined by the CAD file and what the actual dimensions and measurements of the target object and seam are from the sensor input. Differences can occur between when CAD files are generated and when target objects are manufactured. The comparison feature of the present disclosure can identify these differences and generate a new trajectory for performing an operation upon the target object and seam that is more accurate given the differences that can occur during manufacture of the target object and/or seam.

At512, the robot may move to the starting position and traverse the surface of the target object using the second trajectory and apply a sealant to the seam using the end effector dispensing system of the robot. In embodiments, the robot may include one or more image sensors that are configured to obtain images and/or video of the target object and/or seam. The robot may move to the starting position, after applying the sealant, and obtain images or first input of the sealant applied to the seam. The first input or images may be transmitted to another computer system for manual review and identification of any errors during the application operation. If any mistakes are found an indication or message may be transmitted, from the other computer system, which identifies a mistake or error during application of the sealant. In response to receiving such an indication or message the robot may move to the starting position and traverse the surface of the target object and use a sealing wiping tool to remove the sealant from the seam using the second trajectory. Another pass of the surface of the target object and seam may be performed where the sensor system may obtain updated sensor input which is used to generate a third trajectory. The robot may use the third trajectory to traverse the surface of the target object and reapply the sealant to the seam.

FIG.6is a flow for updating trajectories and applying a sealant to a seam of a target object, in accordance with one or more embodiments. At602, a computer system may generate, based on first sensor data generated by a sensor system of a robot, a three-dimensional (3D) of a surface of a target object, the 3D representation comprising a first set of data points. For example, the sensor system may include a LiDAR sensor system that is configured to generate point cloud data that represents a target object. The point cloud data can be segmented to associate point cloud data points to different objects in the point cloud data. The computer system can extract features from the segmented objects to identify a target object, such as an aircraft panel. In embodiments, the computer system may be configured to associate point cloud data to a target object and computing a normal to and/or from the surface of the target object. The computer system can store a three-dimensional representation (e.g., a point cloud or computer-aided design (CAD) file) of a target object for use in comparison to known target objects and known CAD files for the known target objects.

At604, the computer system may localize the target object in a coordinate system of the robot using the 3D representation of the surface of the target object and a known initial location of the robot. For example, the sensor system of the robot may be configured to collect measurement data to render a representation of a target object and determine the position of the robot relative to the target object. The sensor system may include thermal sensors, image-capturing devices, and spectral sensors. In some embodiments, the sensor system may be configured to transmit signals toward the target object and collect reflected signals from the target object. The sensor system can be a LiDAR sensor system to emit a pulsed laser towards the target object. The LiDAR sensors can further be configured to collect reflected signals from the target object. The LiDAR sensors can determine an angle of reflection of the reflected signal and a time elapse (time-of-flight) between transmitting the signal and receiving a reflected signal to determine a position of a reflection point on the surface of the target object relative to the LiDAR sensors of the robot. The robot and sensor system can continuously emit laser pulses and collect reflection signals. The computer system can use the signals and data collected from the sensor system to further generate a point cloud of the target object based on the collected reflection signals. In other embodiments, the sensor system can include laser-based sensors, magnetic sensors, or other appropriate sensors.

The computer system can implement one or more localization techniques to determine a position of the target object relative to a position of the robot. In other words, the computer system can determine six degrees of freedom data of the target object in the coordinate system (e.g., reference system) of the robot. For example, the six degrees of freedom data can include the x-axis, y-axis, and z-axis coordinates, and rotational data around each axis.

In some embodiments, one localization technique can include the computer system accessing a point cloud for the target object. For example, if the target object is a wing of an F-15 Eagle fighter jet, the predefined point cloud is of a reference wing of an F-15 Eagle fighter jet. The computer system can retrieve a predefined point cloud of a target object from a repository based on the specifications of the target object. In some instances, an operator can load the predefined point cloud onto the computer system prior to the robot approaching the target object or upon arriving at the target object. In other instances, the computer system can identify a feature of the target object to identify a predefined point cloud from a repository. For example, the computer system can implement computer vision techniques to identify an identifying feature (e.g., identifier such as a tail number, component, shape) of the target object. The computer system can further use the identifying feature to retrieve a predefined point cloud.

In embodiments, the computer system can compare the data points of the generated point cloud described above and the predefined point cloud to identify the target object. The pre-defined point cloud can be a priori and generated by, for example, a previous scanning of the target object, or via a reference target object. In general, the computer system can identify a pattern of data points of the generated point cloud that are similar to the pattern of data points of the predefined data cloud.

The computer system can, for example, apply an iterative closest point (ICP) technique. The computer system can keep the generated point cloud or the predefined point cloud fixed, while transforming the other point cloud to match the fixed point cloud. The transformation can be based on minimizing a distance between the data points of the fixed point cloud and the data points of the transformed point cloud. Alternatively, the computer system can use a convolutional neural network (CNN) point cloud matching technique or other machine learning algorithm. The computer system can rely upon feature extraction, matching, and outlier rejection. The CNN can receive the generated point cloud and the pre-defined point cloud as inputs. The computer system can extract features from the generated point cloud and the predefined point cloud. Based on the features, the CNN can generate a mapping from the generated point cloud to the predefined point cloud and estimate a rigid transformation. CNN point cloud matching techniques can vary, and it should be appreciated that the computer system is operable to execute each technique as desired.

As the generated point cloud data points include information regarding the relative distance from sensors used to scan the target object for point cloud generation, the computer system can determine the relative position (e.g., x-axis, y-axis, and z-axis coordinates, and rotational data around each axis) of the target object with respect to the coordinate system of the robot. This position data is derived from the above-described process of transforming one point cloud to match the other point cloud.

Another localization technique can be implemented in the event that the computer system can retrieve a predefined computer-assisted design (CAD) model. As described above, the computer system can scan the target object and generate a point cloud. The computer system can further apply a modeling technique to generate a model from the generated point cloud. The computer system can compare the pre-defined CAD model with the model generated by the computer system. Based on identifying a match, the computer system can determine the relative position (e.g., x-axis, y-axis, and z-axis coordinates, and rotational data around each axis) of the target object with respect to the coordinate system of the robot.

The model generated by the computer system can be a polygonal model. The computer system can employ various methods to generate the polygonal model. For instance, combinatorial approaches such as Delaunay triangulation, alpha shapes, and Voronoi diagrams create a triangle-based mesh by interpolating all or most of the points of the point cloud. Other methods directly reconstruct an approximate surface represented in implicit form by defining the implicit function as the sum of radial basis functions (RBFs) centered at the points, known as global fitting methods, or considering subsets of nearby points, known as local fitting methods.

Yet another localization technique that can be implemented by the computer system can include the use of synthetic markers (machine readable codes). The synthetic markers can be fabricated markers having a computer-readable code, such as a QR code or bar-code. The synthetic markers can be affixed to the target object or a surface holding the target object at known points of the target object or the surface holding the target object or having pre-defined distances between them. Each synthetic marker can further be uniquely identified through the computer-readable code. The computer system can use the sensor input from the sensor system from scanning the target object and detect the synthetic markers. The computer system can further transmit a signal from one or more sensors of the sensor system toward the synthetic markers and collect a reflected signal from the synthetic markers. Based on the detection, the computer system can further determine an angle of reflection of a reflected signal from the synthetic markers and a time elapse (time-of-flight) between transmitting a signal and receiving a reflected signal from the synthetic markers to determine a position of the synthetic markers on the surface of the target object relative to the sensors or the surface holding the target object. The computer system can also capture a two-dimensional (2D) image of the synthetic marker and compare the size and location of the image pixels to that of a reference image virtually located at the camera origin to determine a position of the synthetic markers on the surface of the target object relative to the sensors. Based on the determination, the computer system can determine the relative position (e.g., x-axis, y-axis, and z-axis coordinates, and rotational data around each axis) of the target object with respect to the coordinate system of the robot.

Each of the above localization techniques can be used across the whole surface of a target object, over a portion of the surface of the target object, over a surface holding the target object, or a portion of the surface holding the target object. For example, if the target object were an aircraft, the localization techniques could be performed over the entire surface area of the aircraft. In other instances, the computer system can perform the localization techniques over a portion of the aircraft, such as the wing or fuselage.

Additionally, each of the above localization techniques can be performed individually or in combination. Each of the above localization techniques can include one or more elements that can be incorporated into another technique. For example, the computer system can match a generated point cloud and pre-defined point cloud and also employ synthetic markers to assist in localizing the target object with respect to the coordinate system of the robot.

At606, the computer system may generate a computer aided design (CAD) file for the target object using the three-dimensional representation and the first set of data points, the CAD file specifying a starting position to move the robot to from an initial position of the robot and a first trajectory for traversing the surface of the target object, the first trajectory corresponding to a seam of the target object.

At608, the computer system can instruct the robot to move from the initial position to the starting position based at least in part on the CAD file, the robot including an end effector dispensing system.

At610, the computer system can instruct the robot to traverse the surface of the target object and obtain sensor input from the sensor system starting from the starting position, the traversal of the surface of the target object using the first trajectory defined by the CAD file, the sensor system configured to obtain the sensor input for the seam and the target object.

At612, the computer system can receive the sensor input from the sensor system including a first height to the target object, a width of the seam, and a depth of the seam.

At614, the computer system can generate a second trajectory for traversing the surface of the target object and modifies the first trajectory based at least in part on the sensor input and identifying changes between at least one of the first height to the target object, the width of the seam, or the depth of the seam as determined from the sensor input compared to the first height to the target object, the width of the seam, or the depth of the seam specified in the CAD file.

At616, the computer system can instruct the robot to move to the starting position and traverse the surface of the target object using the second trajectory and apply a sealant to the seam using the end effector dispensing system of the robot.

FIG.7is a block diagram of an example of a computing device700usable for implementing some aspects of the present disclosure. The computing device700includes a processor702coupled to a memory704via a bus712. The processor702can include one processing device or multiple processing devices. Examples of the processor702include a Field-Programmable Gate Array (FPGA), an application-specific integrated circuit (ASIC), a microprocessor, or any combination of these. The processor702can execute instructions706stored in the memory704to perform operations. For example, the processor can execute instructions706for using a robotic device to identify a seam or structural property of a target object, register the target object in a local coordinate system of the robot, determine adjustments or changes between a CAD specified trajectory for the seam of the target object, traverse the seam of the target object using a CAD specified trajectory or a generated trajectory that uses input from sensors to generate the trajectory that is different from the CAD specified trajectory, and apply sealant or perform other operations to the seam or structural property of the target object. In some examples, the instructions706can include processor-specific instructions generated by a compiler or an interpreter from code written in any suitable computer-programming language, such as C, C++, Python, or Java.

The memory704can include one memory device or multiple memory devices. The memory704may be non-volatile and include any type of memory device that retains stored information when powered off. Examples of the memory704can include electrically erasable and programmable read-only memory (EEPROM), flash memory, or any other type of non-volatile memory. At least some of the memory704includes a non-transitory computer-readable medium from which the processor702can read instructions706. A computer-readable medium can include electronic, optical, magnetic, or other storage devices capable of providing the processor702with computer-readable instructions or other program code. Examples of a computer-readable medium include magnetic disks, memory chips, ROM, random-access memory (RAM), an ASIC, a configured processor, optical storage, or any other medium from which a computer processor can read the instructions706.

The computing device700may also include other input and output (I/O) components. The input component708can include a mouse, a keyboard, a trackball, a touch pad, a touch-screen display, or any combination of these. The output component710can include a visual display, an audio display, a haptic display, or any combination of these. Examples of a visual display can include a liquid crystal display (LCD), a light-emitting diode (LED) display, and a touch-screen display. An example of an audio display can include speakers. Examples of a haptic display may include a piezoelectric device or an eccentric rotating mass (ERM) device.

The above description of certain examples, including illustrated examples, has been presented only for the purpose of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Modifications, adaptations, and uses thereof will be apparent to those skilled in the art without departing from the scope of the disclosure. For instance, any examples described herein can be combined with any other examples.

The use of the terms “a” and “an” and “the” and similar referents in the context of describing the disclosed embodiments (especially in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms “comprising,” “having,” “including,” and “containing” are to be construed as open-ended terms (i.e., meaning “including, but not limited to,”) unless otherwise noted. The term “connected” is to be construed as partly or wholly contained within, attached to, or joined together, even if there is something intervening. Recitations of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein and each separate value is incorporated into the specification as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (e.g., “such as”) provided herein, is intended merely to better illuminate embodiments and does not pose a limitation on the scope of the disclosure unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.