Patent Description:
A robotic device can control an end-effector to follow a particular toolpath in order to perform a task in connection with a target object. In practice, an end-effector generally follows a predefined trajectory, and therefore, if the target object deviates from its model, the end-effector following the predetermined trajectory is not likely to perform the task as desired. Disclosed herein is a system and method for generating instructions for an end-effector to perform a path-based task in connection with the target object. The instructions are generated based on a parametric model of the target object. In particular, the system can determine for the end-effector a parametric toolpath that is defined with respect to one or more features of the parametric model. Then, at runtime, the system can map the parametric toolpath to the features of the as-built target object. By defining the toolpath parametrically with respect to the object and/or worksite, and then mapping the toolpath to the as-built target object and/or worksite, the end-effector can perform the path-based task as desired even if the target object deviates from its model.

In one aspect, a method as recited in claim <NUM> provided.

In another aspect, a system as recited in claim <NUM> is provided. The system includes at least one communication interface for communication with a robotic device, where the robotic device comprises an end-effector configured to interact with object surfaces. The system also includes at least one communication interface for communication with a scanner device, and at least one processor. The system further includes a non-transitory computer readable medium (CRM) including program instructions executable by the at least one processor perform functions, the functions including: determining one or more parametric instructions for a path-based task by the end-effector on a surface of an object, where the one or more parametric instructions indicate a path on the surface, and where the one or more parametric instructions are based on a nominal representation of the surface. The functions also include determining that fabrication of the surface is complete and responsively cause the scanner device to perform a scan of the surface. The functions further include determining an observed representation of the surface based on the scan, and based on a comparison of the observed representation of the surface to the nominal representation of the surface, updating the one or more parametric-based instructions to provide for performance of the path-based task on the surface in accordance with the observed representation of the surface. The functions additionally include sending, via the at least one communication interface, the one or more updated instructions to the robotic device, so as to cause the robotic device to perform the path-based task on the surface.

These as well as other aspects, advantages, and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description with reference where appropriate to the accompanying drawings. Further, it should be understood that the description provided in this summary section and elsewhere in this document is intended to illustrate the claimed subject matter by way of example and not by way of limitation.

The following detailed description describes various features and functions of the disclosed systems and methods with reference to the accompanying figures. In the figures, similar symbols identify similar components, unless context dictates otherwise. The illustrative system and method embodiments described herein are not meant to be limiting. It may be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations, all of which are contemplated herein.

A robotic control system can include a robotic device located in a worksite, and the robotic device can include one or more end-effectors (e.g., end-of-arm tools) that allow the robotic device to perform tasks that involve manipulating or otherwise engaging physical objects. The robotic device can also include other components (e.g., sensors) that guide the end-effector's manipulation of physical objects. Within examples, the controller can send instructions to the robotic device to use the end-effector(s) and sensor(s) to perform tasks that involve manipulation of or interaction with physical objects. The task can be a part of a larger process or project, e.g., a construction project, that the system is implementing in the worksite.

In practice, robotic control systems can program an end-effector to perform different types of tasks, such as path-based tasks. In order to perform path-based tasks, the end-effector can be programmed to follow a defined trajectory (e.g., a path), and is provided with instructions as to the actions it should take (or not take) at various locations or areas while moving along the path. In particular, the task can include a number of actions to be performed by the end-effector at different points or areas along the defined trajectory. The trajectory can be predetermined by a user (e.g., a designer) or can be predetermined by the control system. Further, the robotic device can be configured to repeatedly perform the predetermined trajectory when executing a type of task. For example, the task may be a deposition task where an end-effector, which includes an adhesive applicator, is tasked with depositing an adhesive onto an object. The robotic device can move the end-effector in a predetermined trajectory in order to deposit the adhesive onto an object. In some examples, the robotic device can be configured to repeatedly perform a task by moving the end-effector along the predetermined trajectory (e.g., in a manufacturing or processing environment).

An example of a path-based task may be a deposition task in which an adhesive such as glue, caulk, mortar, on a surface having contours, is deposited onto a surface. A robotic device that is assigned the task may be equipped with an end-effector that is capable of dispensing an adhesive onto surfaces. Robotic instructions specify the movement of a robotic arm that controls the end-effector. The instructions may cause the robotic arm to move the end-effector to deposit the adhesive onto the surface in a desired pattern. As explained above, in practice, robotic devices may include predetermined instructions that cause the robotic arm to move along a predetermined trajectory to perform the adhesive deposition task, such as in a manufacturing process.

In some cases, the robotic device may perform a path-based task as part of a larger project. The robotic device may be part of a system that is executing a project to build an output product based on a design. Part of the process of building the output product may involve the robotic device performing a path-based task. For example, the robotic device may be configured to paint a surface of an output product after the product is fabricated or built by the system.

However, predetermined programming of a toolpath, as is being done in practice, is not suitable for path-based tasks, such as material deposition, sanding, deburring, grinding, insertion with tight tolerances, polishing, etc. Programming the robotic device by predefining the trajectory may result in inconsistencies while performing path-based tasks, unless the model is identical each time the task is performed. The inconsistencies may result from the "as-built" physical product deviating from its nominal design.

Disclosed herein are methods and systems that can generate a parametric toolpath for an end-effector to follow when the end-effector performs a task in connection with a target object. In an implementation, during a pre-build phase, the system can analyze a model of the object in order to determine how to move the end-effector so that the end-effector manipulates the object to achieve a result (i.e., task parameter) specified by the model. For instance, the model of an object may indicate that a surface of the object has a desired finish. The system can then analyze the model to (i) determine the desired finish of the surface, and (ii) generate a toolpath for a sanding end-effector such that the end-effector, when following the toolpath, can sand the surface to the desired finish. In particular, the system can define the toolpath as a function of one or more parameters. For instance, the parametric toolpath can be defined as a function of a position (e.g., distance and/or orientation) relative to one or more features of the modeled object.

Then once the system is scheduled to perform the task (i.e., runtime) during a build phase, the system can receive real-time sensor data indicative of the worksite, and based on the data, can generate a representation of the worksite. The system can then map the parametric toolpath to the representation of the worksite. Mapping the parametric toolpath includes (i) identifying on the representation the features with respect to which the toolpath is defined, and (ii) generating the toolpath with respect to those features.

The system can then generate instructions that cause the end-effector to perform the path-based task. Among other things, these instructions can cause the end-effector to move along the mapped parametric toolpath when performing the task. The system can then send the instructions to the robotic device so that the robotic device can perform the task.

By defining a parametric toolpath, as opposed to predetermined positional instructions, the system can account for any deviations in the worksite and/or in the as-built target object from their respective models. In particular, mapping the parametric toolpath to the real-time representation of the worksite accounts for any deviations because the mapped toolpath is defined with respect to the actual positions of the features and not to the positions indicated by a model of the object and/or worksite.

To illustrate this, consider a task of depositing an adhesive onto a surface of an object. Current systems use a model of the object to determine where the surface will be located in the worksite. These systems then define the toolpath based on where the model indicates the surface will be located. However, in a scenario where the as-built deviates from the model such that the location of the surface has shifted from the location indicated by the model, the end-effector following the toolpath is not likely to deposit the adhesive onto the desired areas of the surface. This result is due to the fact the toolpath was defined based on where the model indicated the surface will be located, and not on the actual location of the surface.

Conversely, in the system disclosed herein, the parametric toolpath can be defined with respect to features of the surface. Then, at runtime, the system can scan the object and can map the parametric toolpath to the features of the surface. In particular, mapping the toolpath includes identifying the features of the surface and determining coordinates for the toolpath such that the toolpath is at the defined distance from the features of the surface. Therefore, even if the particular surface is no longer at the location indicated by the object's model, the end-effector following the mapped parametric toolpath will deposit the adhesive onto the particular surface because the parametric toolpath was mapped to the as-built target object. Thus, the system disclosed herein, can adjust on-the-fly to any deviations or changes in the worksite or the object from their respective models. The ability to adjust on-the-fly allows the system to perform tasks without significant delays since the system does not have to update the model and generate new task instructions in response to detecting deviations or changes from the model.

<FIG> illustrates an example configuration of a robotic control system <NUM> that may be used in connection with the embodiments described herein. The robotic control system <NUM> may be configured to operate autonomously, semi-autonomously, and/or using directions provided by user(s). The robotic control system <NUM> may be responsible for managing a worksite, such as a construction site, a production site, a manufacturing site, an inspection or quality control site, etc. For example, in a construction or manufacturing worksite, the robotic control system <NUM> may be responsible for coordinating the construction or manufacturing of a product (also referred to interchangeably as "output product"). In particular, such a robotic control system may control one or more robotic devices to construct the product, and may also monitor the environment using one or more sensors.

As illustrated in <FIG>, the robotic device control system <NUM> includes a robotic device <NUM> configured to control an end-effector <NUM>. The end-effector <NUM> may be a tool end-effector that is configured to perform a task on a work surface (e.g., a surface of the output product) and may be mounted to a moveable component, such as a robotic arm, of the robotic device <NUM>. The robotic device <NUM> may be located within a worksite (e.g., production site <NUM> depicted in <FIG>).

According to one example, the worksite may be a factory floor where robotic devices install parts in an assembly line to assemble a product (e.g., a table, airplane wing, etc.). According to an additional example, rather than an assembly line, the worksite may be a worksite where robotic devices combine a variety of parts to construct a physical structure. In these examples, the worksite may be a temporary location from which the final physical structure may be delivered (e.g., as a product) to another location (e.g., a distributor or customer location) when completely built.

According to yet another example, the worksite may be a municipal site where robotic devices work with heavy construction materials to construct a bridge or a road. According to a further example, the worksite may be a construction site where robotic devices work with construction materials to construct a house or a building. The worksite may also be the interior of a house where robotic devices install housing materials to construct a section of the house. In these examples, the final physical structure is installed in the worksite. According to yet another example, the worksite may be a quality or inspection control site where robotic devices scan products to determine whether the products meet specified standards.

The robotic control system <NUM> may further include local sensor(s) <NUM> and global sensor(s) <NUM> configured to may provide environment data representative of the worksite. For example, the local sensor(s) <NUM> and global sensor(s) <NUM> may determine the location of various objects in the worksite, such as a product that is being built by the system, for example. As another example, the local sensor(s) <NUM> and the global sensor(s) <NUM> may provide the robotic control system <NUM> with data that the robotic control system <NUM> can use to generate a "world map" of the worksite. The world map may be indicative of a real-time or near real-time representation of the worksite. Thus, the local sensor(s) <NUM> and global sensor(s) <NUM> may provide the robotic control system <NUM> with data such that the robotic control system <NUM> can update the world map continuously or periodically. The robotic control system <NUM> could also use the sensor data and/or the world map to track the performance of a task in the worksite. In particular, the robotic control system <NUM> could track any robotic devices participating in the task, any materials used in performing the task, and any changes to the as-built product as a result of performing the task.

Additionally, the local sensor(s) <NUM> may be arranged on or within the robotic device <NUM> and may be configured to measure the location of the end-effector <NUM> with respect to a work surface (e.g., a surface of the product being built). The local sensor(s) <NUM> may also be configured to scan or capture features of the work surface. The global sensor(s) <NUM>, on the other hand, may be arranged within the worksite and may be configured to measure the location of the output product with respect to a coordinate system in the production site. The global sensor(s) <NUM> may also be configured to measure the location of the end-effector <NUM> with respect to the coordinate system or with respect to another object (e.g., location of the base of the robotic device). Further, the global sensor(s) may also be configured to measure the location of the robotic device <NUM>.

In an embodiment, the global sensor(s) <NUM> may include a laser tracker system with very high resolution (e.g., hundredths of a millimeter). The laser tracker system may be used to determine locations of objects in the worksite. However, the global sensor(s) <NUM> are not limited to laser tracker systems, but may include any sensor capable of capturing features of objects located in the worksite, such as motion capture sensors, scanners, light detection and ranging (LIDAR) sensors, point cloud sensors, ultrasonic range sensors, Global Positioning System (GPS) receivers, sonar, optical sensors, Radio Frequency identification (RFID) systems, Near Field Communication (NFC) chips, wireless sensors, radio sensors, radars, cameras (e.g., color cameras, grayscale cameras, and/or infrared cameras), and/or range sensors (e.g., ultrasonic and/or infrared), among others.

And the local sensor(s) <NUM> may include a high speed camera for providing optical flow data or an inertial measurement unit (IMU). However, the local sensor(s) <NUM> are not limited to high speed cameras or IMUs, but may include any sensor capable of measuring the location of the end-effector <NUM> with respect to a work surface or capturing features of the work surface. Such sensors include force sensors, proximity sensors, motion sensors (e.g., gyroscopes, and/or accelerometers), load sensors, position sensors, thermal imaging sensors, depth sensors (e.g., RGB-D, laser, structured-light, and/or a time-of-flight camera), ultrasonic range sensors, infrared sensors, optical sensors, Radio Frequency identification (RFID) systems, Near Field Communication (NFC) chips, wireless sensors, light sensors, touch sensors (e.g., capacitive sensors), scanners, cameras (e.g., color cameras, grayscale cameras, and/or infrared cameras), and/or range sensors (e.g., ultrasonic and/or infrared), among others. In some embodiments, the location of the end-effector <NUM> with respect to a work surface may be determined using wheel odometry and/or robot forward kinematics.

Additionally, the local sensor(s) <NUM> and global sensor(s) <NUM> may be positioned within or in the vicinity of the worksite, among other possible locations. For example, the local sensor(s) <NUM> may be attached to the robotic device <NUM>. In some embodiments, the global sensor(s) <NUM> may be arranged in fixed locations throughout the worksite, for example, as a dedicated sensing installation. Further, an example implementation may also use sensors incorporated within existing devices, such as mobile phones, laptops, and/or tablets. These devices may be in possession of workers located in the production site, such as construction workers in a construction site.

<FIG> also depicts a controller <NUM> that may receive data from the local sensor(s) <NUM> and global sensor(s) <NUM>. In particular, the local sensor(s) <NUM> and global sensor(s) <NUM> may provide sensor data to the controller <NUM> through a communication unit <NUM>. The communication unit <NUM> may include wired links and/or wireless links (e.g., using various wireless transmitters and receivers). A wired link may include, for example, a parallel bus or a serial bus such as a Universal Serial Bus (USB). A wireless link may include, for example, Bluetooth, IEEE <NUM>(IEEE <NUM> may refer to IEEE <NUM>-<NUM>, IEEE <NUM>. 11n-<NUM>, or any other IEEE <NUM> revision), Cellular (such as GSM, GPRS, CDMA, UMTS, EV-DO, WiMAX, HSPDA, or LTE), or Zigbee, among other possibilities. Furthermore, multiple wired and/or wireless protocols may be used, such as "<NUM>" or "<NUM>" data connectivity using a cellular communication protocol (e.g., CDMA, GSM, or WiMAX, as well as for "WiFi" connectivity using <NUM>).

In other examples, the robotic control system <NUM> may include access points through which the local sensor(s) <NUM> and global sensor(s) <NUM> and/or controller <NUM> may communicate with a cloud server. Access points may take various forms such as the form of a wireless access point (WAP) or wireless router. Further, if a connection is made using a cellular air-interface protocol, such as a CDMA or GSM protocol, an access point may be a base station in a cellular network that provides Internet connectivity via the cellular network.

The controller <NUM> is shown to include one or more processor(s) <NUM>, data storage <NUM>, program instructions <NUM>, an input/output unit <NUM>, and a power source <NUM>. Note that the controller <NUM> is shown for illustration purposes only, as the controller <NUM> may include additional components and/or have one or more components removed without departing from the scope of the disclosure. Further, note that the various components of the controller <NUM> may be arranged and connected in any manner. The controller <NUM> may be incorporated in whole or in part into the robotic device <NUM> or may take the form of a desktop computer, a laptop, a tablet, a wearable computing device, and/or a mobile phone, among other possibilities.

Each processor, from the one or more processor(s) <NUM>, may be a general-purpose processor or a special purpose processor (e.g., digital signal processors, application specific integrated circuits, etc.). The processor(s) <NUM> can be configured to execute computer-readable program instructions <NUM> that are stored in the data storage <NUM> and are executable to provide the functionality of the controller <NUM> described herein. For instance, the program instructions <NUM> may be executable to provide for processing of sensor data received from the local sensor(s) <NUM> and global sensor(s) <NUM>.

The data storage <NUM> may include or take the form of one or more computer-readable storage media that can be read or accessed by the processor(s) <NUM>. The one or more computer-readable storage media can include volatile and/or non-volatile storage components, such as optical, magnetic, organic or other memory or disc storage, which can be integrated in whole or in part with the processor(s) <NUM>. In some embodiments, the data storage <NUM> can be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disc storage unit), while in other embodiments, the data storage <NUM> can be implemented using two or more physical devices. Further, in addition to the computer-readable program instructions <NUM>, the data storage <NUM> may include additional data such as diagnostic data, among other possibilities. Further, the controller <NUM> may also include a power source <NUM> configured to supply power to various components of the controller <NUM>. Any type of power source may be used, such as direct current from a battery or alternating current from mains electricity.

The data storage <NUM> may also include designs (also referred to herein as models) <NUM> of output products that can built or assembled by the robotic control system <NUM>. The designs <NUM> may be two-dimensional (2D) and/or three-dimensional (3D) computer-aided designs (CAD). Also, the designs <NUM> may be embodied as computer-readable program instructions stored on the data storage <NUM>. Further, the designs <NUM> may be designs of objects that the robotic device <NUM> can manipulate and/or interact with. These objects, such as construction materials (e.g., bricks, tiles, etc.), may be located in the worksite and may be used by the robotic device <NUM> to construct the output product.

The designs <NUM> may also be indicative of how a physical structure of the output product will be configured and how it will perform once constructed by the robotic device <NUM>. As such, and as explained below, the controller <NUM> may use the designs <NUM> as a guide to generate instructions for the robotic device <NUM>. The instructions may be indicative of specific robotic actions that can be performed by the robotic device <NUM> to construct a portion of or the entire output product.

The designs <NUM> may also be indicative of a set of properties of the output product. For example, the set of properties may be indicative of aesthetic properties, such as color, markings, visual patterns, and surface finish/texture, among other properties. Additionally or alternatively, the set of properties may be indicative of mechanical properties, such as bending strength, brittleness, bulk modulus, coefficient of friction, compressive strength, creep, elasticity, fatigue strength, flexibility, fracture toughness, hardness, plasticity, resilience, shear strength, stiffness, stress/strain properties, surface roughness, tensile strength, toughness, viscosity, yield strength, and weight, among other properties. Additionally and/or alternatively, the set of properties may be indicative of geometric properties such as shape, size, orientation, angles, etc..

Additionally or alternatively, the set of properties may be indicative of electrical and/or magnetic properties, such as capacitance, conductivity, density, dielectric strength, field properties, inductance, permittivity, and resistance, among other properties. Additionally or alternatively, the set of properties may be indicative of chemical properties, such as corrosion resistance, flammability, pH, reactivity, stability, surface energy/tension, and toxicity, among other properties. Additionally or alternatively, the set of properties may be indicative of manufacturing properties for coating, cutting, drilling, forming and shaping processes, heat treating, joining, machining, rolling, sanding, and welding, among other techniques. Additionally or alternatively, the set of properties may be indicative of optical properties, such as absorbance, fluorescence, photosensitivity, reflectivity, refractive index, scattering, and transmittance, among other properties. Additionally or alternatively, the set of properties may be indicative of thermal properties, such as boiling point, critical point, emissivity, melting point, specific heat, thermal conductivity, thermal diffusivity, and thermal expansion, among other properties.

<FIG> further depicts the controller <NUM> including an input/output unit <NUM>. The input/output unit <NUM> may output information to a user via a display. The display may take on any form and may be arranged to project images and/or graphics to a user of the controller <NUM>. In an example arrangement, a projector within the input/output unit <NUM> may be configured to project various projections of images and/or graphics onto a surface of the display. The display may include: an opaque or a transparent (or semi-transparent) matrix display, such as an electroluminescent display or a liquid crystal display, one or more waveguides for delivering an image to the user's eyes, or other optical elements capable of delivering an image to the user. A corresponding display driver may be disposed within the controller <NUM> for driving such a matrix display. Other arrangements may also be possible for the display. As such, the display may show a graphical interface that may provide an application through which the user may interact with the systems disclosed herein.

Further, the robotic control system <NUM> may display the world map on the display of the input/output unit <NUM>. Therefore, the input/output unit <NUM> may display a real-time or near real-time representation of the worksite, including the as-built product. Accordingly, a user may monitor the progress of building or assembling the output product. Based on the real-time feedback data (e.g., data from local sensor(s) <NUM> and global sensor(s) <NUM>) indicative of the worksite, the displayed world map may be updated to reflect the real-time changes in the worksite.

The input/output unit <NUM> may also include task controls. The task controls may provide a user with real-time control of task execution. For instance, the user may be able to provide an input that may start, stop, skip, or modify a task. For instance, a graphical interface displayed on display may include a task queue of the tasks that the robotic control system <NUM> will perform. The graphical interface may allow the user to start, stop, skip, or modify a task. In some implementations, the graphical interface may allow the user to enter parameters relating to the output product. The graphical interface may allow the user to enter parameters that may relate to aspects of the output product, including dimensions, density, curvature properties, other geometric properties, materials to be used, and/or other numeric inputs.

In further examples, the graphical interface may contain a timeline of the building the output product. The timeline may have a cursor representing a current timestamp, which may represent a particular point in time of the process of building the output product. In addition, the timeline may contain buttons to play through the process at a particular speed, or fast-forward or rewind through the process. The timeline may be used to control the point in time at which the geometry and/or other aspects of the worksite are displayed within the display. Further, the timeline may be used to indicate a particular point in time either for purposes of simulating the output product or for visualizing within software an actual physical building process taking place within the worksite. Further, a user may modify the design of the output product via the graphical interface.

In some examples, the display may provide users with multiple 3D views of the worksite, and may allow a user to change the orientation and/or zoom of a particular view. In other examples, the display may present other types of representations of the worksite, such as numerical representations, as well or instead. In further examples, users may be provided with a three-dimensional (3D) modeling graphical interface that allows the user to alter one or more variables describing a worksite and/or the desired output product that affect a building process in the worksite.

In further examples, the graphical interface may include parameters describing aspects of the process during runtime. In particular, robot parameters may be displayed that describe characteristics of the robotic device <NUM>, such as the position of the robotic device <NUM>, physical tools currently being used by the robotic device <NUM>, and/or axes along which the robotic device <NUM> is currently operating within the worksite. Additionally, tool parameters may be displayed describing operating characteristics of the end-effector <NUM>. For instance, an amount of power being supplied to a spindle or an amount of force being used with a gripper could be displayed within an example graphical interface. Additionally, the graphical interface may display sensor data. The graphical interface may also contain controls related to ordering and/or speed of execution of tasks. Further, the graphical interface may contain controls relating to the robot actors, such as robot positions and diagnostics. Additionally, the graphical interface may allow for control of different attributes of the output product. Within the graphical interface, controls may be provided for manipulating one or more tasks being executed during runtime. For example, a user may be able to interact with graphical using touch input in order to modify a building process by altering planned tasks in real time or almost real time.

In some examples, a graphical interface may include a device control in order to select a particular device within a worksite. For example, the graphical interface may display the robot actors within worksite and may allow for a selection of a particular robotic device. Additionally, the graphical interface may include robot parameters, such as position information describing the current position of robotic devices. In some examples, the position could be displayed as Cartesian coordinates, as robot axes values, or both. In further examples, the position information may reflect the position of an end-effector of a robot actor or of a physical tool mounted on the robot's end-effector.

Additionally, the input/output unit <NUM> may receive user-input (e.g., from the user of the controller <NUM>). In particular, the input/output unit <NUM> may allow for interaction with the graphical interface such as for scrolling, providing text, and/or selecting various features of the application, among other possible interactions. The input/output unit <NUM> may take on various forms. In one example, the input/output unit <NUM> may include a pointing device such as a computing mouse used for control of the graphical interface. However, if the input/output unit <NUM> includes a touch screen display, touch-input can be received (e.g., such as using a finger or a stylus) that allows for control of the graphical interface. In another example, the input/output unit <NUM> may include a keyboard that provides for selection of numbers, characters and/or symbols to be displayed via the graphical interface. For instance, in the arrangement where the input/output unit <NUM> includes a touch screen display, portions the display may show the keyboard. Thus, touch-input on the portion of the display including the keyboard may result in user-input such as selection of specific numbers, characters, and/or symbols to be shown on the graphical interface via the display. In yet another example, the input/output unit <NUM> may include a voice input device that receives audio input, such as from a user through a microphone, that is then interpretable using one of various speech recognition techniques into one or more characters that may be shown via the display. Other examples may also be possible.

<FIG> illustrates a robotic device <NUM>, according to an example embodiment. The robotic device <NUM> may include aspects of the robotic device <NUM> depicted in <FIG>. As illustrated in <FIG>, the robotic device <NUM> may include a robotic arm <NUM> with an end-effector <NUM>. The end-effector <NUM> can be equipped with one or more other tools, grippers, or guides. The robotic arm <NUM> can move along one or more degrees of freedom. For example, the robotic arm <NUM> can move along six degrees of freedom, depicted in <FIG> as A1-A6. In certain examples, the robotic device <NUM> can move along one or more axes A0, such as along a rail (not shown) that allows side to side movement.

In certain embodiments, instructions may be given to the robotic device <NUM> to position the robotic arm <NUM> at a specific location. As described herein, positions of the robotic arm <NUM> along A1-A6 and/or the positions of the robotic device <NUM> along one or more axes A0 may be calculated by a process of the related controller. In alternative embodiments, position control of the robotic device <NUM> and/or the robotic arm <NUM> may require separate, individual settings and control commands. Robotic devices operating with fewer degrees of freedom may be used in some examples as well or instead.

The robotic device <NUM> may also be able to interchange end-effectors. In order to interchange end-effectors, the robotic device <NUM> may have access to a plurality of end-effectors that may be stored on or near the robotic device <NUM>. The plurality of end-effectors may include end-effectors of different types, such as tools, grippers, sensors, and guide end-effectors. As such, the robotic device <NUM>, which has the ability to interchange end-effectors, may be assigned different tasks that require different types of end-effectors. The task that the robotic device <NUM> is assigned may determine the end-effector that the robotic device <NUM> selects.

Within examples, the robotic device <NUM> may be used to perform tasks to assemble or construct an output product. Generally, a task may involve the robotic device <NUM> using an end-effector to interact with or manipulate an object (also referred to herein as a target object). To perform a task, the robotic device can move the end-effector along a path in order to interact with or manipulate the object.

As discussed above, generally existing robotic control systems cause an end-effector to follow a predetermined trajectory (e.g., a glue depositor in a production line that repeatedly deposits glue in the same pattern on each object in the production line). Typically, such robotic control systems analyze the model of the output product and then generate a predetermined toolpath for an end-effector based on the model of the output product.

However, as discussed above, the as-built product may deviate from the model of the output product. Therefore, when an end-effector follows a predetermined toolpath when performing a task that interacts with the as-built product, the result of the manipulation of the target object may also deviate from the product's model. As such, performing tasks by following predetermined toolpaths could introduce variations/errors into the as-built product. Even if a variation/error on its own may appear to be minor, the accrual of variations/errors may prevent the production process from producing the final product as defined by the model. To help overcome this issue, disclosed herein are various implementations for generating a toolpath for an end-effector such that the end-effector's performance of a task satisfies the model of the output product.

<FIG> illustrates a production site <NUM> in which the robotic device <NUM> is located, according to an exemplary embodiment. In particular, a controller <NUM> (e.g., a controller of a robotic control system) of the robotic device <NUM> could receive an instruction to build a new object or interact with an existing object in the worksite. For instance, the controller <NUM> could receive an instruction to modify an existing object in the worksite. As illustrated in <FIG>, the controller <NUM> can operate the robotic device <NUM> and can cause the robotic device <NUM> to interact with or manipulate physical objects located in or near the production site <NUM>, perhaps to build a new object or to modify an existing object. <FIG> further illustrates an as-built product <NUM>. The as-built product can be a built portion of the output product that is being built by the robotic device <NUM>. Within examples, the process of building the output product could involve a design phase (i.e., a pre-build phase) and a build phase.

In an embodiment, during the design phase, the controller <NUM> could determine a model of the object that the robotic device <NUM> is building and/or manipulating (also referred to herein as the output product). In particular, the controller <NUM> may determine a parametric design (e.g., design <NUM>) of the output product, and could analyze the design in order to determine how to build and/or manipulate the output product. Since the parametric design is indicative of the materials of the output product and defines the output product in terms of parameters, the controller <NUM> could determine, based on the parametric model, how to construct the output product. Constructing the output product includes building the output product (i.e., defined by functional features) and modifying the as-built output product to achieve the nonfunctional (e.g., aesthetic features) of the output product. Specifically, the output product may include one or more physical pieces each respectively made of at least one material. And these piece(s) may be worked on to produce the output product, such as by being cut, added together, and/or shaped into the desired product by one or more tools. By way of example, the output product could include a piece of metal and a piece of wood that could be added together and/or respectively shaped, among other possibilities.

In an embodiment, the parametric model may represent various features of the output product. Generally, each such feature may be a distinctive attribute or aspect of the product. By way of example, a feature of a product may be a surface of a table, legs of the table, and/or cup holders formed on the surface of the table, among other possibilities. Further, each feature of the product may also be respectively defined by one or more parameters. In particular, a parameter may be any numerical or other measurable factor that helps define a feature of the product and/or conditions of the product's operation. Furthermore, each parameter may have a respective value that explicitly defines the manner in which a respective feature should be produced. For example, a leg of a table may be defined by the parameter of length and that length may have a value (e.g., five meters (<NUM>)) explicitly defining the length of the table's leg and/or the length that the leg should have. As such, the model of the product may specify one or more parameters for one or more features as well as a respective value for each such parameter, so that the product is then produced to in accordance with those specifications.

Additionally, the relationship between the features of a model may be defined by one or more parameters and/or algorithms. For example, the relationship between two elements may be defined by the parameter of length and that length may have a value (e.g., five meters (<NUM>)). As such, the model of the product may specify one or more parameters that define a relationship between the features as well as a respective value for each such parameter, so that the product is then produced in accordance with those specifications.

In practice, various parameters could be feasibly specified in a model for a product and/or otherwise exhibited by the product when it is produced or being produced. Although some parameters are discussed herein by way of example, it should be understood that other parameters are possible as well without departing from the scope of the present disclosure. For instance, other parameters may include any of those that could be specified in currently-known and/or future developed computer-aided design (CAD) software (e.g.,concentricity), among others.

In one case, a parameter may be a position of a feature of the product. In one example, the position could be defined by a coordinate value relative to a coordinate system. In another example, the position could be defined by a distance value indicating a distance to a point in physical space and/or a point on the structure, among others. In yet another example, the position could be defined by a spatial vector indicating both magnitude (e.g., distance) and direction in physical space (e.g., a direction that the feature should face).

In another case, a parameter may be an orientation of a feature of the product. In one example, the orientation could be defined by an angle value indicating an angle of the feature relative to another. For instance, the orientation could be defined by an indication of parallelism or perpendicularity to another feature.

In yet another case, a parameter may be a material property of a feature of the product. For example, a material property could be defined by a value or other indication of material density, curvature, material strength, size, shape, conductivity, temperature, and/or color, among others.

In yet another case, a parameter may be an attribute of an interaction with a feature of the structure. For example, an attribute of an interaction could be defined by a force value indicating a force that was applied and/or should be applied to the feature. In another example, an attribute of an interaction could be defined by a pressure value indicating a pressure that was applied and/or should be applied to the feature. In yet another example, an attribute of an interaction could be defined by an indication of an approach for moving the feature (e.g., turn a knob clockwise versus counterclockwise during production of the structure). In yet another example, an attribute of an interaction could be defined by an indication of a movement characteristic that the feature should exhibit, such as by a maximum speed at which the feature is to be able to rotate. Other cases and examples are also possible.

Once the controller <NUM> has analyzed the parametric model output product, the controller <NUM> could then generate tasks that could be performed in order to build and/or modify the output product. A task could be any type of task that could be performed in connection with building a product. Example types of tasks include construction tasks, manufacturing tasks, assembly tasks, processing tasks, etc. Additionally, a task could involve interacting with objects located in the worksite <NUM>, such as parts, tools, obstacles, etc. Furthermore, a task could be performed using one or more tools available in the worksite <NUM>. In an example, the controller <NUM> may generate one or more path-based tasks to be performed by the robotic device <NUM>. Also, during the design phase, the controller <NUM> can determine an order in which to perform the task (i.e., order of operations).

In generating a task, the controller <NUM> could define the task based on the analysis of the parametric model of the output product. For instance, for a path-based task, the controller <NUM> could determine based on the parametric model a task parameter for the path-based task. Then, the controller <NUM> could define the task such that the task parameter is satisfied. By way of example, for a painting task, the controller <NUM> could determine based on the parametric model of the object one or more surface of the object that are to be painted. The controller <NUM> may determine based on the parametric model a task parameter of the painting task. For example, the task parameter could define a surface area of the object to be painted, a particular paint to use, a particular design to paint, among other examples.

Additionally, the controller <NUM> could determine a toolpath for an end-effector to perform the path-based task. For instance, the controller <NUM> could determine a painting pattern that would satisfy the paint task parameters. The controller <NUM> could then generate a toolpath for a paint brush end-effector such that the paint brush can paint the object according to the painting pattern. In an embodiment, the controller <NUM> could define the generated toolpath parametrically. For instance, the generated toolpath could be defined as a function of distance from a particular feature of the output product and/or as a function of distance from a particular feature or coordinate system in the worksite. Other example parameters are possible.

Once the controller <NUM> determines one or more tasks to be performed in connection with the output product, the controller <NUM> can then commence (immediately or at a later time) the production phase in which the output product is built and/or modified. In particular, the controller <NUM> can cause the robotic device <NUM> to perform the one or more tasks. For instance, when the robotic device <NUM> is scheduled to perform a task (i.e., runtime of the task), the controller <NUM> can generate instructions for the robotic device <NUM> to perform the task.

In an embodiment, the controller <NUM> could generate the instructions based on the task parameters or task data and real-time worksite data. In particular, the instructions could include instructions indicative of the toolpath for the end-effector. Within examples, the instructions that are generated by the controller <NUM>, and that cause an end-effector to perform a task by following a parametric toolpath, could be referred to as "parametric instructions. " To generate the instructions indicative of the toolpath, the controller <NUM> could use the real-time worksite data to receive or generate a representation of the worksite. Then, the controller <NUM> could identify the features (e.g., features of the as-built and/or the worksite) with respect to which the toolpath (that was generated in the design phase) is defined. Once the features are identified, the controller <NUM> could map the parametric toolpath to the identified features. The controller <NUM> could then generate instructions indicative of the mapped toolpath. By generating instructions/mapping the parametric instructions for tasks at runtime, the controller <NUM> can account for any changes in the worksite and/or any deviations from the parametric model. And by doing so, a task is performed more accurately with respect to a model/ideal task.

<FIG> and <FIG> illustrate an example task to be performed by the robotic device <NUM>, in accordance with an example embodiment. In this example, the task is an adhesion application task, and therefore, the end-effector <NUM> of the robotic device <NUM> includes an applicator or dispenser tool <NUM>. The applicator <NUM> can include a valve that stores a material, such as a sealant, adhesive, coating, caulk, binder, plaster, etc. And the material can have different forms including a solid, gas, gel, liquid, aerosol, etc..

Further, the applicator <NUM> can be configured to apply or dispense the material through an opening (e.g., a nozzle). For example, the applicator <NUM> can apply the material onto an object. Accordingly, the robotic device <NUM> can be configured to use the applicator <NUM> to perform tasks that involve material deposition. In this example, the task that is assigned to the robotic device <NUM> may be an adhesive application task in which the robotic device <NUM> applies an adhesive onto a target object, which in this example, is a tile.

In an embodiment, the controller <NUM> can generate instructions for the robotic device <NUM> to perform the adhesion application task. In particular, the instructions can be based on a parametric model 400A of the tile, which is illustrated in <FIG>. The parametric model 400A may be part of a larger parametric model of the output product. Further, the parametric model 400A can describe a desired parameter of the deposition. In an implementation, the system can analyze the parametric model in order to determine the parameter. By way of example, the parametric model indicates that two distinct pieces are joined together in the product. Accordingly, the system can determine that two pieces can be coupled together using an adhesive. Then, the system can determine the parameter that describes the desired deposition. For instance, the parameter can be a desired deposition coverage area on a surface 404A of the tile. The desired deposition coverage area can include specific areas to be covered on the surface 410A of the tile or can be defined as a percentage of an area on a surface of the tile to be covered (e.g., <NUM>% of a surface to be covered). In another example, the desired parameter of the deposition can be a uniformity of deposition, a desired strength, or a desired pulling area.

Based on the one or more parameters, the controller <NUM> can determine an adhesive deposition pattern to be deposited on the surface 410A of the tile. In particular, the adhesive deposition pattern is determined such that the pattern satisfies the desired parameter (e.g., deposition coverage area) indicated by the parametric model 400A. Within examples, the system can use one or more characteristics of the tile and/or one more characteristics of the worksite to determine the adhesive deposition pattern that satisfies the one or more parameters. The one or more object characteristics can be determined from the parametric model 400A of the tile and/or sensor data (e.g., data from local sensors and/or global sensors). Examples of relevant object characteristics for the adhesive application task may include dimensions of the target object, material of the target object, potential tension stress, compression stress, and/or strain of the target object, potential impact and vibration forces that the target object may experience, potential fatigue, potential contact with other chemicals, etc. And examples of relevant worksite characteristics include temperature, humidity, air-pressure, UV exposure, etc..

<FIG> illustrates an example deposition pattern 412A. As illustrated in <FIG>, the deposition pattern 412A, the deposition pattern may include segments 401A, 403A, and 405A. Determining the deposition pattern 412A involves determining dimensions of the pattern, such as a thickness/depth, width, and length of each segment of the deposition pattern 412A. It may also involve determining other features of the deposition, such as a type of adhesive to deposit. Other features can also be determined.

Once the system determines a deposition pattern, the system can then define the deposition pattern as a function of one or more parameters. In this example, the deposition pattern is defined as a function of distance from one or more features of the tile. As illustrated in <FIG>, the deposition pattern 412A on the surface 410A is defined with respect to edges 404A, 406A, and 408A such that there is a distance d<NUM> between edge 406A of the tile and segment 401A of the deposition pattern, a distance d<NUM> between edge 404A of the tile and segment 403A of the deposition pattern, and a distance d<NUM> between edge 408A of the tile and segment 405A of the deposition pattern.

Additionally and/or alternatively, the controller <NUM> can determine for the applicator <NUM> a toolpath to follow so that the applicator <NUM> can deposit the adhesive according to the parametric deposition pattern <NUM>. In particular, the controller <NUM> can use characteristics of the robotic device <NUM>, such as kinematic constraints (e.g., collision, joint limits, etc.) and/or freedoms, to determine the toolpath that the end-effector can follow in order to deposit adhesive in the deposition pattern 412A. After determining the toolpath, the controller <NUM> can then define the toolpath with respect to one or more features of the tile. For instance, similar to the deposition pattern 412A, the toolpath can be defined as a function of distance from the edges 404A, 406A, and 408A of the tile.

Subsequently, at runtime for example, the controller <NUM> can generate task instructions that can cause the robotic device <NUM> to perform the adhesive application task. In particular, the task instructions can cause the robotic device <NUM> to control the end-effector to deposit the adhesive according to the deposition pattern 412A. Therefore, the instructions can be indicative of the toolpath that the end-effector <NUM> follows in order to deposit adhesive in the deposition pattern 412A. Additionally, the instructions can be indicative of control parameters that cause the end-effector to deposit the adhesive in the deposition pattern 412A, and to satisfy the other features of the deposition (e.g., depth, adhesive type, curing time, etc.).

To determine the instructions indicative of the toolpath, the controller <NUM> can use sensor data to determine a location of the edges with respect to which the toolpath is defined. For instance, the controller <NUM> can use sensors to determine information indicative of the tile, including information indicative of the properties of the tile (e.g., material from which it is manufactured, surface smoothness, etc.). Additionally, the controller <NUM> can use the sensor data to generate or receive a 2D or 3D representation of the tile. The representation of the tile can be used to determine features of the as-built tile, such as the location and dimensions of the edges of the tile.

<FIG> illustrates a representation 400B of the as-built tile, according to an exemplary embodiment. As illustrated in <FIG>, the as-built tile 400B includes an edge 404B, an edge 406B, and an edge 408B. The edge 404B, edge 406B, and edge 408B of the as-built correspond to edge 404A, edge 406A, and edge 408A of the parametric model respectively. And a surface 410B of the as-built corresponds to the surface 410A of the model. In this example, however, due to manufacturing errors in making the tile, the dimensions of the as-built tile 400B deviate from the dimensions of the nominal model 400A of the tile. In particular, the length of the edges 404B, 406B, and 408B of the as-built are shorter than the lengths of edges 404A, 406A, and 408A of the parametric model respectively.

Within examples, once the controller has identified the features of the as-built, the controller <NUM> can then map the toolpath of the adhesive applicator <NUM> to the as-built. Mapping the toolpath can include defining the toolpath with respect to the edges 404B, 406B, and 408B of the as-built tile. Then once the adhesive applicator <NUM> follows the mapped toolpath, the applicator <NUM> can deposit adhesive in the deposition pattern 412B on the surface 410B. As illustrated in <FIG>, the deposition pattern 412B is located with respect to edges 404B, 406B, and 408B such that there is the distance d<NUM> between edge 406B of the tile and segment 401B of the deposition pattern, distance d<NUM> between edge 404B of the tile and segment 403B of the deposition pattern, and distance d<NUM> between edge 408B of the tile and segment 405B of the deposition pattern. The distances d<NUM>, d<NUM>, and d<NUM> shown in <FIG> are equivalent to the distances d<NUM>, d<NUM>, and d<NUM> shown in <FIG>. Therefore, despite the deviation in lengths of the edges 404B, 406B, and 408B from the model edges 404A, 406A, and 408A, the deposition pattern 412B is located with respect to the edges 404B, 406B, and 408B as specified by the model (i.e., location of the deposition pattern 412A with respect to the edges 404A, 406A, and 408A).

The controller <NUM> can then generate instructions that cause the end-effector to perform the adhesive deposition task. In particular, the controller <NUM> generates instructions indicative of a toolpath and one or more control parameters of the applicator <NUM>. The control parameters of the applicator <NUM> may include a speed of the applicator <NUM>, a pressure at which the applicator <NUM> deposits an adhesive, a type of adhesive to be deposited, etc. In some examples, a size of the nozzle (i.e., tip size) of the applicator <NUM> may be adjustable. In such examples, the parametric instructions can also specify the size of the nozzle. The instructions can also specify a dispense time for the applicator <NUM>. Other control parameters are possible.

Although <FIG> depict an example adhesion application task that can be performed using an adhesive applicator <NUM>, this example should not be limiting. For example, the robotic device <NUM> can include another tool in addition to the applicator <NUM>. The tool can be any tool that can shape, pattern, smooth, and/or move any material deposited by the applicator <NUM>. In such examples, the applicator <NUM> can deposit the material onto the surface 410B, and then the second tool (e.g., a trowel) may be used to shape the material into a particular shape or pattern.

<FIG> illustrates of another example of a path-based task performed in connection with the as-built object <NUM> (illustrated in <FIG>), according to an exemplary embodiment. In this example, the controller <NUM> can determine, based on a model of the object, that the production process of the object includes a task of depositing an adhesive onto a curved surface <NUM> of an object <NUM>. Then, the system can generate a deposition pattern that achieves a desired result indicated by the model. In an example, the deposition pattern can be based on features of the worksite and/or features of the object <NUM>. As explained above, the relevant features for the adhesive application task can include dimensions of the target object, material of the target object, potential tension stress, compression stress, and/or strain of the target object, potential impact and vibration forces that the target object may experience, potential fatigue, potential contact with other chemicals, etc. As illustrated in <FIG>, the controller <NUM> can generate a deposition pattern that includes segments 504A, 504B, 504C, 504D, 504E, and 504F. Within examples, the controller <NUM> can parametrically define the deposition pattern by defining the pattern with respect to one or more features of the object <NUM>.

<FIG> illustrates a side-view of the object <NUM>, according to an exemplary embodiment. As illustrated in <FIG>, the controller <NUM> can determine that each segment of the deposition pattern has a rectangular shape. As further illustrated in <FIG>, segment 504C of the deposition pattern may have a width (w<NUM>), a height (h<NUM>), and a length (not illustrated in <FIG>). Further, there may be a spacing (s<NUM>) between the segment 504C and segment 504D of the deposition pattern. Note that different portions of the deposition pattern on the surface <NUM> may have different shapes, dimensions, and spacings.

Additionally and/or alternatively, the control system <NUM> can determine one or more control parameters of the end-effector such that the end-effector can deposit the adhesive in the deposition pattern. The control parameters include a trajectory of the applicator, a speed of the applicator, a pressure at which the applicator deposits an adhesive, a type of adhesive to be deposited, a size of the nozzle (i.e., tip size) of the applicator, a dispense time, an adhesive type, etc..

At runtime, the controller <NUM> can scan the object <NUM> in order to determine any deviations from the model. In response to detecting a deviation, the controller <NUM> can update the one or more control parameters in order to generate a deposition pattern that accounts for the deviation. For example, the controller <NUM> can generate the deposition pattern illustrated in <FIG>. As illustrated in <FIG>, the segments of the deposition pattern can have a triangular shape. As further illustrated in <FIG>, a segment <NUM> of the deposition pattern may have a base width (w<NUM>), a height (h<NUM>), and a length (not illustrated in <FIG>). Further, there may be a spacing (s<NUM>) between the segment <NUM> and a segment <NUM> of the deposition pattern. <FIG> illustrates yet another example deposition pattern. As illustrated in <FIG>, the segments of the deposition pattern can have a circular or semicircular shape. As further illustrated in <FIG>, a segment <NUM> of the deposition pattern may have a diameter (di) and a length (not illustrated in <FIG>). Further, there may be a spacing (s<NUM>) between the segment <NUM> and a segment <NUM> of the deposition pattern. Other deposition patterns, shapes, and dimensions are possible.

Although <FIG> and <FIG> depict example adhesion application tasks, these examples should not be limiting. Other example path-based tasks may include any task where an end-effector moves with respect to a surface of an object in a particular trajectory. Such path-based tasks can be performed using other types of tools. Such tools can manipulate an object (e.g., remove material from a surface) and/or interact with the object without manipulating the object (e.g., sensors, scanners, etc.). Examples tools that can remove materials from a surface include tools that are configured to perform sanding, deburring, polishing, etc..

The robotic control system can perform a sanding task as part of a production process of an output product. The system can analyze a parametric model of the output product to define a parameter of the sanding task. For instance, the system can determine that a surface of the object has a particular finish, and then, can define the parameter to be the desired finish of the surface. Then, the system can generate along the surface of the object a sanding pattern that achieves the desired finish of the target object. Additionally, the system can define the sanding pattern parametrically, and can use the sanding pattern to generate a parametric toolpath for the sander end-effector.

At runtime, the system can use sensor data to detect deviations of the object from its parametric model. For example, the system can detect changes in the worksite and/or deviations from the model that would cause different interaction than originally determined between the tooling and the surface. For instance, due to a change to the surface, e.g., wood hardness, the system can determine a different pressure to exert using the sander to achieve the desired surface finish. Additionally, based on the sensor data, the system can map the parametric toolpath to the target object. Then the system can determine instructions for performing the sanding task, such as instructions indicative of the toolpath and/or instructions indicative of control parameters (e.g., rotational speed of the sander, pressure applied using the sander, etc.) of the sander.

In some embodiments, the robotic control system can manipulate or interact with an object in the worksite for which the system does not have a model. In such examples, the system, can generate the parametric toolpath for the end-effector in real-time. For instance, the system can use sensor data to generate a representation of the object. The system can then use the representation of the object to generate a parametric toolpath for the object.

<FIG> illustrates a carving task, according to an exemplary embodiment. In this example, the task is a carving task to carve the surface <NUM>. The system can cause a scanner (not illustrated) to scan the surface <NUM>, and based on the scan of the surface, the system can determine relevant features of the surface <NUM>. The relevant features for the carving task may include dimensions of the target object, material of the target object, potential tension stress, compression stress, and/or strain of the target object, potential impact and vibration forces that the target object may experience, potential fatigue, etc..

Based on the features, the system can determine one or more values for control parameters for the carving tool end-effector. The control parameters include a trajectory of the tool, a speed of the tool, a pressure with which to apply the tool to the surface, a size of the tool or tool-tip, etc. In particular, by determining values for these control parameters the system may determine a carving pattern <NUM> on the surface <NUM>. The carving pattern may result in a specified amount of material being removed from the surface, and therefore, results in certain shapes being carved into the surface. In some examples, the system may use the scan to determine the carving pattern. Then the system may determine the values of the control parameters that achieve the desired carving pattern.

As described above, the system can generate instructions for task a during the build phase (i.e., at the runtime of that particular task). However, in another embodiment, the system can also generate instructions during a design phase. In particular, after generating one or more tasks in the design phase, the system can then generate instructions for the one or more tasks, such as a path-based task. To generate the instructions, the system can define the desired path-based task in terms of one or more task parameters. The system can then generate based on the one or more task parameters of the path-based task, the system can generate parametric instructions for the task. The parametric instructions can be indicative of values of control parameters of the end-effector. In particular, if the control parameters of the end-effector are set at the determined values, the end-effector would perform the task such that the result of the performance meets the task parameters. In some examples, the system can use the parameters of the path-based task and a nominal model of the target object to generate the parametric instructions for the task. By way of example, the system can generate parametric instructions indicative of the toolpath of the end-effector. Such parametric instructions define the toolpath parametrically with respect to one or more features of the object and/or the worksite.

As explained above, in some cases, the target object can be constructed or assembled in the production site. However, the "as-built" target object can deviate from the nominal model of the object. In order to account for any deviations that may exist between the nominal model and the as-built physical object, the control system can cause a scanner to scan the as-built physical object. Based on the scan of the object, the control system can determine any deviations between the as-built physical object and the nominal model of the object.

If the control system detects any deviations, the control system may update the parametric instructions to account for the deviations. In particular, since the toolpath is described parametrically, the control system may rapidly update the instructions to account for the deviations such that the updated instructions satisfy the task parameters. The flexibility afforded to the system to adjust the parametric instructions, as opposed to predetermined positional instructions, allows the system to account for any deviations or changes in the production site. It also allows the system to rapidly account for any deviations without significant delays in the production process.

Operations relating to the robotic control system described above may be implemented as a method by one or more processors. As explained above, the robotic control system may operate one or more robotic devices. Therefore, there may be exchange of signals between the robotic device and the robotic control system. Example methods <NUM>, <NUM>, <NUM>, and <NUM> that describe the operations of a robotic control system are illustrated in the form of flowcharts in <FIG>, <FIG>, <FIG>, and <FIG> respectively.

<FIG> illustrates an example method <NUM> of the robotic control system performing a task, according to an example embodiment. As illustrated in block <NUM>, method <NUM> involves determining task information of a path-based task by an end-effector on an object, where the task information comprises (i) at least one task parameter, and (ii) a nominal representation of the object. As illustrated by block <NUM>, the method <NUM> also involves, based on the task information, determining one or more parametric instructions for the end-effector to perform the task, where the one or more parametric instructions indicate a toolpath for the end-effector to follow when performing the path-based task.

As shown in block <NUM>, the method <NUM> further involves generating, based on sensor data, an observed representation of the object. As illustrated by block <NUM>, the method <NUM> further involves comparing the observed and the nominal representations of the object. As illustrated by block <NUM>, the method <NUM> yet further involves based on the comparison, mapping the one or more parametric instructions to the observed representation of the object. In some examples, the one or more parametric instructions can be updated in real-time. As shown by block <NUM>, the method <NUM> additionally involves sending the mapped instructions to the end-effector to cause the robotic device to perform the path-based task.

<FIG> illustrates another example method <NUM> of the robotic control system performing a task, according to an example embodiment. As illustrated in block <NUM>, method <NUM> involves determining one or more parametric instructions for a path-based task by the end-effector on a surface of an object, where the one or more parametric instructions indicate a path on the surface, and where the one or more parametric instructions are based on a nominal representation of the surface. As illustrated by block <NUM>, the method <NUM> also involves determining that fabrication of the surface is complete and responsively cause the scanner device to perform a scan of the surface. For instance, the robotic control system could determine based on sensor data that the fabrication is complete. Additionally and/or alternatively, the robotic control system could receive from a robotic device controlling the end-effector an indication that the end-effector has completed fabrication of the surface.

As shown in block <NUM>, the method <NUM> further involves determining an observed representation of the surface based on the scan. As illustrated by block <NUM>, the method <NUM> further involves based on a comparison of the observed representation of the surface to the nominal representation of the surface, updating the one or more parametric-based instructions to provide for performance of the path-based task on the surface in accordance with the observed representation of the surface. As shown by block <NUM>, the method <NUM> additionally involves sending, via the at least one communication interface, the one or more updated instructions to the robotic device, so as to cause the robotic device to perform the path-based task on the surface.

<FIG> illustrates another example method <NUM> of the robotic control system performing a task, according to an example embodiment. As illustrated in block <NUM>, method <NUM> involves during a design phase, defining a task for an end-effector on a surface of an object based on at least one parameter of the robotic task. As illustrated by block <NUM>, the method <NUM> also involves based on a nominal representation of the surface and the parametrically defined task, determining one or more parametric instructions for the task, where the one or more parametric instructions indicate a path on the surface for the end-effector to follow when performing the path-based task.

As shown in block <NUM>, the method <NUM> further involves during a build phase, receiving, from a scanner, an observed representation of the surface. As illustrated by block <NUM>, the method <NUM> further involves comparing the observed and the nominal representations of the surface. As shown by block <NUM>, the method <NUM> additionally involves based on the comparison, updating the one or more parametric instructions to provide for performance of the path-based task on the surface according to the observed representation of the surface. As shown by block <NUM>, the method <NUM> additionally involves sending the updated instructions to the end-effector to cause the robotic device to perform the path-based task on the surface.

<FIG> illustrates another example method <NUM> of the robotic control system performing a task, according to an example embodiment. As shown by block <NUM>, the method <NUM> involves analyzing a parametric model of an object. Further, as shown by block <NUM>, the method <NUM> involves, based on the analysis, (i) determining a parameter of a path-based task to be performed on the object by an end-effector, and (ii) generating a parametric toolpath for the end-effector. As shown by block <NUM>, the method <NUM> also involves at runtime of the path-based task, receiving sensor data indicative of a representation the object. Furthermore, as shown by block <NUM>, the method <NUM> involves based on the sensor data, mapping the parametric toolpath to the representation of the object. Further, as shown by block <NUM>, the method <NUM> further involves generating instructions that cause the end-effector to perform the path-based task by following the mapped toolpath. Yet further, as shown by block <NUM>, the method <NUM> further involves sending the instructions to the end-effector to cause the robotic device to perform the path-based task on the object.

<FIG> shows an example configuration of a robotic device <NUM>. Generally, a robotic device <NUM> may be any device that has a computing ability and interacts with its surroundings with an actuation capability and/or with ability to emit/generate physical phenomena such as light and/or sound, among others. For instance, the robotic device <NUM> may be a humanoid robot, a robotic arm, or a quadruped robot, among others. A robotic device may also be any device that is generally understood to those of ordinary skill in the art as being a "robotic. " The robotic device <NUM> may also be referred to as a robotic device, a robotic manipulator, a robot client, or a robot, among others.

The robotic device <NUM> is shown to include processor(s) <NUM>, data storage <NUM>, program instructions <NUM>, controller <NUM>, sensor(s) <NUM>, power source(s) <NUM>, actuator(s) <NUM>, and movable component(s) <NUM>. Note that the robotic device <NUM> is shown for illustration purposes only and robotic device <NUM> may include additional components and/or have one or more components removed without departing from the scope of the disclosure. Further, note that the various components of robotic device <NUM> may be arranged and connected in any manner.

Moreover, the above description of processor(s) <NUM>, data storage <NUM>, program instructions <NUM>, sensors (e.g., local sensor(s) <NUM> and global sensor(s) <NUM>), and/or power source <NUM>, may apply to any discussion below relating to the respective component being used in another system or arrangements. For instance, as noted, <FIG> (among other possible figures) illustrates processors, data storage, program instructions, sensors, and/or power as being incorporated in another arrangement. These components at issue may thus take on the same or similar characteristics (and/or form) as the respective components discussed above in association with <FIG>. However, the components at issue could also take on other characteristics (and/or form) without departing from the scope of the disclosure.

As noted, the robotic device <NUM> may include a controller <NUM> (e.g., taking the form of a microcontroller). The controller <NUM> may include processing unit and data storage, and may be arranged to manage or carry out various operations (e.g., individually or in collaboration with processor(s) <NUM>). Thus, this controller <NUM> could take on the same or similar characteristics (and/or form) as the above-mentioned controller <NUM>, but could take on other characteristics (and/or form) as well. So in some implementations, the controller <NUM> may be incorporated as part the robotic device <NUM> and thus controller <NUM> may itself be controller <NUM>. In other implementation, controller <NUM> may be included as part of the robotic device <NUM> and controller <NUM> may be separate from the robotic device <NUM>. Regardless of the implementations, these controllers may take various forms. For instance, a controller may take the form of a chip set, a server system, a digital signal processor, a programmable logic controller, and/or a sampled-data system, among other possibilities. Moreover, a controller could also be referred to herein as a control system, among other.

Additionally, the robotic device <NUM> may also include one or more actuator(s) <NUM>. An actuator is a mechanism that may be used to introduce mechanical motion. In particular, an actuator may be configured to convert stored energy into movement of one or more components. Various mechanisms may be used to power an actuator. For instance, actuators may be powered by chemicals, compressed air, hydraulics, or electricity, among other possibilities. With this arrangement, actuator(s) <NUM> may cause movement of various movable component(s) <NUM> of the robotic device <NUM>. The moveable component(s) <NUM> may include appendages/members such as robotic arms, legs, and/or hands, among others. The moveable component(s) <NUM> may also include a movable base, wheels, and/or end-effectors, among others. Further, when a robotic device <NUM> includes at least one end-effector, such an end-effector may be a tool (e.g., a screwdriver, drill, welding iron, or some combination thereof) and/or a gripper, among others as discussed above.

The present disclosure is not to be limited in terms of the particular embodiments described in this application, which are intended as illustrations of various aspects. Many modifications and variations can be made without departing from its scope, as will be apparent to those skilled in the art. Functionally equivalent methods and apparatuses within the scope of the disclosure, in addition to those enumerated herein, will be apparent to those skilled in the art from the foregoing descriptions. Such modifications and variations are intended to fall within the scope of the appended claims.

The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The example embodiments described herein and in the figures are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

A block that represents a processing of information, such as a block of a method described above, may correspond to circuitry that can be configured to perform the specific logical functions of a herein-described method or technique. Alternatively or additionally, a block that represents a processing of information may correspond to a module, a segment, or a portion of program code (including related data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or actions in the method or technique. The program code and/or related data may be stored on any type of computer readable medium such as a storage device including a disk or hard drive or other storage medium.

The computer readable medium may also include non-transitory computer readable media such as computer-readable media that stores data for short periods of time like register memory, processor cache, and random access memory (RAM). The computer readable media may also include non-transitory computer readable media that stores program code and/or data for longer periods of time, such as secondary or persistent long term storage, like read only memory (ROM), optical or magnetic disks, compact-disc read only memory (CD-ROM), for example. The computer readable media may also be any other volatile or non-volatile storage systems. A computer readable medium may be considered a computer readable storage medium, for example, or a tangible storage device.

Moreover, a block that represents one or more information transmissions may correspond to information transmissions between software and/or hardware modules in the same physical device. However, other information transmissions may be between software modules and/or hardware modules in different physical devices.

Claim 1:
A method comprising:
determining task information of a path-based task to be performed by an end-effector (<NUM>) of a robotic device (<NUM>) on an object, wherein the task information comprises (i) at least one task parameter, and (ii) a nominal representation of the object;
based on the task information, determining one or more parametric instructions for the end-effector (<NUM>) to perform the task, wherein the one or more parametric instructions indicate a toolpath for the end-effector (<NUM>) to follow when performing the path-based task;
generating, based on sensor data, an observed representation of the object;
identifying one or more deviations between the observed representation of the object and the nominal representation of the object;
based at least on the one or more deviations, updating the one or more parametric instructions to provide for performance of the path-based task;
mapping the updated one or more parametric instructions to the observed representation of the object; and
sending the mapped parametric instructions to the end-effector to cause the robotic device (<NUM>) to perform the path-based task,
wherein the one or more parametric instructions further indicate one or more values for one or more control parameters controlling action of the end-effector (<NUM>) as the end-effector (<NUM>) moves along the toolpath, the one or more control parameters being updated based on the one or more deviations when the one or more parametric instructions are updated; and
wherein the path-based task is a sanding task, wherein the end-effector (<NUM>) comprises a sander, and wherein the task parameter comprises a desired surface finish, and further comprising:
determining material property information indicative of a type of material of the object; and
determining, based on the material property information and the task parameter, (i) a speed of motion of the end-effector (<NUM>) while moving along the toolpath, and (ii) pressures to be exerted by the end-effector (<NUM>).