Patent Description:
A prior art arrangement is known from <CIT>, which discloses a method and system for providing landmarks to facilitate robot localization and visual odometry. It describes an automated system according to the preamble of claim <NUM>.

It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are generally represented by like reference numerals for illustrative purposes throughout the figures. It also should be noted that the figures are only intended to facilitate the description of the preferred embodiments. The figures do not illustrate every aspect of the described embodiments and do not limit the scope of the present disclosure.

The following disclosure pertains to an automated drywalling system, which in embodiments is used for drywalling, including performing mud work.

Disclosed are systems and methods for automated mixing, delivering, applying, curing, and/or drying coatings onto a substrate. An automated drywalling system can be used to mix, deliver, apply, and dry joint compound on drywall boards. The automated drywalling system can be used to apply tape on seams between boards, apply joint compound or plaster onto the tape and boards, expedite the drying process, or any combination of these processes. The automated drywalling system can also be used to apply the joint tape and compound and achieve any level of drywall finish including between level <NUM> and level <NUM>. The automated drywalling system utilizes joint compound known as mud or setting type compound also known as hot mud. Joint compound as discussed herein can encompass pre-mixed, topping, taping, multi-use, all-purpose, and setting type compounds. The automated drywalling system can also be used with other coatings including plaster, cement, stucco, and paint applied onto drywall, lath, mesh or another suitable substrate. The automated drywalling system can cover how the coating is prepared, how it is delivered onto the substrate and how it is set, cured or dried.

According to the present invention in a first aspect, there is provided an automated drywalling system according to Claim <NUM>.

Further, preferable, features are presented in the dependent claims.

Turning to <FIG> and <FIG>, examples of an automated drywalling system <NUM> are illustrated, which includes a base unit <NUM>, a robotic arm <NUM> and an end effector <NUM>. The base unit <NUM> comprises a platform <NUM> and a cart <NUM> with a lift <NUM> disposed between the platform <NUM> and cart <NUM>. The cart <NUM> can be configured to be disposed on the ground and move within an XY plane defined by axes X and Y, and the lift <NUM> can be configured to raise the platform <NUM> up and down along axis Z, which is perpendicular to axes X and Y.

In the examples of <FIG> and <FIG>, the cart <NUM> can comprise a plurality of wheels <NUM>, which can be used to move the cart <NUM> and drywalling system <NUM> on the ground in the XY plane. Such movement can be motorized or can be non-motorized. For example, in some embodiments, the drywalling system <NUM> can be configured for automated movement of the cart <NUM>, motorized movement based on input from a user and/or non-motorized movement based on physical movement by a user. Additionally, while an example having wheels <NUM> is shown in some examples herein, it should be clear that the cart <NUM> can be configured for motorized and/or non-motorized movement via any suitable structures, systems, or the like.

In the examples of <FIG> and <FIG>, the lift <NUM> is shown comprising a scissor lift that can raise and lower the platform <NUM> relative to the cart <NUM> along axis Z. Such movement can be motorized or can be non-motorized. For example, in some embodiments, the drywalling system <NUM> can be configured for automated movement of the lift <NUM>, motorized movement of the lift <NUM> based on input from a user and/or non-motorized movement based on physical operation of the lift <NUM> by a user. Additionally, while an example of a scissor lift is shown herein, it should be clear that any suitable lift system can comprise the lift <NUM> without limitation.

The platform <NUM> can comprise a hub <NUM>, which can couple with the robotic arm <NUM> at a base end <NUM> of the robotic arm <NUM>. The hub <NUM> can comprise an input interface <NUM> that allows for various systems to couple with the hub <NUM>, which can allow for resources provided by such systems to be provided to the robotic arm <NUM> and/or the end effector <NUM> coupled at a distal end <NUM> of the robotic arm <NUM> as discussed in more detail herein. For example, a pneumatic source, a power source, a vacuum source, a paint source, a mud or joint compound source, or the like can be coupled to the hub <NUM>. <FIG> illustrates an example having an air compressor <NUM> and a vacuum source <NUM> coupled to the hub <NUM>. <FIG> illustrates an example having an air compressor <NUM> coupled to the hub <NUM>, which can be used to power pneumatic actuators <NUM> of the robotic arm <NUM> and/or provide compressed air to the end effector <NUM> at the distal end <NUM> of the robotic arm <NUM>.

In various embodiments, the robotic arm <NUM> can comprise any suitable robotic arm system, which can include pneumatic actuators, electric actuators, and the like. The robotic arm <NUM> can have any suitable number of degrees of freedom. Although the examples of <FIG> and <FIG> illustrate an example having pneumatic actuator units <NUM> separated by arm couplers <NUM>, this example configuration should not be construed to be limiting on the wide variety of robotic arms <NUM> that are within the scope and spirit of the present disclosure.

As discussed in more detail herein, an end effector <NUM> can be coupled at the distal end <NUM> of the robotic arm <NUM>. In some examples, the automated drywalling system <NUM> can comprise modular and/or multi-use end effectors <NUM>, which can be configured for various drywalling, construction, or other tasks. For example, as discussed herein, end effectors <NUM> can be configured for drywall planning, drywall hanging, applying mud or joint compound to hung drywall, sanding mudded drywall, painting, and the like. Although various examples herein relate to drywalling and construction, further embodiments of the drywalling system <NUM> can be configured for any suitable tasks, including construction tasks, manufacturing tasks, gardening tasks, farming tasks, domestic tasks, and the like. Accordingly, the discussions herein related to drywalling and construction should not be construed to be limiting on the wide variety of tasks that the system <NUM> can be configured for.

Turning to <FIG>, a block diagram of a drywalling system <NUM> is illustrated, which includes a base unit <NUM> coupled to a robotic arm <NUM>, which is coupled to an end effector <NUM>. The base unit <NUM> is shown comprising a control system <NUM>, which is operably coupled to a vision system <NUM>, sensors <NUM>, and a movement system <NUM>. The robotic arm <NUM> is shown comprising sensors <NUM> and a movement system <NUM>, which are operably coupled to the control system <NUM>. The example end effector <NUM> is shown comprising a vision system <NUM>, sensors <NUM>, a movement system <NUM>, and one or more end effector devices <NUM>, which are operably connected to the control system <NUM>.

In various embodiments, the connections between the control system <NUM> and respective vision systems <NUM>, <NUM>; respective sensors <NUM>, <NUM>, <NUM>; respective movement systems <NUM>, <NUM>, <NUM>; and end effector devices <NUM> can comprise any suitable type of connection including wired and/or wireless connections. For example, such connections can be configured for digital and/or analog communication of information between respective elements.

The vision systems <NUM>, <NUM> can comprise one or more suitable vision system including one or more visible spectrum camera, radar, light detection and ranging (LIDAR) system, sonar, infrared camera, thermal camera, stereo cameras, structured light camera, laser scanners, and the like. The vision systems <NUM>, <NUM> can comprise the same or different elements. Additionally, in some embodiments, one or both of the vision systems <NUM>, <NUM> can be absent. In some embodiments, the robotic arm <NUM> can comprise a vision system.

The sensors <NUM>, <NUM>, <NUM> can comprise any suitable sensors in various embodiments including one or more sensors of humidity, temperature, air flow, laser curtains, proximity sensors, force and torque sensors, pressure sensors, limit switches, rotameter, spring and piston flow meter, ultrasonic flow meter, turbine meter, paddlewheel meter, variable area meter, positive displacement, vortex meter, pitot tube or differential pressure meters, magnetic meters, humidity sensor, conductivity sensor and depth or thickness sensors. The sensors <NUM>, <NUM>, <NUM> can comprise the same or different elements. Additionally, in some embodiments, one or more of the sensors <NUM>, <NUM>, <NUM> can be absent.

The movement systems <NUM>, <NUM>, <NUM> can comprise any suitable movement systems in various embodiments including one or more of an electric motor, pneumatic actuators, piezo electric actuator, and the like. For example, in some embodiments the movement system <NUM> of the base unit <NUM> can comprise the lift <NUM> and motors that drive wheels <NUM> of the cart <NUM> (see <FIG> and <FIG>). In another example, the movement system <NUM> of the robotic arm <NUM> can comprise pneumatic actuators <NUM> as illustrated in the examples of <FIG> and <FIG>. In various embodiments, the movement system <NUM> of the end effector <NUM> can comprise motors or other systems that are configured to move, change the orientation of, rotate, or otherwise configure the end effector <NUM>. In some embodiments, one or more of the movement systems <NUM>, <NUM>, <NUM> can be absent.

As discussed herein, the one or more end effector devices <NUM> can comprise various suitable devices, including a cutting device, hanging device, mudding device, sanding device, painting device, vacuum device, and the like. Other suitable devices can be part of an end effector <NUM> and can be selected based on any desired task that the end effector <NUM> may be used for.

As discussed in more detail herein, the control system <NUM> can receive data from the vision systems <NUM>, <NUM> and sensors <NUM>, <NUM>, <NUM> and can drive the movement systems <NUM>, <NUM>, <NUM> and one or more end effector devices <NUM> to perform various tasks including drywall planning, drywall hanging, applying mud or joint compound to hung drywall, sanding mudded drywall, painting, and the like. Accordingly, the control system <NUM> can drive the drywalling system <NUM> to perform various suitable tasks, with some or all portions of such tasks being automated and performed with or without user interaction. The control system can comprise various suitable computing systems, including one or more processor and one or more memory storing instructions that if executed by the one or more processor, provide for the execution of tasks by the automated drywalling system <NUM> as discussed in detail herein. Additionally, while a control system <NUM> is shown as being part of the base unit <NUM>, in further embodiments, the control system can be part of the robotic arm <NUM> or end effector <NUM>. Also, further examples can include a plurality of control systems and/or control sub-systems, which can be suitably disposed in one or more of the base unit <NUM>, robotic arm <NUM>, and or end effector <NUM>.

Turning to <FIG>, an exemplary block diagram illustrating systems of an automated drywalling system <NUM> that includes a base unit <NUM> coupled to a robotic arm <NUM> and including a plurality of end effectors <NUM> configured to couple to the distal end <NUM> of the robotic arm <NUM>. In this example, the end effectors <NUM> include a cutting end effector 160C, a hanging end effector <NUM>, a mudding end effector <NUM>, a sanding end effector <NUM> and a painting end effector 160P.

As shown in <FIG>, the base unit <NUM> can comprise a vacuum source <NUM>, a paint source <NUM>, a mud source <NUM>, a power source <NUM>, and one or more base unit devices <NUM>. In various embodiments, one or more of the vacuum source <NUM>, paint source <NUM>, mud source <NUM>, and power source <NUM> can couple with a hub <NUM> (<FIG> and <FIG>) and provide resources to an end effector <NUM> coupled at the distal end <NUM> of the robotic arm <NUM> and/or to the robotic arm <NUM>. For example, the vacuum source <NUM> can be coupled with a vacuum tube <NUM> that extends via the robotic arm <NUM> to an end 424E, which can couple with an end effector <NUM> as discussed herein. The paint source <NUM> can be coupled with a paint tube <NUM> that extends via the robotic arm <NUM> to an end 432E, which can couple with an end effector <NUM> as discussed herein. The mud source <NUM> can be coupled with a mud tube <NUM> that extends via the robotic arm <NUM> to an end 432E, which can couple with an end effector <NUM> as discussed herein.

The power source <NUM> can be coupled with a power line <NUM> that extends via the robotic arm <NUM> to an end 436E, which can couple with an end effector <NUM> as discussed herein. Additionally, the power source <NUM> can provide power to arm devices <NUM> of the robotic arm <NUM> (e.g., sensors <NUM> and movement system <NUM>) and to base unit devices <NUM> of the base unit <NUM> (e.g., control system <NUM>, vision system <NUM>, sensors <NUM> and movement system <NUM>). In various embodiments, the power source can comprise one or more batteries and/or can be configured to plug into wall receptacles at a work site. For example, a power cord can be coupled to the power source <NUM>, which allow the drywalling system <NUM> to be powered by local power at a worksite via a wall receptacle, generator, external batteries, or the like. However, in some embodiments, the automated drywalling system <NUM> can be completely self-powered and can be configured to operate without external power sources at a worksite. In further embodiments, the robotic arm <NUM> and/or end effectors <NUM> can comprise a separate power source that can be separate from the power source <NUM> of the base unit.

In various embodiments, the automated drywalling system <NUM> can be configured to perform a plurality of tasks related to installing and finishing drywall in construction. In such embodiments, it can be desirable to have a base unit <NUM> and robotic arm <NUM> that can couple with and operate a plurality of different end effectors <NUM> to perform one or more tasks or portions of tasks related to drywalling. For example, the cutting end effector 160C, hanging end effector <NUM>, mudding end effector <NUM>, sanding end effector <NUM> and painting end effector 160P can be selectively coupled with the robotic arm <NUM> at the distal end <NUM> to perform respective tasks or portions of tasks related to drywalling.

For example, the cutting end effector 160C can be coupled at the distal end <NUM> of the robotic arm <NUM> and coupled with the power line <NUM> to power cutting devices <NUM> of the cutting end effector 160C. The cutting end effector 160C can be controlled by the automated drywalling system <NUM> to cut drywall or perform other cutting operations. In some examples, the cutting end effector 160C can comprise a cutting vacuum that is coupled to vacuum source <NUM> via the vacuum line <NUM> to ingest debris generated by cutting done by the cutting end effector 160C.

The hanging end effector <NUM> can alternatively be coupled at the distal end <NUM> of the robotic arm <NUM> and coupled with the power line <NUM> to power hanging devices <NUM> of the hanging end effector <NUM>. The hanging end effector <NUM> can be controlled by the automated drywalling system <NUM> to hang drywall, assist with drywall hanging, or the like.

The mudding end effector <NUM> can alternatively be coupled at the distal end <NUM> of the robotic arm <NUM> and coupled with the power line <NUM> to power mudding devices <NUM> and/or mudding applicators <NUM> of the mudding end effector <NUM>. The mudding end effector <NUM> can be controlled by the automated drywalling system <NUM> to perform "mudding" or "mud work" associated with drywalling, including application of joint compound (also known as "mud") to joints between pieces of hung drywall, and the like. Additionally, the mudding end effector can also be configured to apply joint tape, or the like. Additionally, the mudding end effector <NUM> can comprise a mudding vacuum <NUM> that is coupled to vacuum source <NUM> via the vacuum line <NUM> to ingest excess joint compound or mud generated by the mudding end effector <NUM>.

The sanding end effector <NUM> can alternatively be coupled at the distal end <NUM> of the robotic arm <NUM> and coupled with the power line <NUM> to power sanding devices <NUM> of the sanding end effector <NUM>. The sanding end effector <NUM> can be controlled by the automated drywalling system <NUM> to sand mudded drywall, and the like. Additionally, the sanding end effector <NUM> can comprise a sanding vacuum <NUM> that is coupled to vacuum source <NUM> via the vacuum line <NUM> to ingest debris generated by sanding done by the sanding end effector <NUM>.

The painting end effector 160P can alternatively be coupled at the distal end <NUM> of the robotic arm <NUM> and coupled with the power line <NUM> to power a paint sprayer <NUM> and/or painting devices <NUM> of the painting end effector 160P. The painting end effector 160P can be controlled by the automated drywalling system <NUM> to paint drywall or other surfaces. Additionally, the painting end effector 160P can comprise a painting vacuum <NUM> that is coupled to vacuum source <NUM> via the vacuum line <NUM> to ingest excess paint spray generated by painting done by the painting end effector 160P.

Although the example automated drywalling system <NUM> of <FIG> is illustrated having five modular end effectors <NUM>, other embodiments can include any suitable plurality of modular end effectors <NUM>, with such end effectors <NUM> having any suitable configuration, and being for any suitable task or purpose. In further examples, the automated drywalling system <NUM> can comprise a single end effector <NUM>, which can be permanently or removably coupled to the robotic arm <NUM>. Additionally, in some examples a given end effector <NUM> can be configured to perform a plurality of tasks. For example, in one embodiment, an end effector <NUM> can be configured for mud work, sanding and painting. Accordingly, the example of <FIG> should not be construed to be limiting on the wide variety of other embodiments that are within the scope and spirit of the present disclosure.

Turning to <FIG>, a method <NUM> of drywalling is illustrated, which can be performed in whole or in part by an automated drywalling system <NUM> as discussed herein. The example method <NUM> or portions thereof can be performed automatically by the automated drywalling system <NUM> with or without user interaction.

The method <NUM> begins at <NUM>, where a configuration and location of drywall pieces is planned. For example, in some embodiments, the automated drywalling system <NUM> can be configured for automated scanning and mapping of a worksite (e.g., framing elements of a house or building) and automated planning of the shapes and sizes of drywall to be disposed at the worksite to generate walls, ceilings, and the like. Such scanning and mapping can include use of vision systems <NUM>, <NUM> (<FIG>) and the like. Planning of shapes and sizes of drywall can be based at least in part on the scanning and mapping and can be performed by a computing device <NUM> of the automated drywalling system <NUM> or other suitable device which can be proximate or remote from the automated drywalling system <NUM>. In some embodiments, such planning can be based at least in part on building plans or maps that were not generated by the automated drywalling system <NUM>.

The method <NUM> continues to <NUM>, where drywall pieces are cut. Such cutting can be based at least in part on the scanning, mapping and planning discussed above. Additionally, such cutting can be performed by the automated drywalling system <NUM> at a worksite (e.g., via a cutting end effector 160C) or can be performed by a system remote from the worksite and generated drywall pieces can be delivered to the worksite.

At <NUM>, generated pieces of drywall can be hung at the worksite, including hanging on studs, beams, posts, wall plates, lintels, joists, and the like, to define walls, ceilings and the like. Screws, nails or other suitable fasteners can be used to hang the drywall pieces. In some embodiments, the automated drywalling system <NUM> can be configured to hang drywall pieces including positioning the drywall pieces and coupling the drywall pieces in a desired location. In some examples, the automated drywall system <NUM> can be configured to assist a user in hanging drywall, including holding the drywall and/or tools in place while the user fixes the drywall pieces in place. In various examples, a hanging end effector <NUM> can be used for such drywall hanging.

At <NUM>, mud work can be performed on the pieces of hung drywall. For example, joint compound (known also as "mud") can be applied to seams or joints between adjacent pieces of drywall, over surfaces of the drywall, and/or can be applied over fasteners such as drywall screws or the like. In various examples, a mudding end effector <NUM> can be used to perform such mud work.

At <NUM>, sanding can be performed on the mudded pieces of drywall. For example, where wet joint compound is applied to hung drywall pieces, the joint compound can be allowed to dry and can then be sanded by a sanding end effector <NUM> of an automated drywall system <NUM>. In various examples, sanding can be performed to smooth out joint compound to generate a planar or otherwise consistent profile on the pieces of drywall in preparation for painting. At <NUM>, the sanded drywall pieces can be painted. For example, in various examples, a painting end effector 160P of an automated drywalling system <NUM> can be used to paint the drywall pieces.

Although the method <NUM> of <FIG> relates to hanging and finishing drywall, it should be clear that other hanging and finishing methods can similarly be employed by the automated drywalling system <NUM>, including methods related to hanging particle board, plywood, sheet rock, laminate, tile, wall boards, metal sheeting, lath and the like. Similarly the methods can be used with different coatings including plaster, polymer coatings, cement, stucco, organic coatings, and the like. Accordingly, the method <NUM> of <FIG> should not be construed to be limiting.

In one aspect, the present disclosure pertains to systems and methods for automated mixing, delivering, applying, curing, and/or drying coatings onto a substrate. In one embodiment, an automated drywalling system <NUM> can be used to mix, deliver, apply, and dry joint compound on drywall boards. The automated drywalling system <NUM> can be used to apply tape on seams between boards, apply joint compound or plaster onto the tape and boards, expedite the drying process, or any combination of these processes. The automated drywalling system <NUM> can also be used to apply the joint tape and compound and achieve any level of drywall finish including between level <NUM> and level <NUM>. The automated drywalling system <NUM> can utilize joint compound known as mud or setting type compound also known as hot mud. Joint compound as discussed herein can encompass pre-mixed, topping, taping, multi-use and all-purpose compounds. The automated drywalling system <NUM> can also be used with other coatings including plaster, cement, stucco, and paint applied onto drywall, lath, mesh or another suitable substrate. The automated drywalling system <NUM> can cover how the coating is prepared, how it is delivered onto the substrate and how it is set, cured or dried.

The automated drywalling system <NUM> can include humidity, temperature, air flow sensors, or the like, to establish environmental conditions for a task. Such sensors can comprise sensors <NUM>, <NUM>, <NUM> of a base unit <NUM>, robotic arm <NUM> and/or end effector <NUM> of the automated drywalling system <NUM> (see, e.g., <FIG>). An automated coating system can utilize these environmental sensors to determine optimal joint compound mixture ratios, set path parameters such as feed speed, thickness of mud applied, blade profiles and pressures, and sprayer settings. The environmental information in conjunction with the joint compound parameters can be used to determine or estimate drying and setting times for the mud allowing the automated drywalling system <NUM> to plan when a next step should begin.

The automated drywalling system <NUM> can also determine when the joint compound has set and dried by measuring the moisture content, thermal conductivity of the covered seam, using a thermal imaging camera or thermometer (contact or non-contact), detecting differences in colors using a camera, or the like. Thermal measurements can be used to infer the moisture content by comparing the temperature of the joint compound to the surrounding materials, and as the water evaporates from the mixture, the temperature of the compound can be lower than that of the surrounding materials.

Models of the joint compound drying process can also be used to estimate the time to dry or cure given a set of starting conditions and information about the environment. Similarly, the models of the joint compound in combination with environmental and substrate information can be used to estimate the drying shrinkage of the joint compound.

Environmental sensors can be used in conjunction with an HVAC system, heater, air conditioner, fans, or the like, to control the room conditions. The sensor readings can trigger any of these systems or a combination to maintain the room at the desired conditions for quality, reduced drying or setting time, or comfort of the operator. In some embodiments, such environmental control systems can be a part of the automated drywalling system <NUM> or can be located external to the automated drywalling system <NUM> including environmental controls systems of a worksite. Accordingly, in various embodiments, the automated drywalling system <NUM> can be configured to control environmental control systems that are a part of or external to the automated drywalling system <NUM>, including via wired and/or wireless communication.

A mudding system can comprise of a variety of tools that enable the mudding system to mix, deliver, apply, smooth, dry, cure a coating of mud, or any combination of these. Such tools can be positioned and controlled using a robotic manipulator, positioning stage, gantry or any combination of these. A single end effector <NUM> or any multitude of end effectors <NUM> can be used to complete the task through coordinated or individual paths. The robotic arms <NUM> or tool stages can be moved around the room using a mobile base unit <NUM> that can be powered or moved manually by an operator. For example, in some embodiments a mudding system of an automated drywall system <NUM> can include one or more mudding end effector <NUM>, and elements associate with the base unit <NUM>, including a mud source <NUM> (see <FIG>).

The mobile base unit <NUM>, one or more end effectors <NUM> and/or one or more robotic arms <NUM> can include sensors (e.g., sensors <NUM>, <NUM>, <NUM> as discussed in <FIG>) to ensure safe operation next to the user. Safety sensors can include but are not limited to laser curtains, proximity sensors, force and torque sensors, pressure sensors, limit switches, or the like. Additionally, the automated drywall system <NUM> can include systems to track location of one or more user relative to end effector <NUM>, robotic arm <NUM> and/or mobile base unit <NUM>, including speed limiters and/or vision systems, such as LIDAR, radar, sonar, or any combination of these (for example, vision systems <NUM>, <NUM> of <FIG>).

As discussed herein, the mobile base <NUM> can include a vertical lift <NUM> that can be powered or unpowered. The vertical lift <NUM> can be used to lift or lower the robotic arm <NUM>, end effector <NUM> and portions of a mudding system, which can be disposed on the end effector <NUM>, platform <NUM>, a gantry or the like. The lift can be instrumented with a position sensor that can be used to capture and control the height of the lift <NUM>. For example such a sensor can comprise the sensors <NUM> as illustrated in <FIG>.

Elements of mudding system of the automated drywalling system <NUM> can be controlled using the control system <NUM> that takes a variety of inputs (e.g., from sensors <NUM>, <NUM>, <NUM> and/or vision systems <NUM>, <NUM>) to determine tool paths and/or tool parameters for the platform <NUM> relative to the cart <NUM>, robotic arm <NUM>, and mudding devices <NUM> and or mud applicator <NUM> of a mudding end effector <NUM>, which are required to achieve desired mud coating characteristics.

In various embodiments, the automated drywall system <NUM> can create a map of the target surfaces such as pieces of drywall, joints between pieces of drywall, and the like. This map or model can be created by importing building information modeling (BIM) and/or 2D, 3D plans into a planner system. The map can be created directly by the system by utilizing computer vision or mapping sensors to scan the room (e.g., the automated drywall system <NUM>). The scanning technologies can include, and suitable devices including stereo cameras, structured light cameras, LIDAR, radar, sonar, laser scanners, thermal imaging or any combination of these components. For example, in some embodiments, such scanning or vision systems can comprise the vision systems <NUM>, <NUM>.

Uploaded 3D or 2D plans can be combined with field data to create a more accurate map of the environment in some examples. The data from different sources can be combined using key features and user input. The map can include the location of framing studs, drywall joints, openings, protrusions, as well as pipes, electrical conduit, ventilation ducts, and any other components installed on the walls or ceilings. These locations may have been derived from the uploaded plans, the room scan, user inputs, and the like. To facilitate the creation of the map, a user can help identify features through analysis of images, tagging of the features physically or digitally. The user can physically tag components using various suitable methods, including but not limited to, a laser, tags, markers or a combination of these. The scanning or vision system can pick up these tags or track them as the user moves around the room and locates the features. The mapping system or planner can also take as an input a layout of how the drywall boards were hung in the room to locate seams. This layout can be an input from the automated drywalling system <NUM> or a system that is separate from the automated drywalling system. The location of framing, type of anchors used and layout of the drywall can provide information on the planarity, flatness of the wall, and location of high or low points, which can be used determine tool paths and tool parameters.

The automated drywalling system <NUM> can include a computational planner (e.g., implemented by the control system <NUM> of the base unit <NUM>) which can utilize a map uploaded to the system <NUM> or created by the system <NUM> to determine tool paths and/or tool parameters to achieve a desired coating application. The planner can create toolpaths off a global map of a room and then update these paths given updated local measurements once the end effector <NUM>, robotic arm <NUM>, and/or mobile base <NUM> are in place. The planner can be informed by vision system data (e.g. obtained by one or both of vision systems <NUM>, <NUM>) on the flatness of the wall, user inputs, location of seams as specified by a layout planner or a scan of the room after the substrate was applied. The planner can determine toolpaths and/or tool parameters to enable the automated drywalling system <NUM> to apply joint compound to smooth out joints, seams, low points, high points, and other features to create a visually flat wall.

For example, tool paths can include information corresponding to, or used to determine, instructions for one or more of movement systems <NUM>, <NUM>, <NUM> to drive the base unit <NUM>, robotic arm <NUM> and/or end effector <NUM> to move to perform desired tasks, including applying joint compound, applying joint tape, and the like. Tool parameters can include various setting for components of the end effector <NUM> (e.g., setting for the mud applicator <NUM> and/or mudding devices <NUM> of a mudding end effector <NUM>), including a nozzle selection, a nozzle size setting, a mud flow rate, and the like as discussed in more detail herein.

The toolpaths and/or tool parameters can also be determined based on a desired or required finish for completed mud work or for a completed wall assembly. For example, areas of a wall or ceiling that are exposed to changing, harsh, or bright lights can receive a higher quality finish with tighter controls on tool planarity, tool overlaps, thickness and characteristics of compound applied, texture.

The application of mud to a surface can inform how the surface is to be sanded, smoothed or polished to achieve a desired finish. For example, toolpaths and/or tool parameters generated during mud work can serve as inputs for generating toolpaths and/or tool parameters for sanding, which in some examples can enable sanding to be tuned according to the application of the compound, features, and compound characteristics such as how the compound was dried, compound type, compound hardness, and layers of compound applied.

For example, the automated drywalling system <NUM> can determine toolpaths and/or tool parameters for performing mud work with a mudding end effector <NUM>, and these determined toolpaths, tool parameters, and/or data associated thereto can be used to determine toolpaths and/or tool parameters for one or more sanding tasks to be performed by the automated drywalling system <NUM> using a sanding end effector <NUM>.

Similarly, determining toolpaths and/or tool parameters for performing mud work with a mudding end effector <NUM> can be based on various suitable inputs, including toolpaths, tool parameters, and/or the like associated with hanging drywall or applying insulation to a wall assembly on which the drywall is hung. For example, the automated drywalling system <NUM> can determine toolpaths and/or tool parameters for performing drywall hanging with a hanging end effector <NUM>, and these determined toolpaths, tool parameters, and/or data associated thereto can be used to determine toolpaths and/or tool parameters for one or more mudding tasks to be performed by the automated drywalling system <NUM> using a mudding end effector <NUM>.

During mud work, automated drywalling system <NUM> can apply a layer or profile of compound that is greater than a thickness that can be conventionally manually applied by human workers to allow for a sanding system (e.g., a sanding end effector <NUM>) to sand down the compound to a desired plane. For example, in some examples, manual joint compound application mud can be profiled to taper from high points. The automated drywalling system <NUM> can apply a thicker layer than normal enabling a sanding system to sand down high points to be level to the adjacent surfaces.

For example, <FIG> illustrate one example of a mud application profile for a pair of drywall pieces 610A, 610B that form a seam <NUM>, where joint compound <NUM> is applied over consecutive layers, which can include joint tape <NUM>, to taper out the high points of joint compound <NUM> over a wider area. Sanding can then be used to smooth out the final profile. The high points of joint compound <NUM> can be caused by various features, including the seam <NUM>, feature, raised stud, defect, or any combination of these. In some embodiments, such a mud application can be undesirable for automated application; however, in further embodiments, such a mud application profile can be employed by an automated system such as the automated drywalling system <NUM>.

<FIG> illustrate an example joint compound application process where the joint compound <NUM> is applied in a thick layer using a sprayer that generates a mud spray <NUM>. Such an application process can be performed by the automated drywalling system <NUM> in various embodiments. The thickness of the joint compound <NUM> being applied to the pieces of drywall 610A, 610B defining the seam <NUM> can allow for a sanding system to be used to sand back high points of joint compound <NUM> to a level surface. The high points of joint compound <NUM> can be caused by the seam <NUM>, feature, raised stud, defect, or any combination of these.

The 2D or 3D maps created by the automated drywall system <NUM> can be registered to the physical environment utilizing recognizable features such as doors, windows, outlets, corners, or the like. Such registration can also be done using markers, tags, laser outlines that are placed in the room, or the like. A projection and/or visualization system of the automated drywall system <NUM> can find the features or markers and can locate the maps created using these found features or markers. The automated drywall system <NUM> can utilize a user interface to enable the user to help locate the map or projection relative to the environment and resolve any issues or discrepancies. A user can utilize a physical marker to signify key features for the automated drywall system <NUM> allowing the automated drywall system <NUM> to locate the plan relative to the environment. The automated drywall system <NUM> can also use a robotic manipulator or end effector <NUM> to find target features, markers or surfaces and locate them relative to its own base unit <NUM> which can be located using a localization system including, but not limited to laser range finders, computer vision, LIDAR, radar, sonar, stereo vision, odometry, IMUs, or any combination of these.

The robotic arm <NUM> can utilize a compliant end effector <NUM> to enable safe contact with the environment allowing the automated drywalling system <NUM> to accurately locate target surfaces, features or components, accommodate errors in positioning without damaging the substrate or the end effector <NUM>. By utilizing the robotic arm <NUM> and compliant end effector <NUM> to locate a physical component, the system <NUM> can establish a point, line, or plane and therefore locate the virtual plan on the environment. Toolpaths can be updated from the virtual plane to the physical plane. Refitting of the toolpaths onto the contacted surfaces can enable the system <NUM> to deal with errors and discrepancies between the modeled and physical environment. Such tools, features or elements of the system <NUM> can enable quick on-site calibration using global room wide maps and local measurements. Refitting the toolpaths can allow for errors in positioning of end effector <NUM>, mobile base <NUM> or robotic arm <NUM>. The system <NUM>, including an end effector <NUM> can utilize radar, sonar, thermal imaging to establish what is behind the substrate (e.g., drywall), this information can be used to update a virtual map and ensure that no damage is done to any electrical, plumbing or ventilation while working on or about the substrate.

The planner can output tool poses or tool paths for the automated drywalling system <NUM> (e.g., for an end effector <NUM>, robotic arm <NUM>, base unit <NUM>) including, but not limited to joint commands, target poses and end effector positions, or any combination of these. The system <NUM> can also output paths for a gantry system or positioning stage which can be used in conjunction with the robotic arm <NUM> and/or end effector <NUM> or without a robot to move and position coating tools (e.g., mudding devices <NUM> and/or mudding applicators <NUM> of a mudding end effector <NUM>). The planner can also output paths for the mobile base <NUM> to position a gantry, positioning stage, robotic arm <NUM>, end effector <NUM>, or to move a tool to assist a user in the finishing process, or to position visualization and lighting equipment, which may or may not be a part of the automated drywalling system <NUM>. The mobile base <NUM> and vertical lift <NUM> may work in coordination with a user, robotic arm <NUM>, end effector <NUM> or a combination of these to execute the task. The planner system can control different components of the automated drywalling system <NUM> (e.g., the base unit <NUM>, robotic arm <NUM> and/or end effector <NUM>) allowing for coordinated movements and forces with the target goal of moving the end effector <NUM> or portions thereof to a desired position under the prescribed forces and moments. The mobile base unit <NUM> can be used as a rough positioning stage, with the vertical lift <NUM> setting the height of the robotic arm <NUM> and end effector <NUM> which may act as a fine positioning stage.

Turning to <FIG> and <FIG>, examples of a wall assembly <NUM> including a plurality of drywall pieces 610A, 610B, 610C, 610D is illustrated. The wall assembly <NUM> can comprise a header <NUM> and footer <NUM>, with a plurality of studs <NUM> extending therebetween. As shown in <FIG>, the drywall pieces <NUM> can be coupled to the studs <NUM> via a plurality of fasteners (e.g., drywall screws) that extend though the drywall pieces <NUM> and into the studs <NUM>. The drywall pieces <NUM> can define one or more seams <NUM>, including in the example of <FIG> a vertical seam 620V and a horizontal seam <NUM>. In some embodiments, mud work can be performed on the seams <NUM> as shown in <FIG> and leaving portions of the drywall pieces <NUM> without joint compound <NUM>. Additionally or alternatively, joint compound can be applied to portions of the drywall pieces <NUM> in addition to about the seams <NUM> as shown in <FIG>.

<FIG> illustrates one example embodiment of the automated drywall system <NUM>, having a mudding end effector <NUM> that is configured to generate a mud spray. In this example embodiment, the system <NUM> is shown comprising a robotic arm <NUM> with a spraying end effector <NUM> with integrated overspray collection <NUM>. The robotic arm <NUM> and end effector <NUM> are shown mounted on a mobile base <NUM> with a vertical lift <NUM>. The base unit <NUM> can carry supporting systems for the automated drywall system <NUM> as discussed herein.

An end effector <NUM>, such as the embodiment 160M1 of a mudding end effector <NUM>, shown in <FIG> can utilize a tool to automatically dispense and apply joint tape <NUM> at the seams <NUM> between drywall boards <NUM>. In this example embodiment 160M1, joint compound <NUM> can be dispensed from a flat box <NUM> and joint tape <NUM> can dispensed from a roller <NUM>. The joint tape <NUM> can come into contact with the compound before a blade <NUM>, which can be used to apply the joint tape <NUM> and joint compound <NUM> onto the seam <NUM>. The blade <NUM> can smooth the tape <NUM> down and can apply the joint compound <NUM> on the seam <NUM>.

In one embodiment, joint tape <NUM> can be fed off a roll <NUM> onto a joint <NUM> defined by a first and second drywall piece 610A, 610B after being covered with joint compound <NUM>. In some embodiments, joint compound <NUM> can be delivered ahead of the tape <NUM> and the <NUM> tape can be flattened onto the surface of the drywall pieces <NUM> and seam <NUM> using a blade or trowel <NUM>. The end effector <NUM> or other portion of the system <NUM> can also be used to automatically apply the tape <NUM> using tools such as banjo and bazooka systems. Tracking the position of the end effector <NUM> and portions thereof with devices, sensors, vision systems or other elements of the end effector <NUM>, robotic arm <NUM> and/or base unit <NUM> can enable the planner to create an updated map of the room with the location of the tape <NUM> and/or joint compound <NUM> and the conditions under which one or both were applied.

One or more vision system <NUM>, <NUM> can also use tags or markers to track as an end effector <NUM> or as a user applies tape <NUM> on the surfaces and/or seam <NUM> of one or more drywall pieces <NUM> and that information can be communicated to and stored by the planner. The end effector <NUM> and/or robotic arm <NUM> can be used to control the orientation of tools or devices of the end effector <NUM> and the force applied on a surface as tape <NUM> is applied, which can be desirable in some examples to ensure that the tape <NUM> is embedded within joint compound <NUM> as desired. The drywalling system <NUM> can apply, solid, porous and/or mesh joint tape <NUM> with or without adhesive that can be covered with joint compound <NUM> using a separate tool or a tool associated with an end effector <NUM>.

Joint tape <NUM> can be applied by the automated drywalling system <NUM> and/or by an operator. Additionally, in some embodiments, joint tape <NUM> can be colored, dyed or marked so that it is easier for a vision system <NUM>, <NUM> to identify the joint tape <NUM>. Different color tapes <NUM>, or tapes <NUM> having different identifying features (e.g., textures, images, barcodes, or the like) can be used in some embodiments to provide information to the automated drywalling system <NUM> about the identity or characteristics of a specific joint <NUM> or other feature of one or more piece of drywall <NUM>. For example, butt joints can be covered with a first color tape <NUM>, tapered joints can be covered with a second color tape <NUM>, and factory joints can be covered with a third color tape <NUM>. An end effector <NUM> can also use a joint compound <NUM> that comprises fibers in addition to, or as an alternative to, tape <NUM>. One or more vision system <NUM>, <NUM> can be used to identify seams <NUM> between drywall pieces <NUM> and data from such vision systems <NUM>, <NUM> can be used to guide an end effector <NUM> during taping. The end effector <NUM> can also be guided using the planner's map of the surface which is located on the environment using relevant features such as markers, corners, openings, or the like.

The joint compound <NUM> or other coating can be delivered or applied onto joint tape <NUM>, seams <NUM> and/or surfaces of drywall pieces <NUM> using a variety end effectors <NUM> having a variety of elements, devices, or tools. For example, <FIG> illustrates one embodiment 160M2 of a mudding end effector <NUM> that includes a spray gun <NUM> that is coupled onto the robotic arm <NUM>. A trigger <NUM> can be actuated with an actuator <NUM> (e.g., a servo, solenoid, pneumatic cylinder, or the like) which can pull on the trigger <NUM> to open the nozzle <NUM> to generate a mud spray <NUM>.

In another example, <FIG> illustrates another embodiment 160M3 of a mudding end effector <NUM> that includes a spray gun <NUM> that is coupled onto the robotic arm <NUM>. An internal trigger (not shown) can be actuated with an actuator (e.g., a servo, solenoid, pneumatic cylinder, or the like) which can open the nozzle <NUM> to generate a mud spray <NUM>. In the examples of <FIG> and <FIG>, mud can be fed to the spray gun <NUM> and nozzle <NUM> via a mud tube <NUM>, which can feed mud (e.g., joint compound, or the like) from a mud source <NUM> disposed at the base unit <NUM> (See <FIG>).

In various embodiments a spray gun <NUM> can comprise an airless spray system or air assisted spray system. A pump can be used to move the joint compound <NUM> from the mud source <NUM> to the spray gun <NUM>. The joint compound <NUM> can be pumped at high pressures, in some examples, to enable the joint compound <NUM> to be sprayed or aerosolized. In some examples, high joint compound particle speeds can produce a smoother finish, which can be desirable in some examples.

The pressure, flow rate, piping system resistance and the like, can be tuned or controlled by the automated drywalling system <NUM> to change the speed and amount of joint compound <NUM> being delivered to the spray gun <NUM> and ejected from the nozzle <NUM> as a spray <NUM>. The automated drywalling system <NUM> can use any suitable actuator (e.g., a servo, solenoid, air cylinder, linear actuator, or any combination of these) to open and close the nozzle <NUM> of the spray gun <NUM>. As shown in the example of <FIG>, a manual spray gun <NUM> can be instrumented to use an electro-mechanical system <NUM> to pull the trigger <NUM> allowing the system <NUM> to control the timing of the joint compound delivery as well as the opening and closing of the nozzle <NUM>.

As shown in the example of <FIG>, an automatic spray gun <NUM> can also be used and controlled by the system <NUM> directly. The robotic arm <NUM> and end effector <NUM> and/or base unit <NUM> can thereby be used to spray the joint compound <NUM> as a spray <NUM> onto surfaces of drywall pieces <NUM> and/or seams <NUM> defined by one or more drywall pieces <NUM>. The joint compound <NUM> can be sprayed before and/or after applying joint tape <NUM>. The automated drywalling system <NUM> can use a mesh or porous tape <NUM> in some examples to allow the joint compound <NUM> to be sprayed through the joint tape <NUM> to fill a gap under the joint tape <NUM> (e.g., a seam <NUM> or the like).

The spray gun <NUM> can use a variety of suitable nozzles <NUM> including fan shape, bell shape, or the like. The system <NUM> can also use a tunable spray gun <NUM> that can control the shape of the nozzle <NUM>. The shape of the mud spray <NUM> may be controlled in some examples by physically changing the shape of the nozzle <NUM>. The shape of the mud spray <NUM> can also be controlled using air streams, or the like which can act on the mud spray <NUM>.

In some embodiments, a cassette with different nozzles <NUM> can be installed on the spray gun <NUM> allowing the automated drywalling system <NUM> to select a desired nozzle <NUM> to control the shape of the spray <NUM>. A fan shape can also be tuned by using a set of sliding mechanisms to set the fan width and opening of the nozzle <NUM>. The diameter of a bell may also be tuned by a sliding cone with expanding orifice size. The robotic arm <NUM> and/or base unit <NUM> can also be used to move the nozzle <NUM> closer or farther away from a target surface resulting in a narrower or wider fan or bell spray pattern respectively. The system <NUM> can utilize an array or series of nozzles <NUM> to spray the coating over a larger surface. The nozzles <NUM> can be individually controlled and tuned or such nozzles <NUM> can be controlled as a unit.

A series of tests can be performed to establish the characteristics of a pattern of mud spray <NUM> delivered by a nozzle <NUM>. In one embodiment, one or more vision system <NUM>, <NUM> can be used to characterize a pattern of mud spray <NUM> and provide feedback for tuning parameters including tool parameters related to a nozzle <NUM>, spray gun <NUM>, mud source <NUM>, or the like, as discussed herein. Another embodiment can utilize an array of sensors (e.g., piezo sensors or other force sensors) on a test board which can be used to measure the force applied by the pattern of mud spray <NUM> as it hits the sensors. The force pattern can be used to estimate a profile of the pattern of mud spray <NUM> as it is hitting the surface. The feedback from these sensors may be used to tune the profile of one or more spray nozzles <NUM>, spray gun <NUM>, mud source <NUM>, or the like.

The automated drywalling system <NUM> can include a mixer, pump and the like that can deliver mixed joint compound <NUM> to the various tools including a spray gun <NUM>. Such a mixer, pump and the like can be part of a mud source <NUM> disposed at the base unit <NUM> or disposed external to the system <NUM>. A mixer may utilize sensors to control a mixing ratio of water, slurry or dry compound, and any additives that enhance structure of the compound, color the compound, decrease setting or drying time, or the like. The mixer can control the mix ratio by measuring the mass, volume, density, or viscosity of the components or the mixture that defines joint compound <NUM> or portions thereof. The mixing system can utilize pre-mixed joint compound <NUM> and can add water and/or additives as desired.

The automated drywalling system <NUM> can also use a spray gun <NUM> that has been designed to mix the components of the compound at the nozzle <NUM>. For example <FIG> illustrates an example of an in-line nozzle <NUM> for mixing the joint compound <NUM>, water, and any additives at the application site. The nozzle <NUM> can be detachable in some examples to be cleaned or to be disposable.

In various embodiments, a nozzle <NUM> can deliver a controllable ratio of water, air, slurry or dry joint compound, as well as additives that modify the joint compound, including enhancing the structure of the joint compound, color the joint compound, or decrease or increase setting or drying time. Nozzles <NUM> as discussed herein can be used with any suitable type of mud, joint compound <NUM>, or other material that can be sprayed, including but not limited to hot mud, plaster, or other curing compounds that set and cannot be washed off with water.

Compound lines <NUM>, nozzle <NUM>, a pump, or the like, can be instrumented with sensors to measure flow rate, pressure and other desirable parameters. Pressure sensors can be used to monitor the pressure along a compound line <NUM> enabling the detection of changes in the pressure, flow rate, as well as the detection of clogs. In some examples, an orifice plate may be used to measure the flow rate through the system mud system in combination with a set of pressure sensors. Other flow rate sensors can include, but are not limited to a rotameter, spring and piston flow meter, ultrasonic flow meter, turbine meter, paddlewheel meter, variable area meter, positive displacement, vortex meter, pitot tube or differential pressure meters, or magnetic meters for conductive coatings. Detecting a change in flow, pressure in the mud line <NUM>, or reaction force at the end effector <NUM> (e.g. at a spray gun <NUM>) can be used to determine that a clog has occurred. The spray gun <NUM> can produce a reaction force when spraying so if that reaction force changes the system <NUM> can identify that the spray <NUM> has changed, which can be indicative of a clog or other issue.

A pattern of the mud spray <NUM> can also be monitored to detect clogs or wear of the nozzle <NUM>. For example, <FIG> illustrates an example embodiment 160M4 of a mudding end effector <NUM> that includes a spray pattern detection mechanism <NUM>, in which a vision system <NUM> can be used to monitor the pattern of mud spray <NUM> coming out of the nozzle <NUM> to detect clogs, nozzle wear, low pressure, or other problems with the spray gun <NUM> or related system such as mud lines <NUM>, mud source <NUM> or the like.

In some examples, the stream of mud spray <NUM> can be monitored or the pattern of mud spray on a target wall can be monitored. The stream of mud spray <NUM> and/or pattern of mud spray <NUM> can be monitored using vision sensors <NUM>, which can include any suitable vision system, including but not limited to thermal sensors, moisture sensors, capacitance sensors, or the like.

In one embodiment, a camera can be placed next to the stream of mud spray <NUM> so that the profile of the mud spray <NUM> is captured. Image processing can be used to identify when the shape of the stream of mud spray <NUM> has changed. In another embodiment a laser curtain may be placed across the stream of mud spray <NUM>, if the flow is interrupted along any part of the fan or bell the laser would complete its path and be detected by a sensor on the other side of the stream of mud spray <NUM>.

A mixer, pump, mud lines <NUM>, and nozzle <NUM>, and other suitable elements can be fitted with filters which can be used to catch debris or particles that may clog the nozzle <NUM> or mud lines <NUM>. The filters can be placed an inlet of the pump, outlet and inlet of the mixer, directly before the mud line <NUM>, directly before the nozzle <NUM>, or any point along or within the mudding system. The automated drywalling system <NUM> can monitor the pressure before and after the filters to detect when the filters need to be changed. Flow rate sensors can also be used to detect a clogged filter. The automated drywall system <NUM> can reverse its flow to clear clogs from the mud line <NUM>, nozzle <NUM>, filters, or other components.

The spray gun <NUM> or other mudding end effector <NUM> may also include a vacuum system <NUM>, spray guards, or the like, that can be used to minimize overspray and reduce the amount of excess joint compound <NUM> in the air. For example, <FIG> illustrates an example embodiment 160M5 of a mudding end effector <NUM> that comprises a vacuum system <NUM> that includes a vacuum hood <NUM> disposed around an end and nozzle <NUM> of a spray gun <NUM> to capture overspray. The vacuum hood <NUM> can surround the spray gun <NUM> and can include an adjustable vacuum setting. The vacuum hood <NUM> can be coupled to the vacuum line <NUM>, which is connected to the vacuum source <NUM> to provide a vacuum to the vacuum hood <NUM>.

<FIG> illustrates an example embodiment 160M6 of a mudding end effector <NUM> that comprises a spray guard <NUM> that partially extends about and past the face of the nozzle <NUM> of the spray gun <NUM>. In this example, the spray guard <NUM> is shown being generally triangular and fanning out from where the spray guard is coupled to the end effector <NUM>. In some examples, the spray guard <NUM> can be selectively deployed by the system <NUM> or a user to prevent overspray onto an undesired surface. The spray guard <NUM> can be deployed in various suitable ways, including but not limited to, via a servo, pneumatic cylinder, solenoid or other electromechanical actuator, which can rotate or otherwise deploy the spray guard <NUM> into place.

In various embodiments, a mudding end effector <NUM> can comprise one or both of a vacuum system <NUM> and spray guard <NUM> of various suitable configurations. The guard <NUM> and/or vacuum system <NUM> can be deployed when the automated drywall system <NUM> is spraying near another surface or a feature. The spray guards <NUM> and/or vacuum systems such as a vacuum hood <NUM> can be retracted using a linear actuator, solenoid, air cylinder, or other suitable electro-mechanical actuator. In some embodiments, a spray guard <NUM> can also be mounted on a rotary stage such that the spray guard <NUM> can be rotated into place next to the sprayer <NUM> by actuating the motor or servo. Accordingly, in some examples, the position of the spray guard about a circumference of the spray gun <NUM> can be selected by the system <NUM> and/or a user.

In some embodiments, joint compound <NUM> can be applied and/or smoothed by using a blade that is dragged over applied joint compound <NUM>. Such a blade can be part of an end effector <NUM> having a spray gun <NUM> or can be a separate end effector <NUM>. In some embodiments, a mudding end effector <NUM> can apply mud <NUM> and tape <NUM> at the same time for a layer, or can apply joint compound <NUM> over the tape <NUM> that has been previously applied. The shape, profile, and size of a mudding blade can be controlled to deliver a desired profile of compound <NUM>. Similarly, the pressure or force on the mudding blade can also be controlled to change the thickness and profile of the applied compound <NUM>, which can be based on data from the system <NUM> obtained from one or more vision system <NUM>, <NUM>, sensors <NUM>, <NUM>, <NUM>, or the like.

The automated drywall system <NUM> can also include a mudding end effector <NUM> that comprises a drywall flat box <NUM> to apply the joint compound <NUM> as illustrated in the example embodiment 160M7 of <FIG>. In various embodiments, the automated drywall system <NUM> can move the box <NUM> along the seam <NUM>. An actuator <NUM> can control the shape and/or position of a blade <NUM> to tune the profile of mud <NUM> applied on the seam <NUM>. Various tool parameters, including box opening size, blade size, blade shape, and the like, can be controlled to simulate different sized boxes that are used to create a profile that feathers or blends a defect created by the seam <NUM> over a large portion of the boards <NUM> to simulate flatness.

The end effector box <NUM> can be automatically fed using a mud pump and mud line <NUM>. The mudding end effector <NUM> may also include sensors <NUM> (e.g., proximity, force, contact sensors) to ensure that the box <NUM> is in contact with the drywall <NUM> during the application of joint compound <NUM>. Additionally, a vision system <NUM>, <NUM> of the end effector <NUM> or base unit <NUM> can also be used to ensure that the flat box <NUM> is in contact with the surface of the drywall <NUM> during application of joint compound <NUM>.

In some embodiments, a mudding end effector <NUM> can deliver joint compound <NUM> through a sprayer <NUM> and/or nozzle <NUM> and then utilize a physical blade, trowel, air blade, roller or any other type of forming mechanism to smooth and profile the compound <NUM>. The mudding end effector <NUM> can utilize surrounding surfaces as datums. For example, a roller, wheel, blade, or the like, can be pushed in contact with the datum surface for reference. These contact points can extend away from a mud application zone to enable the use of datums away from the defect or joint <NUM>. The mudding end effector <NUM> can control the position of the contact points such that the correct or optimal datum surface is used. The force and pressure on the contact points may also be controlled. Force may be directly measured or estimated by monitoring the deflection of the mounting structure.

The mud application or coating tools can be mounted in series with a structure that limits, sets, or controls the amount of force applied on a target surface. The structure can limit, set or control the normal force applied on the surface by the blades, rollers, trowels, and the like, and/or it can limit, set or control forces applied by the tools along the target surface as well as torques applied. Such blades or rollers can be mounted on an air bag, air shock, air cylinder, air bellows, with a fixed or variable pressure setting. The pressure and the normal area of the pressure vessel can set the amount of forces applied by the tool on the target surface. The blade or roller can also be mounted on a spring, tunable spring, shock, or the like, in order to set, limit or control the forces applied on the target surface. The forces may also be set, limited, or controlled using a pressure controlled hydraulic system including, but not limited to a cylinder, bellows, or reservoir. In one embodiment, a short-stroke low-mass end effector linear actuator mechanism can be used for fast tracking of surface contours and constant normal force. In embodiments with more than one blade or roller, the tools can be mounted on a single force limiting structure, or each head or multiple tools can be mounted on separate structures. Mounting the tools or group of tools on separate structures can allows for the applied forces and moments to be set, limited, or controlled separately.

Mudding or coating tools can include sensors <NUM> and/or a vision system <NUM> to ensure the desired orientation of the blades or rollers relative to the wall. For example, one application includes ensuring planarity of the tool to the wall; however, the mechanism may also set the blade or roller to a specific target angle relative to the surface. The planarity may be established by utilizing the vision system <NUM> to detect the plane of the surface and then match the tool position using the degrees of freedom of the system <NUM>. The planarity may also be established by utilizing one or more sensor <NUM> at the end effector <NUM> (e.g., a set of proximity, range, or contact sensors to establish the position of a tool head relative to a wall). Blade or roller orientation can be controlled directly by setting the joint angles of the robotic arm <NUM>, by a powered gimbal or joint at the end effector <NUM>, and/or by a passive gimbal that allows the tool to tip and tilt relative to the end of the robotic arm <NUM>. A passive gimbal can enable the contact tool to follow the plane of a target surface despite errors in the position of the system <NUM>.

In another embodiment, the position of the contact may be controlled through the active gimbal using feedback from one or more of sensors <NUM>, <NUM>, <NUM> and/or vision systems <NUM>, <NUM> that can establish the relative orientation between blades or rollers and surface. Powered or passive gimbals or end effector degrees of freedom can be encoded (e.g., via sensors <NUM>) such that the orientation of the tool and/or end effector <NUM> is known to the system <NUM>.

A mudding end effector <NUM> can also utilize outriggers such as rollers to use adjacent surfaces or raised edges as datums to guide the application of mud <NUM> and achieve accurate corners. These rollers may be instrumented with sensors <NUM> and/or a vision system <NUM> to measure or determine force, contact, proximity, or the like. Additionally, or alternatively, such rollers can passively make contact while the drywalling system <NUM> utilizes its sensors <NUM>, <NUM>, <NUM> (e.g., force and torque sensing) and/or vision systems <NUM>, <NUM> to maintain a pressure or force against the datum surface. The information obtained or determined about tool orientation relative to the portions of the end effector <NUM>, robotic arm <NUM> and/or base unit <NUM> can be used to alter the toolpath, tool parameters and/or other system configurations to ensure the coating automation system can carry out the process without running into limitations of the hardware.

In both passive and active embodiments, the angular position of a gimbal or other portion of an end effector <NUM> can be recorded (e.g., via sensors <NUM> or vision system <NUM>) to locate and establish the plane of the target surface. The angular position of the gimbal can be recorded using elements including, but not limited to encoders on the rotary axis, laser range finders, capacitance sensors, IMUs, an external vision system, sonar sensors, potentiometers, motor loads, or any combination of these.

The gimbal system may be tuned to minimize dynamic effects by using springs, dampers or a combination of these. In some embodiments with more than one blade or roller, all tools may be mounted on a single gimbal structure or each tool or groups of tools may be mounted on separate gimbals. Mounting the blades or rollers on separate gimbals can allows for tool surface planes to be set, limited, or controlled separately. Mud application tools can be mounted on a gimbal in series with a compliant system described above that limits, sets, or controls the force applied on the surface.

In some embodiments, a mudding end effector <NUM> can include elements including, but not limited to a heater, curing light, blower or a combination of these. For example, <FIG> illustrates an example embodiment 160M8 of a mudding end effector <NUM> that comprises a first blower <NUM> and a second blower <NUM>. The first blower can be configured to apply cool and/or dry air to joint compound <NUM> that has been applied to the drywall board <NUM> by the mudding end effector <NUM>. The second blower <NUM> can be configured to apply heat and/or dry air to a surface of drywall <NUM> on which joint compound <NUM> will be applied. As shown in <FIG>, the mudding end effector <NUM> can include a mud applicator <NUM> that can include a tracking knife <NUM> that can be used to profile the mud <NUM>. In various embodiments, preheating and drying the surface of drywall <NUM> on which mud <NUM> is being applied can improve the mud application process. Cooling and/or drying the applied mud <NUM> via the first blower <NUM> can be desirable to speed the drying/curing process of the mud <NUM> and can improve the finish of the mud <NUM>.

In various embodiments, elements including but not limited to a heater, fan, UV light, microwave emitter, or a combination of these elements can also be a separate part of the automated drywalling system <NUM>. These components can be mounted on an end effector <NUM>, a robotic arm <NUM>, mobile base <NUM>, positioning stage <NUM>, gantry, or the like, or can be static in the room and separate from the automated drywalling system <NUM>. A purpose of these components can be to speed up the curing, drying, or setting time of the compound <NUM>, but can also be used to prepare the surface for the application of tape <NUM> or mud <NUM>. An embodiment of the end effector <NUM> utilizes a heater that leads the mud application for preheating the surface of drywall <NUM> on which mud <NUM> will be applied by the mudding end effector <NUM>. The mud application point can be followed by a blower which can act over the applied mud <NUM>. The mudding end effector <NUM> can also utilize two heaters leading and following the joint compound application or utilize two fans or a combination of these. The tool parameters or settings on the fan, heaters, or lights may be determined by the planning system (e.g., by the control system <NUM>) using information from one or more of sensors <NUM>, <NUM>, <NUM> and/or vision systems <NUM>, <NUM>. For example, environmental sensors (e.g., temperature, humidity, and the like) and a prescribed joint compound composition and applied thickness can be used to determine tool parameters for environmental control tools or systems such heaters, coolers, blowers, or the like. In another example, the mudding end effector can comprise a thermal imaging camera to assess the temperature of the mud <NUM> and calculate the moisture content of the mud <NUM>. The automated drywalling system <NUM> can also have a humidity sensor, conductivity sensor and depth or thickness sensors such as laser range finders, sonar, radar, LIDAR, and the like. Toolpaths, tool parameters settings, mud composition, fan, heater, light settings, and the like can be adjusted in real-time based at least in part on the measurements, sensing or data obtained from such sensors or visions systems.

The automated drywalling system <NUM> can utilize additives such as plaster of paris to accelerate the setting time of a coating of joint compound <NUM>. An accelerant can be mixed into the joint compound <NUM> during preparation, added in at the nozzle <NUM>, applied to a coating of joint compound <NUM> after deposition, or any combination of these. The automated drywalling system <NUM> can utilize environmental information to decide the amount of accelerant to add and at what point in the process it should be introduced. In other words data from one or more vision system <NUM>, <NUM> and/or sensors <NUM>, <NUM>, <NUM> to automatically modify the parameters of the composition, preparation, and application of joint compound <NUM>. In some examples, accelerant may be sprayed on to a coating of joint compound <NUM> after the joint compound <NUM> has been applied onto the target surface.

The automated drywalling system <NUM> can utilize sensors (e.g., humidity or conductivity sensors) that are mounted on a surface of drywall <NUM> before mud application, which can provide for tracking of the moisture content of the surface of drywall <NUM> and/or joint compound <NUM> applied to the surface of drywall <NUM>. Such sensors can be mounted directly onto the target surface, may be embedded in a joint <NUM>, or can be mounted on a coupon that is covered at the beginning of the process with the same parameters. Such sensors can be connected to a wireless communication system to send signals/data to the automated drywalling system <NUM>. Moisture content and other information collected by such sensors can be used to control or adjust the settings on fans, blowers, heaters, curing lights, an HVAC system, or the like. The drying speed can also be used to adjust the composition of the mud compound <NUM>. Monitoring the moisture content can allow the system <NUM> to accurately estimate the time when the next drywalling process can begin (e.g., sanding, painting or the like).

The automated drywalling system <NUM> can also determine when the joint compound has set and dried by measuring the thermal conductivity of the covered seam <NUM>, using a vision system (such as a thermal imaging camera); using a sensor such as a thermometer (contact or non-contact), or by detecting differences in colors using a vision system (e.g., due to color changes that occur between wet and dry joint compound <NUM>. Various measurements can be used to infer the moisture content of joint compound <NUM> by comparing a determined temperature of the joint compound <NUM> to the surrounding materials such as a sheet of drywall <NUM>. For example, as water or other solvent evaporates from a mixture of joint compound <NUM>, the temperature of the joint compound <NUM> can be lower than that of the surrounding materials. Models of the joint compound drying process can also be used to estimate the time to dry or cure given a set of starting conditions and information about the environment. The environmental sensors and/or vision systems can be used in conjunction with an HVAC system or heater, air conditioner, fans, or the like, to control the room conditions at a worksite. The sensor readings can automatically trigger any of these systems or a combination to maintain the room at the desired conditions for quality, reduced drying time, or comfort of the operator.

In various embodiments, the automated drywall system <NUM> can use one or more vision system <NUM>, <NUM> and/or sensors <NUM>, <NUM>, <NUM> to establish a condition of a wall of drywall before and after compound application to determine appropriate toolpaths and/or tool parameters. The system <NUM> can use computer vision, structured lights, stereo cameras, images, lights and shadows, LIDAR, radar, sonar, point clouds or any combination of these to establish the conditions of a target surface. These conditions can include establishing the surface plane relative to a mud application tool or another surface, detecting high or low points, curvature, and defects. One or more of the vision system <NUM>, <NUM> can be used to create a topographical map of the surface to identify high and low spots. The map can be created after drywall <NUM> or other substrate has been hung. The map can also be an input from a board layout system that specifies the location and types of joints <NUM> and features in the room. The map can be updated by the one or more vision system <NUM>, <NUM> as the system <NUM> is moved or moves around the room. The system <NUM> can also utilize rollers, proximity sensors, contact sensors, profilometers, and the like, to measure the profile of the surface. The robotic arm <NUM>, end effector <NUM> and/or base unit <NUM> can be used to make contact with rollers or other mechanism on an encoded linear stage and then move these over the surface creating a topographical map. This can be done over joints or seams to determine the profile. The system <NUM> can then compute how the mud <NUM> should be applied and tapered to create a visually flat wall assembly.

To achieve the coating thickness on the drywall <NUM> or other substrate, the system <NUM> can optimize the delivery of the compound <NUM> to build up more compound <NUM> on low spots and less on high spots. The system <NUM> can also use information of the joint location to profile the mud delivery to account for the height variations typical of joints <NUM>. The end effector <NUM> can then be used to apply a specific profile of joint compound <NUM> to the wall. This can be done by controlling the profile of the sprayer <NUM>, the shape and size of a troweling blade, the distance between the end effector <NUM> and board of drywall <NUM>, the flow rate of joint compound <NUM>, the tool speed, the number of passes over a given spot, or the consistency of the joint compound <NUM>. The robotic arm <NUM> and/or end effector <NUM> can utilize force control to apply the pressure required to deliver a desired amount of joint compound <NUM> or to achieve a desired surface texture or roughness.

A thickness measurement can also be used to determine the amount of compound <NUM> that is to be delivered to a given spot. The system <NUM> can also tune the profile of the delivered mud <NUM> to account for overlap of the subsequent application. The mud thickness at the edges can be reduced or feathered such that the overlap region achieves the final desired thickness. This approach can also be used to increase overlap error tolerance at transition points between robot workspaces. The automated system <NUM> can utilize the information about the room, compound mixture and desired compound profile to determine the application profile desired to account for shrinkage of the joint compound <NUM>. The system <NUM> can also use shrinkage models with environmental information obtained from sensors or vision systems to anticipate the shrinkage of the joint compound 630as it dries. The delivered profile can account for shrinkage by increasing thickness of compound <NUM> applied such that the final post-shrinkage profile is the desired profile to achieve a visually flat wall. Compound mixture definition can include real-time automatic adjustments of gypsum, plaster of paris, and water content for optimal results given environmental conditions (determined based on data from sensors and/or visions systems), and layer finish requirements.

The system <NUM> can be instrumented with vision systems <NUM>, <NUM> and/or sensors <NUM>, <NUM>, <NUM> that can be used to improve operation and ensure quality. During compound application the system <NUM> can use sensors <NUM> (e.g., force and torque sensors) mounted directly on the end effector <NUM>, or sensors <NUM> on the robotic arm <NUM>, and/or force and torque estimates determined by sensors <NUM> of robotic joints of the robotic arm <NUM> to apply a desired force during troweling or taping. The vision systems <NUM>, <NUM> and/or sensors <NUM>, <NUM>, <NUM> can monitor force normal to a blade or rollers or on multiple axes including torque measurements and six-axis sensing. The force sensing can be used to control the force or pressure applied by one or more tool of an end effector <NUM>. A minimum force or contact readings can also be used to ensure contact is made before the joint compound <NUM> is allowed to flow, and force below a certain threshold or loss of contact can trigger the stop of joint compound flow. The automated drywalling system <NUM> can use the force information to operate in force control, where the motions and speeds of the system <NUM> are driven to ensure a given force is applied in the desired directions. Similarly, force sensing can be used to detect contact with an object, obstacle, or intersecting wall or ceiling. By monitoring forces and torque on various portions of the robotic arm <NUM>, base unit <NUM> and/or end effectors <NUM>, the system <NUM> can detect that it has made contact with the adjacent wall or ceiling and alter the toolpath accordingly. The measurements can also be used to detect accidental contact and trigger a safety operation such as stopping the system <NUM> or retracting away from contact point. The system <NUM>, including the end effector <NUM> can also use sensors (e.g., contact or proximity sensors) and/or visions sensors to detect that the end effector <NUM> is touching the surface, obstacle, object, or worker, as well as detect the distance to an adjacent surface or contact with that surface. The force, contact, displacement, or proximity sensors can be mounted on outriggers from the end effector <NUM> to sense obstacles, objects, or adjacent surfaces ahead of the end effector <NUM>. The system <NUM> can detect, follow, and use adjacent walls as datums to guide coating application and achieve accurate corners. For example, in some embodiments, the end effector <NUM> can comprise a guiding element configured to engage a target surface, adjacent walls, or the like, to allow the end effector <NUM> to be guided in mudding the target surface. For example, such a guiding element can include an arm extending from the end effector <NUM>, with the arm having a roller at the end of the arm configured to engage the target surface or portion of a wall assembly as a mudding guide.

The base unit <NUM>, robotic arm <NUM> and/or end effector <NUM> can utilize multiple control strategies to complete various tasks. Position control can be used to command the system <NUM> to follow a trajectory given speed, acceleration, and jerk constraints. The system <NUM> can be controlled at the joint level by giving commands to the joints to achieve the desired robot state and tool position, or the control can be done at a higher level allowing a user or program to control end effector position and orientation. The system <NUM> can be controlled in task space where the system <NUM> controls a tool relative to the task. This approach can focus on achieving a desired tool position, orientation, speed, or the like, relative to the target surface rather than on each joint reaching its target goal. The system <NUM> can utilize force control to control the force applied to the target surface, an obstacle, adjacent surfaces, objects and so on. The applied force can be controlled in a single or multiple axes. Hybrid control modes can also be used. For example the system <NUM> can be commanded to achieve a given position as long as a given force is not exceeded.

The one or both of the vision system <NUM>, <NUM> can be used to capture where and how the joint compound <NUM> has been applied. By monitoring the spray pattern applied on the wall the system <NUM> can detect clogs, nozzle or blade wear, or other problems. In one example, a thermal camera can be used to detect the applied compound <NUM>, which can be at a different temperature than the target material. The compound's temperature can be controlled to facilitate detection. Monitoring the compound temperature can also give information on the moisture content of the joint compound <NUM>. The joint compound <NUM> can have a prescribed coloring or additives to create contrast between the target surface and the compound <NUM> facilitating the detection of areas that have been covered by the compound <NUM>. The color can change as the compound <NUM> dries as well as after it has been sanded. The system <NUM> can also apply compounds <NUM> in layers with different colors in different layers of compound <NUM> to facilitate detecting how much compound <NUM> has been removed during application or sanding of joint compound <NUM>. Sensing such as capacitance, radar, resistance, humidity, conductivity, sonar measurements, or any combination of these can also be used to establish the thickness of the compound <NUM>. Lights can be mounted on the system <NUM> or externally to illuminate the surface enabling the detection of coated surfaces, high and low points, tool marks, coating roughness, orange peel, and defects using one or both of vision systems <NUM>, <NUM>.

The system <NUM> can monitor the coverage achieved by the end effector <NUM> and update tool paths and tool parameters to ensure the desired coating profile is being applied. For example, the system <NUM> can dynamically tune a sprayer fan and/or bell until the spray pattern matches the desired shape, thickness, size. The system <NUM> can also move the sprayer <NUM> closer or farther away from the target surface to change the spray pattern. The system <NUM> can also tune the material flow rate, pressure, spray tool speed, or the like, to achieve a desired thickness. The toolpaths and/or tool parameters can also be updated to ensure that the correct overlap is being achieved.

The system <NUM> can also utilize a feedback mechanism for communicating contact, forces, gimbal displacement information, tool orientation, motor loads, humidity and temperature readings, measurements of the applied compound <NUM>, to system <NUM> (e.g., to the control system <NUM>) for the purpose of real time updating of the tool paths and tool parameters for improving finish of joint compound <NUM>. The system <NUM> can use tool position and orientation, captured surface conditions and models to update the robotic toolpaths to ensure that a desired position and/or contact is maintained during application of joint compound <NUM>.

The system <NUM> can also determine areas that need another application of mud <NUM>, rework using automated drywalling system <NUM>, or rework to be done manually by the user. The user can also use a user interface of the system <NUM> to indicate areas that the user has identified as needing rework or need to be coated again. The system <NUM> can use this input along with other information about the previous work to create a new toolpath. Both user and system feedback can be fed into a machine learning algorithm to create a better model for coating future surfaces given a set of initial conditions.

The automated drywalling system <NUM> can utilize a user interface to enable the worker to control, program, debug, plan, and setup the system <NUM>. The user interface can be used to give the user information of all the steps that must be taken to setup the system <NUM>. Each step can be checked off when complete and the user can request more information on each step. The workspace of the system <NUM> can be shown overlaid on a camera feed or projected onto the target surface to help the user position the end effector <NUM>, robotic arm <NUM> and/or mobile base unit <NUM>. The workspace can be projected using lights or lasers. The system <NUM> can also automatically perform certain steps and the user interface can report the progress of each step, as well as give guidance to the steps the user can follow to perform a task. The user interface can be used to setup the system <NUM> and run any calibration routines required. The interface can also be used to plan a job including detecting wall, user definition of path parameters or path itself, auto generation of the tool path, user input of tool parameters, and automatically optimized tool parameters given a set of user inputs.

The user interface can be a graphical user interface and include a 2D or 3D representation of the worksite and workspace. The representation can include camera feeds as well as computer models and reconstructions created using sensor data. The interface can overlay paths, quality visuals, progress, robot model, or the like, over camera or workspace models. As the task is completed the path can be highlighted in different colors or with different style lines to indicate completion, quality achieved, problem areas among others.

Any problems, issues, or bugs can be reported in the user interface. Lights on the end effector <NUM>, mobile base <NUM> and/or robotic arm <NUM> as well as sounds can also be used to indicate problems, movement of the end effector <NUM>, base unit <NUM> and/or robotic arm <NUM>; that work is in progress; that the system <NUM> is on or off; that toolpath is running or paused, that the system <NUM> needs attention or refill of materials; and any other indicators of the system state. The user interface can also display information on the progress, task and tool parameters, and quality metrics of the task being performed. Environmental conditions can also be displayed and recorded by the interface. The system <NUM> can indicate to the user what steps to take to correct or improve conditions including air quality, temperature and humidity. If the system <NUM> detects unsuitable or unsafe conditions it can display a message warning the user and providing guidance on next steps. The system <NUM> can use an optimization to find what parameters could be used to improve the process including reducing work time, increasing quality, and minimizing material usage among others. The user interface can also create reports on the tasks executed, quality metrics, environmental conditions, completion, and performance logs. Information can include robot workspace, tool paths, progress, sequence of approach, application rates and thicknesses, spray pressures and flow rates, forces applied by the tool, coverage record, path speed, tracking error, time to complete the task, tool time, setup time, vacuum waste material collected, cleaning time. The user interface can also display on filter conditions, and the system <NUM> can trigger an alarm or instruction when the filter needs to be replaced or cleaned.

The user can interface with the system <NUM> using a computer, tablet, touch screen, mobile device, pendant, joystick, controller, or buttons directly on the system <NUM>. The worker can also position and train the robotic arm <NUM> and/or end effector <NUM> by directly moving joints of the robotic arm <NUM> or end effector <NUM>. The user interface, controller, or buttons can be used to record positions as well as change the control mode and task.

An augmented reality system can be used to show the worker a toolpath plan generated by the system <NUM>, instructions, original BIM or plan, or a combination of these. The augmented reality can be displayed using a headset, smart goggles, projections, or the like. The worker can be shown areas that require manual coating application. The user can also overlay the location of studs, framing, pipes, ducts, electrical system behind the board to facilitate compound application. Mudding tools, both manual and automated can be tracked in the map using tags, IMUs, or other sensors and a warning can be given to the operator if an attempt is made to apply compound <NUM> in an erroneous position or under the wrong tool settings. The system <NUM> or tools can also utilize radar, sonar, thermal imaging to establish what is behind the substrate.

The automated drywalling system <NUM> can also produce a visualization, paths, or instructions or a combination of these to guide the user in completing manual work. The visualization can include 2D or 3D maps marking the areas of work with labels. The visualization system can also include a projection of the plan onto the target surface this can be done with a laser system, projector or through augmented reality headset or goggles worn by the user.

The coating time, pressure, material flow rate, mud characteristics, and clogs can be tracked to inform when a nozzle <NUM> or blade <NUM> should be cleaned or changed. For example, <FIG> illustrates an example embodiment 160M9 of a mudding end effector <NUM>, which comprises a nozzle cassette system <NUM> where a cassette of nozzles <NUM> is attached to the end of the spray gun <NUM>. The cassette system <NUM> can be rotated (e.g., via an electromechanical system) to deliver a nozzle <NUM> to the spray gun <NUM> for use.

<FIG> illustrates another example embodiment 160M10 of a mudding end effector <NUM> that comprises of a nozzle rotating system <NUM> that can be part of a spray gun <NUM>. In this example, the system <NUM> can utilize an actuator assembly <NUM> (e.g., a servo or other electromechanical actuator) to rotate (e.g., <NUM> degrees) a portion <NUM> of the nozzle <NUM> allowing for joint compound <NUM> to go through the nozzle portion <NUM> in reverse helping clear out clogs.

In various embodiments, nozzle or blade wear models can also take as an input the type and characteristics of joint compound <NUM> applied and the conditions under which such compound <NUM> was applied. One or more vision system <NUM>, <NUM> of the system <NUM> can be used to detect finish, tool pattern and establish if the nozzle <NUM> or blade <NUM> needs to be changed, rotated, cleaned or otherwise modified. A user interface can display the wear on the nozzle <NUM> or blade <NUM> and alert the user when these need to be changed. A mudding end effector <NUM> can also include a mechanism to automatically replace or clean the nozzle <NUM> or portions thereof. One embodiment (e.g., <FIG>) can use a cassette with replacement nozzles <NUM> that can be rotated into place. The sprayer <NUM> can also have a mechanism <NUM> to rotate the nozzle or portion thereof (e.g. a tip or feeding tube) to clear a clog (e.g., <FIG>). The nozzle clearing or replacement can be run automatically by the system <NUM> without any human intervention or as a collaboration between the system <NUM> and the user.

The system <NUM> can generate reports and interface with other software platforms including BIM packages. Reports can be created that can be used for inspection and certification. A report can be customized to provide the information required to pass a standard, test, or certification. The reporting system can also provide a live update of the current task progress and live camera feed. This information can be used to help track asset performance and work progression. The data can be reported to a BIM system or other software to facilitate planning of other trades, next steps, or schedule inspections or other tasks. The reports can include full maps of the joint compound <NUM> applied and tool and path parameters utilized to complete the task. Further images or video can be recorded to facilitate quality checks or for tracking of issues. The system <NUM> can record parameters used to complete the task which can be fed to a machine learning software to enable the system <NUM> to learn from past work. The reports can also be used to optimize workflow and scheduling. The system's optimization function can be updated to meet the desired needs including minimizing task time, completion of the task in a part of the worksite to allow other trades to come in, minimizing cost, optimal use of assets and workforce, among others. The system's reports can also include information on environmental conditions and how the process was changed given the conditions.

The system <NUM> can create a report that shows the process parameters that were used to cover the surface as well as the order of operations. The report can include BIM, 3D and 2D maps or plans, images, video. The maps provided by the system <NUM> can be used to facilitate repairs and maintenance by providing the customer with the location of components behind the wall as well as the location of seams to facilitate the removal of panels or boards.

The updated room models that reflect the as built conditions and measurements can be exported for use in sanding the walls or for certification of quality at delivery. A complete map of the thickness of the compound applied with or without shrinking can be fed into the system <NUM> or a separate automated sanding system which can plan tool paths and parameters desired to achieve the desired finish by sanding. The system <NUM> can work in conjunction with a larger system that plans the full process from mapping a room, to cutting and hanging the drywall to finishing and painting of the surfaces. This system <NUM> can be used for coating surfaces with any suitable material, including but not limited to joint compound <NUM>, plaster, stucco, cement, paint, polymer coating, lacquers, varnishes, or any combination of these. The system <NUM> can apply the coatings on any suitable substrate, including but not limited to drywall, boards, lath, mesh, or other substrates. The system <NUM> can also be used to apply other coverings such as wallpaper, polymer films, or the like.

Claim 1:
An automated system (<NUM>) comprising:
a robotic arm (<NUM>) that extends between a base end (<NUM>) and a distal end (<NUM>);
an end effector (<NUM>) coupled at the distal end of the robotic arm (<NUM>); and
a computing device executing a computational planner,
characterised in that:
the automated system is an automated drywalling system;
the end effector (<NUM>) is a mudding end effector which is configured to apply joint compound or plaster to a target surface; and
the computing device generates instructions for driving the mudding end effector (<NUM>) and robotic arm (<NUM>) to perform at least one mudding task that includes applying joint compound or plaster, via the mudding the end effector (<NUM>), to one or more joints between a plurality of drywall pieces (<NUM>), the generating based at least in part on obtained target surface data; and
drives the end effector (<NUM>) and robotic arm (<NUM>) to perform the at least one mudding task.