Patent ID: 12221329

DESCRIPTION OF THE EMBODIMENTS

The following description of embodiments of the invention is not intended to limit the invention to these embodiments but rather to enable a person skilled in the art to make and use this invention. Variations, configurations, implementations, example implementations, and examples described herein are optional and are not exclusive to the variations, configurations, implementations, example implementations, and examples they describe. The invention described herein can include any and all permutations of these variations, configurations, implementations, example implementations, and examples.

1. Method

As shown inFIGS.1and2, a method S100includes, at an autonomous off-road vehicle including a set of forks113and a robotic arm114: autonomously loading an initial container (e.g., a pallet), containing an initial set of objects (e.g., solar panels), onto the set of forks113in Block S110; and autonomously navigating across a non-uniform outdoor terrain to locate the robotic arm114and the initial container proximal an initial install location on an initial structure in Block S112.

The method S100further includes, prior to retrieving an initial object142from the initial set of objects in the initial container via the robotic arm114: accessing an initial image from an primary optical sensor120defining an initial field of view intersecting the initial install location in Block S120; detecting an initial set of install features152at the initial install location based on the initial image in Block S122; calculating an initial gross install pose of the initial object142that locates the initial object142proximal the initial install location and offset from the initial set of install features152by an initial target offset distance in Block S130; and calculating an initial install path navigable by the robotic arm114to retrieve the initial object142from the initial set of objects in the initial container and to maneuver the initial object142to the initial gross install pose in Block S132.

The method S100also includes, in Block S140, defining an initial keep-in boundary of the initial object142maintained by the robotic arm114. The initial keep-in boundary is arranged proximal the initial install location and encompassing the initial gross install pose.

The method S100further includes, at the autonomous off-road vehicle: autonomously navigating the robotic arm114according to the initial install path to retrieve the initial object142from the initial set of objects in the initial container and to locate the initial object142in the initial gross install pose in Block S150; and, following completion of the initial install path by the robotic arm114, entering a manual manipulation mode in Block S152. The method S100also includes, in the manual manipulation mode: detecting an initial series of forces applied to a distal end of the robotic arm114; and navigating the initial object142in directions of the initial series of forces while supporting a weight of the initial object142and maintaining the initial object142fully within the initial keep-in boundary.

The method S100further includes, in Block S160following installation of the initial object142at the initial install location, autonomously navigating across the non-uniform outdoor terrain to locate the robotic arm114and the initial container proximal a secondary install location on the initial structure.

2. Applications

Generally, Blocks of the method S100can be executed by a robotic system100in cooperation with an off-road vehicle (e.g., skid steer loader) to: maneuver a primary container140(e.g., a pallet), containing a set of objects (e.g., solar panels, solar trackers), loaded onto the off-road vehicle (e.g., loaded onto a set of forks113) across a non-uniform outdoor terrain (e.g., uneven muddy terrain); and support an operator during installation of these objects at a structure150arranged across the outdoor terrain.

More specifically, the system100can: maneuver across the non-uniform outdoor environment to locate a robotic arm114—coupled to the off-road vehicle—proximal an initial install location, such as proximal a structure150arranged across the outdoor terrain; retrieve an initial object142from the primary container140(e.g., pallet of solar panels, pallet of solar trackers) loaded onto the off-road vehicle; locate this initial object142at a gross install pose proximal the initial install location at the structure150(e.g., solar panel rails, piles). Furthermore, the system100can then support a weight of the initial object142while maintaining the initial object142within a boundary proximal the initial install location to prevent collisions of the initial object142to adjacent objects installed on the structure150and prevent movement of the robotic arm114from application of local wind loads at the outdoor environment during installation of the initial object142to the initial install location by an operator.

The system100can include an attachment assembly110including: an attachment housing112; a set of forks113(e.g., pallet fork, utility forks) extending from the attachment housing112and configured to receive loading of a primary container140(e.g., pallet of solar panels); and a robotic arm114coupled to the attachment housing112and configured to retrieve objects from the primary container140(e.g., pallet of solar panels) loaded onto the set of forks113. In one example, the system100can include the attachment assembly110integrated directly onto the off-road vehicle (e.g., skid steer loader) or the attachment assembly110configured to transiently couple to an existing off-road vehicle (e.g., skid steer loader). Additionally, the system100can also include a suite of sensors—such as optical sensors (e.g., color cameras, infrared cameras), proximity sensors (e.g., ultrasonic sensors, light detection and ranging sensors), position modules (e.g., global position modules), force sensors124—integrated into the attachment assembly110and/or arranged on the off-road vehicle separate from the attachment assembly110.

In one example, the system100can: navigate the off-road vehicle (e.g., skid steer loader) across a non-uniform outdoor terrain to deliver a primary container140(e.g., pallet of solar panels) proximal an initial install location at the outdoor environment; and access an image from a primary optical sensor120—such as coupled to the robotic arm114and/or coupled to a chassis of the off-road vehicle-defining a field of view intersecting the initial install location. The system100can then implement computer vision techniques (e.g., object detection, feature extraction) to: detect a set of initial install features (e.g., brackets) on the initial install location represented in the image; and calculate a gross install pose proximal the initial install location defining a target offset distance (e.g., six inches) between an initial object142and the set of initial install features.

Accordingly, the system100can then: implement path planning techniques (e.g., graph-based path planning, sampling-based path planning) to calculate an install path to maneuver the initial object142to the gross install pose; and trigger the robotic arm114to retrieve the initial object142from the primary container140(e.g., pallet of solar panels) and to maneuver the initial object142along the install path to locate the initial object142at the gross install pose. The system100can then, responsive to application of forces at a distal end of the robotic arm114(e.g., control handle118coupled to end effector116), trigger operation of the robotic arm114in the manual manipulation mode to enable the operator to manually control the initial object142(e.g., solar panel), during installation of the initial object142to the initial install location by the operator, while supporting the initial object142(e.g., solar panel).

Therefore, the system100can: deliver a primary container140according to a site plan (e.g., solar farm site plan) across a non-uniform outdoor terrain to deliver the primary container140to a designated installation zone; and support an operator by enabling manual control (e.g., lift, translate) of these objects retrieved from the primary container140—loaded onto the off-road vehicle—during installation onto an initial install location.

2.1 Applications: Keep-In Boundary

The system100can further define a keep-in boundary (e.g., three-dimensional boundary) of the initial object142maintained proximal the initial install location by the robotic arm114. More specifically, the system100can define the keep-in boundary to: define positional tolerances of the initial object142proximal the install location; prevent collisions of the initial object142to structural elements (e.g., installation features) proximal the install features at the install location at the structure150and/or adjacent objects (e.g., adjacent solar panels) installed at the structure150; prevent sudden movement of the initial object142such as, from wind loads applied to the robotic arm114at the outdoor environment; and enable an operator to manually navigate the initial object142within the keep-in boundary during installation of the initial object142at the install location by the operator.

In one example the system100can: access an image from a primary optical sensor120defining a field of view intersecting the installation location; and implement computer vision techniques to, detect a secondary object144adjacent the install location and derive an install plane (e.g., rectangular plane) at the install location. The system100can then define a keep-in boundary (e.g., three-dimensional boundary) that:encompasses the gross install pose; defines an interstice between a lateral side of the secondary object144and a periphery of the keep-in boundary; and extending below the install plane at the install location. During installation of the initial object142at the install location, local winds at the outdoor environment can apply wind loads to the initial object142supported by the robotic arm114proximal the install location which can result in sudden movements (e.g., jerk) of the initial object142away from the install location and collisions with proximal structural elements at the install location. Accordingly, the system100can apply positional tolerances and manipulation resistances, such as by applying braking forces to joints of the robotic arm114and/or locking joints of the to the robotic arm114, to prevent the initial object142—supported on the robotic arm114—from breaching the keep-in boundary.

Therefore, during an installation routine of an initial object142to an install location, the system100retain the initial object142proximal an install location and entirely within a keep-in boundary regardless of environmental wind conditions proximal the install location.

3. System

In one implementation, the system100can include an attachment assembly110including: an attachment housing112(or “chassis”) coupled (e.g., via welding, brackets) to an off-road vehicle (e.g., skid steer loader); a set of forks113(e.g., pallet fork, utility forks) extending from the attachment housing112and configured to receive loading of a primary container140(e.g., pallet of solar panels); a robotic arm114(e.g., articulated robotic arm114) coupled to the attachment housing112; an end effector116(e.g., vacuum gripper) coupled to a distal end of the robotic arm114and configured to retrieve objects (e.g., solar panels) from the primary container140(e.g., pallet of solar panels) loaded onto the set of forks113; a power source (e.g., generator) configured to supply power to the robotic arm114, the end effector116, and the set of forks113; and a controller130configured to execute controls to maneuver the robotic arm114, such as by retrieving an initial object142from the primary container140(e.g., pallet of solar panels) and locating the initial object142proximal an initial install location.

In one example, the system100can include the attachment assembly110integrated into a fully autonomous off-road vehicle (e.g., skid steer loader). In this example, the autonomous off-road vehicle can autonomously: load a primary container140onto the set of forks113; and navigate across the non-uniform outdoor terrain to locate the primary container140proximal an initial install location.

In another example, the attachment assembly110is configured to couple to a manually operated off-road vehicle. In this example, an operator can manually control the off-road vehicle to: load a primary container140onto the set of forks113; and navigate across the non-uniform outdoor terrain to locate the primary container140proximal the initial install location.

Therefore, the system100can: navigate across a non-uniform outdoor terrain to deliver a primary container140(e.g., pallet of solar panels), such as to designated installation zones of a site plan; via the robotic arm114; retrieve an initial object142from the primary container140loaded on the set of forks113; and maneuver the initial object142(i.e., about the outdoor environment) to locate the initial object142proximal an initial install location.

3.1 Sensors

In one implementation, the system100can include a suite of sensors, such as optical sensors (e.g., color cameras, infrared cameras), proximity sensors (e.g., ultrasonic sensors, light detection and ranging sensors), position modules (e.g., global position modules), force sensors124, etc. The suite of sensors can be coupled to the attachment assembly110and/or arranged about a chassis of the off-road vehicle (e.g., skid-steer loader). For example, the system100can include: a primary optical sensor120integrated into the robotic arm114(e.g., coupled to an end effector116) and defining a primary field of view of an initial install location; and secondary optical sensor122arranged on an exterior (e.g., chassis) of the off-road vehicle and defining a secondary field of view, different from the primary field of view, of the initial install location. During maneuver of an initial object142by the robotic arm114, the initial object142can obfuscate the primary field of view of the initial install location thereby degrading image data capture during installation of the initial object142onto the initial install location. Thus, the system100can leverage image data from the secondary optical sensor122to redress the degraded image data captured from the primary optical sensor120.

3.2 Control Handle

In one implementation, the system100can operate the robotic arm114in a manual manipulation mode (or “zero-gravity mode”) to permit the operator to manually maneuver the initial object142—grasped by the end effector116—during installation of the initial object142onto the initial install location by the operator. In this implementation, the system100can further include a control handle118(or “teach handle”) extending from the end effector116; and a force sensor124coupled to the control handle118.

During installation of the initial object142at the initial install location, the operator can apply a force (e.g., push, pull) at the control handle118to transition the robotic arm114into the manual manipulation mode. More specifically, the system100can: access a sequence of force values from the force sensor124coupled to the control handle118; detect the sequence of force values exceeding (e.g., greater than five pounds-force) a threshold force value (e.g., 10 pounds-force); and, in response to the sequence of force values exceeding the threshold force value, trigger the robotic arm114in the manual manipulation mode by releasing joints of the robotic arm114to permit manual motion control of the initial object142by the operator while supporting the initial object142.

Alternatively, the operator can apply a force (e.g., push, pull) directly to the initial object142supported on the robotic arm114to transition the robotic arm114into the manual manipulation mode. In this implementation, the system100can: access a sequence of force values from a force sensor124integrated into the end effector116of the robotic arm114; detect the sequence of force values exceeding (e.g., greater than five pounds-force) a threshold force value (e.g., 10 pounds-force); and, in response to the sequence of force values exceeding the threshold force value, trigger the robotic arm114in the manual manipulation mode by releasing joints of the robotic arm114to permit manual motion control of the initial object142by the operator while supporting the initial object142. Thus, the operator can navigate the initial object142by: applying forces directly to the initial object142supported on the robotic arm114; and/or applying forces to the control handle118coupled to the robotic arm114.

Therefore, the system100can support an operator by permitting the operator to manually control the initial object142—grasped by the end effector116—during installation of an initial object142to an initial install location.

4. Pallet Loading

Block S110of the method S100recites, at an autonomous off-road vehicle including a set of forks113and a robotic arm114, autonomously loading an initial container, containing an initial set of objects, onto the set of forks113in Block S110. Generally, the system100can load a primary container140(e.g., pallet of solar panels) onto the set of forks113in preparation for delivery of the primary container140to a designated installation zone such as, defined in a site map.

In one implementation, the system100can autonomously load a primary container140onto the set of forks113. In this implementation, the system100can: access a site plan (e.g., solar farm site plan) representing a map of a non-uniform outdoor terrain; identify a primary location of a loading zone defined in the site plan; and autonomously navigate the off-road vehicle—and therefore the attachment assembly110—across the non-uniform outdoor terrain to the loading zone. Accordingly, the system100can then: identify (e.g., via tags) a primary container140(e.g., pallet of solar panels) at the loading zone corresponding to an initial installation zone located at the non-uniform outdoor terrain; and implement closed-loop controls to autonomously load the primary container140onto the set of forks113.

In another implementation, an operator can manually control the system100to maneuver the off-road vehicle across the non-uniform outdoor terrain to the loading zone. The operator can then, manipulate controls (e.g., fork controls)—such as at a loading interface (e.g., joystick, buttons)—arranged at or integrated into the off-road vehicle to load the primary container140at the loading zone onto the set of forks113.

In one example, prior to loading the primary container140, the system100can: autonomously navigate the off-road vehicle to a loading zone at the non-uniform outdoor terrain; access an image from a secondary optical sensor122arranged on the autonomous off-road vehicle and defining a field of view intersecting the primary container140arranged at the loading zone; implement computer vision techniques to detect a set of loading features at the primary container140in the image; and calculate a loading path navigable by the autonomous off-road vehicle to locate the autonomous off-road vehicle proximal the primary container140and to couple the set of forks113to the set of loading features. The system100can then autonomously navigate the autonomous off-road vehicle according to the loading path to load the primary container140, containing the set of objects, onto the set of forks113.

Therefore, prior to an installation routine, the system100can autonomously load a primary container140(e.g., pallet of solar panels) in preparation to deliver the primary container140to a designated installation zone of a site plan representing a non-uniform outdoor terrain.

5. Outdoor Navigation+Installation Location

Block S112of the method S100recites, at an autonomous off-road vehicle including a set of forks113and a robotic arm114, autonomously loading an initial container, autonomously navigating across a non-uniform outdoor terrain to locate the robotic arm114and the initial container proximal an initial install location on an initial structure.

Generally, the system100can: following loading of a primary container140(e.g., pallet of solar panels) onto a set of forks113at a loading zone, navigate the off-road vehicle—and therefore the primary container140—across a non-uniform outdoor terrain to deliver the primary container140to an initial installation zone at an outdoor environment; and maneuver the robotic arm114proximal an initial install location at a structure150arranged at the initial installation zone.

In one implementation, the system100can implement a path planning model (e.g., artificial intelligence path planning model) to identify a target sequence of install locations across the outdoor environment, such as based on: a site plan (e.g., solar farm site plan) representing the outdoor environment; availability of materials (e.g., solar panels) scheduled for installation across the outdoor environment; and current and/or predicted weather conditions of the outdoor environment.

In one implementation, as described above, the system100can: access a site plan (e.g., solar farm site plan) representing a map of a non-uniform outdoor terrain; identify an initial installation zone, in the site plan, corresponding to a primary container140(e.g., pallet of solar panels) currently loaded onto the set of forks113; and trigger the off-road vehicle to autonomously maneuver the non-uniform outdoor terrain to deliver the primary container140to the initial installation zone. Additionally, the system100can then: extract a geospatial location of a primary install location of an initial structure at the initial installation zone from the site plan; trigger the off-road vehicle to automatically maneuver across the non-uniform outdoor terrain to locate the robotic arm114proximal the primary install location in preparation for an initial object142installation routine. Additionally, the system100can leverage data accessed from the suite of sensors to autonomously and locally maneuver the system100about the primary installation zone—such as via closed-loop controls—to locate the robotic arm114proximal the initial install location at the initial installation zone.

In another implementation, an operator can manually control the system100to maneuver the off-road vehicle in order to locate the robotic arm114proximal the initial install location at the structure150.

Therefore, in preparation for an initial object142installation routine, the system100can deliver a primary container140(e.g., pallet of solar panels) across non-uniform outdoor terrain in order to locate the primary container140proximal an initial install location of an initial installation zone.

The system100can then repeat the steps described above to maneuver the off-road vehicle—and therefore the attachment assembly110—across a set of install locations arranged on a non-uniform outdoor terrain.

6. Gross Install Pose

Blocks of the Method S100recite, prior to retrieving an initial object142from the initial set of objects in the initial container via the robotic arm114: accessing an initial image from an primary optical sensor120defining an initial field of view intersecting the initial install location in Block S120; detecting an initial set of install features152at the initial install location based on the initial image in Block S122; calculating an initial gross install pose of the initial object142that locates the initial object142proximal the initial install location and offset from the initial set of install features152by an initial target offset distance in Block S130; and calculating an initial install path navigable by the robotic arm114to retrieve the initial object142from the initial set of objects in the initial container and to maneuver the initial object142to the initial gross install pose in Block S132.

Generally, the system100can: access an initial image (e.g., color image) from a primary optical sensor120depicting the initial install location at a structure150; detect an installation feature (e.g., clamp) arranged at the initial install location; derive a gross install pose that is proximal the initial install location and offset from the initial install features by a target offset distance; and derive an install path to maneuver a primary object from the primary container140—loaded onto the set of forks113—to the gross install pose offset from the installation feature at the initial install location.

In one implementation, the system100can: access a baseline offset between an initial object142and a set of initial install features arranged on the initial install location; access a template set of install features152, such as by accessing a virtual model representing the install features from local memory and/or from a remote computer system100; capture an initial image at a primary optical sensor120defining a field of view of the initial install location on the structure150; implement computer vision techniques (e.g., template matching, object detection) to extract a set of visual features from the initial image; and identify presence of an initial set of install features152at the initial install location based on the set of visual features and the template set of install features152. The system100can then: access a baseline offset between the primary object and the set of initial install features at the initial install location; and calculate a gross install pose proximal the initial install location based on the baseline offset and the initial set of install features152.

For example, the system100can implement photogrammetry techniques (e.g., stereophotogrammetry, depth perception) to derive a geospatial position of the initial set of install features152at the initial install location. Accordingly, the system100can then calculate the gross install pose defining a target offset between the initial object142and the set of initial install features based on the geospatial position of the initial set of install features152and the baseline offset.

Therefore, the system100can calculate a gross install pose of an initial object142as offset from install features at the initial install location, thereby enabling an operator proximal the initial install location to observe an interstice between the initial object142and the initial set of install features152during installation of the initial object142at the initial install location.

6.1 Calculating Install Path

In one implementation, the system100can further calculate an install path to maneuver the initial object142from the primary container140(e.g., pallet of solar panels)—loaded onto the set of forks113—to the gross install pose proximal the initial install location. In this implementation, the system100can: access a template pose defining the gross install pose of the initial object142, such as by accessing a virtual model of the initial object142defining the gross install pose from internal memory or accessing the virtual model of the initial object142from a remote computer system100; access an image from a primary optical sensor120defining a field of view intersecting the primary container140loaded onto the set of forks113; extract a set of visual features from the image depicting the initial object142arranged on the primary container140; and derive an initial pose of the initial object142based on the set of visual features. Accordingly, the system100can then implement path planning techniques (e.g., graph-based path planning, sampling-based path planning) to calculate an install path to maneuver the initial object142to the target gross install pose according to the template pose and the initial pose.

Therefore, the system100can calculate an install path to maneuver an initial object142about an outdoor environment from the primary container140—loaded onto the set of forks113—to the gross install pose at the initial install location.

6.2 Adjusting Offset Distance

In one implementation, following maneuver of the initial object142to the gross install pose, the system100can validate positioning of the initial object142at the gross install pose. In this implementation, the initial object142grasped by the robotic arm114can obfuscate a primary field of view of the initial install location captured by a primary optical sensor120arranged on the robotic arm114. Accordingly, the system100can alternatively access an image from a secondary optical sensor122arranged on the chassis of the off-road vehicle and defining a secondary field of view of the initial install location. The system100can then: extract a set of visual features from the image; derive an offset distance between the initial object142and the set of initial install features at the initial install location; and, in response to the offset distance deviating from a target offset distance between the initial object142and the set of initial install features, trigger the robotic arm114to locate the initial object142at the target offset distance from the set of initial install features.

Therefore, the system100can leverage multiple fields of view of optical sensors coupled to the off-road vehicle and/or coupled to the attachment assembly110to increase image resolution and position resolution during an installation routine and thus, validate positioning of the initial object142at the gross install pose proximal the initial install location.

7. Object Retrieval

Blocks of the Method S100recite, at the autonomous off-road vehicle: autonomously navigating the robotic arm114according to the initial install path to retrieve the initial object142from the initial set of objects in the initial container and to locate the initial object142in the initial gross install pose in Block S150; and, following completion of the initial install path by the robotic arm114, entering a manual manipulation mode in Block S152. Generally, the system100can: trigger the robotic arm114to locate an end effector116(e.g., gripper) proximal an initial object142arranged on the primary container140; trigger the end effector116to grasp the initial object142from the primary container140; and trigger the robotic arm114to maneuver the initial object142along the install path from the primary container140—arranged on the set of forks113—to the gross install pose proximal the initial install location.

7.1 Install Path+Stiff Manipulation Mode

In one implementation, the system100can navigate the robotic arm114according to the install path to: retrieve the initial object142from the set of objects within the primary container140; and locate the initial object142in the initial gross install pose proximal the initial install location. Additionally, subsequent to locating the initial object142at the initial gross install pose, the system100can then enter a stiff manipulation mode such as, by applying a brake to joints of the robotic arm114to lock movement of the robotic arm114, to restrict movement of the initial object142while supporting the weight of the initial object142and maintaining the initial object142fully within the keep-in boundary. Therefore, during time periods of local wind at the initial install location, the system100can maintain the initial object142proximal the initial install location.

7.2 Object Orientation

In one implementation, the system100can trigger the robotic arm114to locate the initial object142in a target orientation at the initial install location. In this implementation, the system100can, as described above, trigger the robotic arm114to: retrieve the initial object142from the primary container140loaded onto the set of forks113; and execute a primary install path to locate the initial object142at the gross install pose proximal the initial install location. In this implementation, following location of the initial object142at the gross install pose, the system100can: access an image from a primary optical sensor120defining a field of view intersecting the initial object142arranged proximal the initial install location; extract a set of visual features from the image; and derive an initial orientation of the initial object142proximal the initial install location based on the set of visual features.

The system100can then, in response to the initial orientation of the initial object142deviating from a target orientation, trigger the robotic arm114to unload the initial object142onto the primary container140. More specifically, the system100can implement the install path to maneuver the initial object142from the gross install pose back to the primary container140loaded onto the set of forks113. Thus, following unloading of the initial object142onto the primary container140, the system100can then implement the steps described above to calculate a secondary install path, different from the primary install path, to maneuver the initial object142to the gross install pose according to the target orientation. The system100can then trigger the robotic arm114to retrieve the initial object142from the primary container140and to maneuver the initial object142along this secondary path to locate the initial object142in the target orientation at the gross install pose.

For example, during installation of a solar panel onto a structure150, the system100can implement the steps described above to verify a polarity of the solar panel proximal the install location. Accordingly, in response to detecting the polarity of the solar panel as deviating from a target polarity, the system100can: unload the solar panel onto the primary container140at the off-road vehicle; and trigger the robotic arm114to then retrieve the solar panel from the primary container140and locate the solar panel at the install location according to the target polarity.

Therefore, the system100can repeat the steps described above to locate the initial object142in a target orientation at an initial install location during an installation routine.

8. Manual Manipulation Mode

Block S140of the method S100recites defining an initial keep-in boundary of the initial object142maintained by the robotic arm114. The initial keep-in boundary is arranged proximal the initial install location and encompassing the initial gross install pose. Generally, the system100can: define a geospatial boundary (e.g., sphere, cube) proximal the initial install location and encompassing the gross install pose; and responsive to application of forces at the distal end of the robotic arm114, initialize operation the robotic arm114in a manual manipulation mode (or “zero-gravity mode”)—within the keep-in boundary—to enable an operator to manually maneuver the initial object142during installation of the initial object142to the initial install location by the operator.

8.1 Keep-In Boundary

In one implementation, the system100can: access an initial keep-in boundary, such as from local memory or from a remote computer system100; and following location of the initial object142at the gross install pose, project the keep-in boundary onto a spatial representation (e.g., virtual model) of the initial install location. In one example, the system100can align a center of mass of the keep-in boundary to the gross install pose proximal the initial install location. In another example, the system100can align the keep-in boundary to the initial install features at the initial install location. In this implementation, the system100can trigger operation of the robotic arm114in the manual manipulation mode while the initial object142remains within the keep-in boundary. Alternatively, in response to the initial object142exiting the keep-in boundary, the system100can terminate operation of the robotic arm114in the manual manipulation mode to prevent an operator from maneuvering the initial object142away from the initial install location.

In another implementation, the system100can derive the keep-in boundary based on a geometry of the initial object142at the gross install pose. In this implementation, the system100can: access an image from a primary optical sensor120defining a field of view intersecting the initial object142arranged at the initial install location; extract a set of visual features from the image; derive a geometry of the initial object142based on the set of features; and generate the keep-in boundary based on the geometry of the initial object142. For example, the system100can: generate a three-dimensional boundary approximating a geometry of the initial object142; and scale this three-dimensional boundary, such as by a scalar coefficient (e.g., 1.05), to define the keep-in boundary.

Therefore, the system100can define a boundary for operation of the robotic arm114in the manual manipulation mode to: enable an operator to manually maneuver the initial object142proximal the initial install location during an installation routine; and prevent an operator from maneuvering the initial object142away from the initial install location.

8.2 Contextual Boundary+Collision Avoidance

In one implementation, during navigation of the initial object142—such as by an operator maneuvering the initial object142via the control handle118—proximal the initial install location, the system100can: access an image from the primary optical sensor120defining the field of view intersecting the initial install location; detect structural features adjacent the initial install location which can collide with the initial object142; and define a keep-in boundary to prevent collision of the initial object142to adjacent structural elements (e.g., solar panels, installation features) at the initial install location.

For example, the system100can: access an image from a primary optical sensor120defining the field of view intersecting the initial install location; and implement computer vision techniques to detect a secondary object144(e.g., solar panel) arranged adjacent the initial install location and detect an install plane (e.g., rectangular plane) encompassed by the set of initial install features at the initial install location. The system100can then define the keep-in boundary to: encompass the gross install pose; define an interstice between a lateral side of the secondary object144and a periphery of the keep-in boundary; constrained within the set of initial install features; and extending below the install plane at the initial install location.

Therefore, during navigation of the initial object142within the keep-in boundary, the system100can restrict navigation of the initial object142according to the keep-in boundary to prevent the initial object142from breaching the keep-in boundary and colliding with adjacent structural elements (e.g., the secondary object144, the set of initial install features).

The system100can then repeat these steps across subsequent installation of objects across a set of install locations at the structure150.

8.3 Initializing Manipulation Mode

In one implementation, the system100can detect presence of an operator proximal the initial install location at the structure150, such as by detecting presence of the operator in an image captured from a primary optical sensor120defining a field of view of the initial install location. The system100can then trigger the robotic arm114to retrieve the initial object142from the primary container140loaded onto the set of forks113; maneuver the initial object142along the install path to locate the initial object142at the gross install pose; and locate the control handle118, coupled to an end effector116of the robotic arm114, proximal the operator.

In this implementation, following location of the initial object142at the gross install pose proximal the initial install location, the system100can then enter a manipulation mode of the initial object142. For example, the system100can: receive application of a force (e.g., pull force) at the control handle118extending from the end effector116; read a sequence of force values from a force sensor124coupled to the control handle118; detect a ramp rate in the sequence of force values and, in response to the ramp rate of the sequence of force values exceeding a threshold ramp rate, trigger operation of the robotic arm114in a stiff manipulation mode by locking and/or braking joints of the robotic arm114to restrict motion control of the initial object142by the operator while supporting the initial object142within the keep-in boundary. In another example, in response to the ramp rate of the sequence of force values falling below a threshold ramp rate, the system100can trigger operation of the robotic arm114in a manual manipulation mode by releasing joints of the robotic arm114to enable motion control of the initial object142by the operator while supporting the initial object142within the keep-in boundary.

In another implementation, following location of the initial object142at the gross install pose proximal the initial install location, the system100can: detect presence of the operator proximal the initial install location as described above; and trigger rotation of the end effector116to align the control handle118with the operator. The system100can then, as described above, trigger operation of the robotic arm114in the manual manipulation mode responsive to applied forces at the distal end of the robotic arm114(i.e., at the control handle118).

Therefore, the system100can operate the robotic arm114in a manual manipulation mode within a keep-in boundary to enable an operator to manually maneuver the initial object142during installation of the initial object142to the initial install location by the operator.

8.4 Manipulation Resistance

Generally, in the manual manipulation mode, the system100can apply a manipulation resistance (e.g., translational manipulation resistance, rotational manipulation resistance) to the robotic arm114to maintain the initial object142—supported at the distal end of the robotic arm114—entirely within the keep-in boundary during installation of the initial object142at the initial install location by an operator. More specifically, the system100can trigger brakes at joints of the robotic arm114to resist movement of the initial object142within the keep-in boundary and/or lock the initial object142within the keep-in boundary.

8.4.1 Force Application+Resistance

In one implementation, the system100can: detect a series of forces applied to the distal end of the robotic arm114such as, applied by an operator to the control handle118; and apply a manipulation resistance of the initial object142within the keep-in boundary that is directly proportional to the series of forces applied to the distal end of the robotic arm114.

For example, the system100can: detect an increase in applied force across the series of forces applied to the distal end of the robotic arm114; and in response to detecting this increase, increase manipulation resistance (e.g., translational manipulation resistance, rotational manipulation resistance) of the initial object142within the keep-in boundary. Alternatively, the system100can: detect a decrease in applied force across the series of forces applied to the distal end of the robotic arm114; and in response to detecting this decrease, decrease manipulation resistance (e.g., translational manipulation resistance, rotational manipulation resistance) of the initial object142within the keep-in boundary.

In another example, the system100can: define a translational manipulation resistance-directly proportional to the series of forces—for translating the initial object142in directions of the series of forces within the keep-in boundary.

Additionally, the system100can define defining a rotational manipulation resistance for rotating the initial object142in directions of the series of forces within the keep-in boundary. The rotational manipulation resistance can include: a yaw manipulation resistance constrained to one degree of rotation; a roll manipulation resistance constrained to one degree of rotation; and a pitch manipulation resistance directly proportional to the series of forces.

Therefore, the system100can prevent sudden movement-such as from forces applied by an operator and/or local wind—of the initial object142within the keep-in boundary and retain the initial object142entirely within the keep-in boundary.

8.4.2 Location-Based Manipulation Resistance

In one implementation, the system100can: track locations of the initial object142within the keep-in boundary; and apply a manipulation resistance to the initial object142—supported at the distal end of the robotic arm114—associated to a particular location (e.g., a center zone, periphery zone) within the keep-in boundary.

For example, in response to navigating the initial object142within a center zone-such as encompassing an install plane of the initial install location—of the keep-in boundary, the system100can apply a primary manipulation resistance to initial object142to enable the operator to freely maneuver the initial within this center zone. In another example, in response to navigating the initial object142proximal the periphery of the keep-in boundary, the system100can apply a secondary manipulation resistancegreater than the primary manipulation resistance—to the robotic arm114to prevent the initial object142from breaching the periphery of the keep-in boundary and colliding with the adjacent objects at the initial install location.

Therefore, during an installation routine of an initial object142to an initial install location, the system100can retain the initial object142entirely within the keep-in boundary.

9. Subsequent Installation Location

Block160of the method S100recites, following installation of the initial object142at the initial install location, autonomously navigating across the non-uniform outdoor terrain to locate the robotic arm114and the initial container proximal a secondary install location on the initial structure. Generally, following successful installation of an initial object142to a primary install location, the system100can repeat the steps as described above to support installation of subsequent objects, from the primary container140, onto subsequent install locations along a structure150.

9.1 Obiect Installation Verification and Tracking

In one implementation, the system100can: confirm successful installation of an initial object142at an initial install location; and record installation of the initial object142at the initial install location in a site plan representing the non-uniform outdoor terrain. For example, the system100can: receive a confirmation of successful installation of the initial object142at the initial install location, such as from an operator device associated with the operator and/or an interactive display at the attachment assembly110; access a secondary image from the primary optical sensor120defining a field of view intersecting the initial object142mounted at the initial install location; detect a tag (e.g., QR code) associated with the initial object142in the secondary image; and record a geospatial location corresponding to the tag in the initial installation zone, in a set of installation zones, defined in the site plan.

The system100can then repeat the steps described above for each subsequent installation of objects across a set of install locations. Therefore, the system100can maintain an audit trail of objects mounted across these installation locations.

9.1 Repeat Offset Distance

In one implementation, following successful installation of the initial object142at the initial install location, the system100can, as described above: access a secondary image from the primary optical sensor120defining a field of view intersecting a secondary install location, adjacent the initial install location, at the structure150; register a secondary set of install features152at the secondary install location based on the secondary image; and calculate a secondary gross install pose proximal the secondary install location defining a secondary target offset between the secondary object144and the secondary set of install features152. In this implementation, the secondary target offset approximates (e.g., within 0.001 inches) the initial target offset in order to mimic successful installation of the initial object142at the initial install location.

In this implementation, the system100can also calculate a secondary install path to maneuver the secondary object144at the secondary gross install pose. Similarly, the secondary install path can approximate (e.g., within 0.001 inches) the initial install path in order to mimic successful installation of the initial object142at the initial install location.

The system100can then: trigger the robotic arm114to retrieve the secondary object144from the primary container140and maneuver the secondary object144along the secondary install path to locate the secondary object144at the secondary gross install pose; and define a secondary keep-in boundary, proximal the secondary install location and encompassing the secondary gross install pose, for operating the robotic arm114in the manual manipulation mode. Furthermore, in response to detecting forces applied to the distal end of the robotic arm114, the system100can trigger operation of the robotic arm114in the manual manipulation mode to permit manual motion control of the secondary object144by the operator, during installation of the secondary object144to the secondary set of install features152at the secondary install location, while supporting the secondary object144within the secondary keep-in boundary.

Therefore, the system100can execute a secondary installation routine mimicking a previous successful installation routine to support an operator during installation of the secondary object144at the secondary install location.

The system100can then repeat the steps described above across subsequent install locations across a structure150arranged at a non-uniform outdoor terrain.

9.2 Adaptive Offset Distance

In one implementation, during operation of the robotic arm114in the manual manipulation mode by the operator, the system100can: track an offset distance between the initial object142and the initial set of install features152; and adjust a target offset distance between a secondary object144and a secondary set of install features152during a subsequent installation of a secondary object144to a secondary install location. In this implementation, the system100can record a secondary offset distance-different (e.g., greater than, less than) from the initial offset distance-between the initial object142and the initial set of install features152during operation of the robotic arm114in the manual manipulation mode by the operator.

The system100can then, as described above, following successful installation of the initial object142at the initial install location: access a secondary image from the primary optical sensor120defining a field of view intersecting a secondary install location, adjacent the initial install location, at the structure150; register a secondary set of install features152at the secondary install location based on the secondary image; and calculate a secondary gross install pose proximal the secondary install location defining the secondary target offset between the secondary object144and the secondary set of install features152. In this implementation, the secondary target offset approximates (e.g., within 0.001 inches) offset deviation from the initial target offset during operation of the robotic arm114in the manual manipulation mode by the operator to maneuver the initial object142.

In this implementation, the system100can also calculate a secondary install path to maneuver the secondary object144at the secondary gross install pose. The system100can then: trigger the robotic arm114to retrieve the secondary object144from the primary container140and maneuver the secondary object144along the secondary install path to locate the secondary object144at the secondary gross install pose; and define a secondary keep-in boundary, proximal the secondary install location and encompassing the secondary gross install pose, for operating the robotic arm114in the manual manipulation mode. Furthermore, in response to detecting forces applied to the distal end of the robotic arm114, the system100can trigger operation of the robotic arm114in the manual manipulation mode to permit manual motion control of the secondary object144by the operator, during installation of the secondary object144to the secondary set of install features152at the secondary install location, while supporting the secondary object144within the secondary keep-in boundary.

Therefore, the system100can execute a secondary installation routine adjusted from a previous successful installation routine to support an operator during installation of the secondary object144at the secondary install location.

9.3 Un-Install

In one implementation, the system100can similarly implement the steps and processes described above to: un-install an initial object142from an initial install location; and maneuver the initial object142from the initial install location to the primary container140loaded onto the set of forks113. In this implementation, the system100can: navigate the off-road vehicle about the non-uniform outdoor terrain to locate the robotic system100proximal an initial install location including an initial object142; access a secondary image from the primary optical sensor120defining a field of view intersecting the primary container140; register a set of loading features at the primary container140based on the image; calculate a gross install pose proximal the primary container140defining a target offset between the initial object142and the set of loading features; and calculate an install path to maneuver the initial object142at the secondary gross install pose.

The system100can then: trigger the robotic arm114to retrieve the initial object142from the initial install location and maneuver the initial object142along the install path to locate the initial object142at the gross install pose proximal the primary container140; and define a keep-in boundary, proximal the primary container140and encompassing the gross install pose, for operating the robotic arm114in the manual manipulation mode. Accordingly, in response to detecting forces applied to the distal end of the robotic arm114, the system100can trigger operation of the robotic arm114in the manual manipulation mode to permit manual motion control of the initial object142by an operator, during loading of the initial object142onto the primary container140, while supporting the initial object142within the keep-in boundary. The system100can then repeat the steps described above to un-install objects from a set of install locations across a non-uniform outdoor terrain.

The systems and methods described herein can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions. The instructions can be executed by computer-executable components integrated with the application, applet, host, server, network, website, communication service, communication interface, hardware/firmware/software elements of a user computer or mobile device, wristband, smartphone, or any suitable combination thereof. Other systems and methods of the embodiment can be embodied and/or implemented at least in part as a machine configured to receive a computer-readable medium storing computer-readable instructions. The instructions can be executed by computer-executable components integrated by computer-executable components integrated with apparatuses and networks of the type described above. The computer-readable medium can be stored on any suitable computer readable media such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives, floppy drives, or any suitable device. The computer-executable component can be a processor but any suitable dedicated hardware device can (alternatively or additionally) execute the instructions.

As a person skilled in the art will recognize from the previous detailed description and from the figures and claims, modifications and changes can be made to the embodiments of the invention without departing from the scope of this invention as defined in the following claims.