Patent ID: 12227374

DETAILED DESCRIPTION

Systems and methods for dynamically packing objects (e.g., packages and/or boxes) are described herein. A robotic system (e.g., an integrated system of devices that executes one or more designated tasks) configured in accordance with some embodiments provides enhanced packing and storage efficiency by dynamically deriving optimal storage locations for the objects and stacking them accordingly.

Traditional systems use offline packing simulators to predetermine packing sequences/arrangements. The traditional packing simulators process object information (e.g., case shapes/sizes) for a predetermined or estimated set of cases to generate packing plans. Once determined, the packing plans dictate and/or require specific placement locations/poses of the objects at destinations (e.g., pallets, bins, cages, boxes, etc.), predefined sequences for the placement, and/or predetermined motion plans. From the predetermined packing plans, the traditional packing simulators may derive source requirements (e.g., sequences and/or placements for the objects) that match or enable the packing plans. Because the packing plans are developed offline in traditional systems, the plans are independent of actual packing operations/conditions, object arrivals, and/or other system implementations. Accordingly, the overall operation/implementation will require the received packages (e.g., at the starting/pickup location) to follow fixed sequences that match the predetermined packing plans. As such, traditional systems cannot adapt to real-time conditions and/or deviations in the received packages (e.g., different sequence, location, and/or orientation), unanticipated errors (e.g., collisions and/or lost pieces), real-time packing requirements (e.g., received orders), and/or other real-time factors.

Further, because traditional systems group and pack objects according to rigid predetermined plans/sequences, they require all objects at a source location to either (1) have a same dimension/type and/or (2) arrive according to a known sequence. For example, the traditional systems would require the objects to arrive (via, e.g., conveyor) at a pickup location according to a fixed sequence. Also, for example, the traditional systems would require the objects at the pickup location to be placed at designated locations according to a predetermined pose. As such, traditional systems require one or more operations to order and/or place the objects at the source (i.e., before the packing operation) according to the predetermined sequence/arrangement. Often, the traditional systems require a sequence buffer, which costs upwards of one million U.S. dollars, to order and/or place the objects at the source according to the predetermined sequence/pose.

In contrast to the traditional systems, the robotic system described herein can dynamically (e.g., as the object arrives or is identified and/or after initially starting one or more operations, such as the actual packing operation) derive placement locations of the objects during system operation. In some embodiments, the robotic system can initiate/implement the dynamic derivation of the placement based on a triggering event, such as a re-evaluation timing, a packing/manipulation error (e.g., a collision event or a lost piece event), an unrecognized object (e.g., at the source and/or the destination), a change in locations/orientations of already-placed packages, and/or occurrence of other dynamic conditions. In some embodiments, the placement location can be dynamically derived when the robotic system lacks prior information about the arriving objects, such as for receiving previously unknown objects and/or for receiving objects in random/unknown sequences. In dynamically deriving the placement locations, the robotic system can utilize various real-time conditions (e.g., currently existing or ongoing conditions) that include, e.g., available/arriving objects, object characteristics and/or requirements, placement requirements, and/or other real-time factors.

The robotic system can derive the placement locations based on a discretization mechanism (e.g., a process, a circuit, a function, and/or a routine). For example, the robotic system can use the discretization mechanism to describe physical sizes/shapes of objects and/or target locations according to a discretization unit (i.e., one discrete area/space). The robotic system can generate discretized object profiles that use the discretization units to describe the expected objects and/or discretized destination profiles that describe the target location (e.g., surface on top of the pallet and/or a space/bottom surface inside a bin/case/box). Accordingly, the robotic system can transform continuous real-world space/area into computer-readable digital information. Further, the discretized data can allow a reduction in computational complexity for describing package footprint and for comparing various package placements. For example, package dimensions can correspond to integer numbers of discretization units, which lead to easier mathematical computations, instead of real-world decimal numbers.

In some embodiments, the robotic system can check discretized cells for the placement platform to determine object placement possibilities. For example, the robotic system can use depth measurements or heights of placed objects on the placement platform. The robotic system can determine the depth measure to determine heights at/according to the discretized cells. The robotic system can evaluate the depth measure according to groupings of the discretized cells that correspond to the object targeted for placement. The robotic system can determine the maximum height within the grouping for evaluating the placement possibilities. In other words, the robotic system can determine whether the tested placement location provides sufficient support such that the placed object can be placed relatively flat (e.g., according to predetermined thresholds and/or conditions). Details regarding the dynamic placement derivations are described below.

Accordingly, the robotic system can improve efficiency, speed, and accuracy for dynamically deriving the object placement based on the real-time conditions. For example, the system described herein can derive the placement locations when the real-world conditions present uncertainties associated with and/or deviations from anticipated conditions. As such, the robotic system can receive and pack unknown objects and/or randomly arriving (i.e., without a known/predetermined sequence) objects.

Further, the robotic system can reduce overall costs by eliminating the one or more operations, machines (e.g., sequence buffers), and/or human assistance that would be necessary in traditional systems to order or place the objects at the source and/or for the packing operation (e.g., for error handling). By dynamically deriving placement locations as the objects become available (e.g., based on object arrival and/or triggering events), the robotic system eliminates the need to reorganize or sequence the packages, along with the associated machines/human operations.

In the following description, numerous specific details are set forth to provide a thorough understanding of the presently disclosed technology. In other embodiments, the techniques introduced here can be practiced without these specific details. In other instances, well-known features, such as specific functions or routines, are not described in detail in order to avoid unnecessarily obscuring the present disclosure. References in this description to “an embodiment,” “one embodiment,” or the like mean that a particular feature, structure, material, or characteristic being described is included in at least one embodiment of the present disclosure. Thus, the appearances of such phrases in this specification do not necessarily all refer to the same embodiment. On the other hand, such references are not necessarily mutually exclusive either. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments. It is to be understood that the various embodiments shown in the figures are merely illustrative representations and are not necessarily drawn to scale.

Several details describing structures or processes that are well-known and often associated with robotic systems and subsystems, but that can unnecessarily obscure some significant aspects of the disclosed techniques, are not set forth in the following description for purposes of clarity. Moreover, although the following disclosure sets forth several embodiments of different aspects of the present technology, several other embodiments can have different configurations or different components than those described in this section. Accordingly, the disclosed techniques can have other embodiments with additional elements or without several of the elements described below.

Many embodiments or aspects of the present disclosure described below can take the form of computer- or processor-executable instructions, including routines executed by a programmable computer or processor. Those skilled in the relevant art will appreciate that the disclosed techniques can be practiced on computer or processor systems other than those shown and described below. The techniques described herein can be embodied in a special-purpose computer or data processor that is specifically programmed, configured, or constructed to execute one or more of the computer-executable instructions described below. Accordingly, the terms “computer” and “processor” as generally used herein refer to any data processor and can include Internet appliances and handheld devices (including palm-top computers, wearable computers, cellular or mobile phones, multi-processor systems, processor-based or programmable consumer electronics, network computers, mini computers, and the like). Information handled by these computers and processors can be presented at any suitable display medium, including a liquid crystal display (LCD). Instructions for executing computer- or processor-executable tasks can be stored in or on any suitable computer-readable medium, including hardware, firmware, or a combination of hardware and firmware. Instructions can be contained in any suitable memory device, including, for example, a flash drive and/or other suitable medium.

The terms “coupled” and “connected,” along with their derivatives, can be used herein to describe structural relationships between components. It should be understood that these terms are not intended as synonyms for each other. Rather, in particular embodiments, “connected” can be used to indicate that two or more elements are in direct contact with each other. Unless otherwise made apparent in the context, the term “coupled” can be used to indicate that two or more elements are in either direct or indirect (with other intervening elements between them) contact with each other, or that the two or more elements cooperate or interact with each other (e.g., as in a cause-and-effect relationship, such as for signal transmission/reception or for function calls), or both.

Suitable Environments

FIG.1is an illustration of an example environment in which a robotic system100with a dynamic packing mechanism may operate. The robotic system100can include and/or communicate with one or more units (e.g., robots) configured to execute one or more tasks. Aspects of the dynamic packing mechanism can be practiced or implemented by the various units.

For the example illustrated inFIG.1, the robotic system100can include an unloading unit102, a transfer unit104(e.g., a palletizing robot and/or a piece-picker robot), a transport unit106, a loading unit108, or a combination thereof in a warehouse or a distribution/shipping hub. Each of the units in the robotic system100can be configured to execute one or more tasks. The tasks can be combined in sequence to perform an operation that achieves a goal, such as to unload objects from a truck or a van and store them in a warehouse or to unload objects from storage locations and prepare them for shipping. In some embodiments, the task can include placing the objects on a target location (e.g., on top of a pallet and/or inside a bin/cage/box/case). As described in detail below, the robotic system100can derive individual placement locations/orientations, calculate corresponding motion plans, or a combination thereof for placing and/or stacking the objects. Each of the units can be configured to execute a sequence of actions (e.g., operating one or more components therein) to execute a task.

In some embodiments, the task can include manipulation (e.g., moving and/or reorienting) of a target object112(e.g., one of the packages, boxes, cases, cages, pallets, etc. corresponding to the executing task) from a start/source location114to a task/destination location116. For example, the unloading unit102(e.g., a devanning robot) can be configured to transfer the target object112from a location in a carrier (e.g., a truck) to a location on a conveyor belt. Also, the transfer unit104can be configured to transfer the target object112from one location (e.g., the conveyor belt, a pallet, or a bin) to another location (e.g., a pallet, a bin, etc.). For another example, the transfer unit104(e.g., a palletizing robot) can be configured to transfer the target object112from a source location (e.g., a pallet, a pickup area, and/or a conveyor) to a destination pallet. In completing the operation, the transport unit106can transfer the target object112from an area associated with the transfer unit104to an area associated with the loading unit108, and the loading unit108can transfer the target object112(by, e.g., moving the pallet carrying the target object112) from the transfer unit104to a storage location (e.g., a location on the shelves). Details regarding the task and the associated actions are described below.

For illustrative purposes, the robotic system100is described in the context of a shipping center; however, it is understood that the robotic system100can be configured to execute tasks in other environments/for other purposes, such as for manufacturing, assembly, packaging, healthcare, and/or other types of automation. It is also understood that the robotic system100can include other units, such as manipulators, service robots, modular robots, etc., not shown inFIG.1. For example, in some embodiments, the robotic system100can include a depalletizing unit for transferring the objects from cage carts or pallets onto conveyors or other pallets, a container-switching unit for transferring the objects from one container to another, a packaging unit for wrapping the objects, a sorting unit for grouping objects according to one or more characteristics thereof, a piece-picking unit for manipulating (e.g., for sorting, grouping, and/or transferring) the objects differently according to one or more characteristics thereof, or a combination thereof.

Suitable System

FIG.2is a block diagram illustrating the robotic system100in accordance with one or more embodiments of the present technology. In some embodiments, for example, the robotic system100(e.g., at one or more of the units and/or robots described above) can include electronic/electrical devices, such as one or more processors202, one or more storage devices204, one or more communication devices206, one or more input-output devices208, one or more actuation devices212, one or more transport motors214, one or more sensors216, or a combination thereof. The various devices can be coupled to each other via wire connections and/or wireless connections. For example, the robotic system100can include a bus, such as a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), an IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (also referred to as “Firewire”). Also, for example, the robotic system100can include bridges, adapters, processors, or other signal-related devices for providing the wire connections between the devices. The wireless connections can be based on, for example, cellular communication protocols (e.g., 3G, 4G, LTE, 5G, etc.), wireless local area network (LAN) protocols (e.g., wireless fidelity (Wi-Fi)), peer-to-peer or device-to-device communication protocols (e.g., Bluetooth, Near-Field communication (NFC), etc.), Internet of Things (IoT) protocols (e.g., NB-IoT, LTE-M, etc.), and/or other wireless communication protocols.

The processors202can include data processors (e.g., central processing units (CPUs), special-purpose computers, and/or onboard servers) configured to execute instructions (e.g., software instructions) stored on the storage devices204(e.g., computer memory). In some embodiments, the processors202can be included in a separate/stand-alone controller that is operably coupled to the other electronic/electrical devices illustrated inFIG.2and/or the robotic units illustrated inFIG.1. The processors202can implement the program instructions to control/interface with other devices, thereby causing the robotic system100to execute actions, tasks, and/or operations.

The storage devices204can include non-transitory computer-readable mediums having stored thereon program instructions (e.g., software). Some examples of the storage devices204can include volatile memory (e.g., cache and/or random-access memory (RAM)) and/or non-volatile memory (e.g., flash memory and/or magnetic disk drives). Other examples of the storage devices204can include portable memory and/or cloud storage devices.

In some embodiments, the storage devices204can be used to further store and provide access to processing results and/or predetermined data/thresholds. For example, the storage devices204can store master data252that includes descriptions of objects (e.g., boxes, cases, and/or products) that may be manipulated by the robotic system100. In one or more embodiments, the master data252can include a dimension, a shape (e.g., templates for potential poses and/or computer-generated models for recognizing the object in different poses), a color scheme, an image, identification information (e.g., bar codes, quick response (QR) codes, logos, etc., and/or expected locations thereof), an expected weight, other physical/visual characteristics, or a combination thereof for the objects expected to be manipulated by the robotic system100. In some embodiments, the master data252can include manipulation-related information regarding the objects, such as a center-of-mass (CoM) location on each of the objects, expected sensor measurements (e.g., for force, torque, pressure, and/or contact measurements) corresponding to one or more actions/maneuvers, or a combination thereof. Also, for example, the storage devices204can store object tracking data254. In some embodiments, the object tracking data254can include a log of scanned or manipulated objects. In some embodiments, the object tracking data254can include imaging data (e.g., a picture, point cloud, live video feed, etc.) of the objects at one or more locations (e.g., designated pickup or drop locations and/or conveyor belts). In some embodiments, the object tracking data254can include locations and/or orientations of the objects at the one or more locations.

The communication devices206can include circuits configured to communicate with external or remote devices via a network. For example, the communication devices206can include receivers, transmitters, modulators/demodulators (modems), signal detectors, signal encoders/decoders, connector ports, network cards, etc. The communication devices206can be configured to send, receive, and/or process electrical signals according to one or more communication protocols (e.g., the Internet Protocol (IP), wireless communication protocols, etc.). In some embodiments, the robotic system100can use the communication devices206to exchange information between units of the robotic system100and/or exchange information (e.g., for reporting, data gathering, analyzing, and/or troubleshooting purposes) with systems or devices external to the robotic system100.

The input-output devices208can include user interface devices configured to communicate information to and/or receive information from human operators. For example, the input-output devices208can include a display210and/or other output devices (e.g., a speaker, a haptics circuit, or a tactile feedback device, etc.) for communicating information to the human operator. Also, the input-output devices208can include control or receiving devices, such as a keyboard, a mouse, a touchscreen, a microphone, a user interface (UI) sensor (e.g., a camera for receiving motion commands), a wearable input device, etc. In some embodiments, the robotic system100can use the input-output devices208to interact with the human operators in executing an action, a task, an operation, or a combination thereof.

The robotic system100can include physical or structural members (e.g., robotic manipulator arms) that are connected at joints for motion (e.g., rotational and/or translational displacements). The structural members and the joints can form a kinetic chain configured to manipulate an end-effector (e.g., the gripper) configured to execute one or more tasks (e.g., gripping, spinning, welding, etc.) depending on the use/operation of the robotic system100. The robotic system100can include the actuation devices212(e.g., motors, actuators, wires, artificial muscles, electroactive polymers, etc.) configured to drive or manipulate (e.g., displace and/or reorient) the structural members about or at a corresponding joint. In some embodiments, the robotic system100can include the transport motors214configured to transport the corresponding units/chassis from place to place.

The robotic system100can include the sensors216configured to obtain information used to implement the tasks, such as for manipulating the structural members and/or for transporting the robotic units. The sensors216can include devices configured to detect or measure one or more physical properties of the robotic system100(e.g., a state, a condition, and/or a location of one or more structural members/joints thereof) and/or of a surrounding environment. Some examples of the sensors216can include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, etc.

In some embodiments, for example, the sensors216can include one or more imaging devices222(e.g., visual and/or infrared cameras, 2D and/or 3D imaging cameras, distance measuring devices such as lidars or radars, etc.) configured to detect the surrounding environment. The imaging devices222can generate representations of the detected environment, such as digital images and/or point clouds, that may be processed via machine/computer vision (e.g., for automatic inspection, robot guidance, or other robotic applications). As described in further detail below, the robotic system100(via, e.g., the processors202) can process the digital image and/or the point cloud to identify the target object112ofFIG.1, the start location114ofFIG.1, the task location116ofFIG.1, a pose of the target object112, a confidence measure regarding the start location114and/or the pose, or a combination thereof.

For manipulating the target object112, the robotic system100(via, e.g., the various circuits/devices described above) can capture and analyze image data of a designated area (e.g., a pickup location, such as inside the truck or on the conveyor belt) to identify the target object112and the start location114thereof. Similarly, the robotic system100can capture and analyze image data of another designated area (e.g., a drop location for placing objects on the conveyor, a location for placing objects inside the container, or a location on the pallet for stacking purposes) to identify the task location116. For example, the imaging devices222can include one or more cameras configured to generate image data of the pickup area and/or one or more cameras configured to generate image data of the task area (e.g., drop area). Based on the image data, as described below, the robotic system100can determine the start location114, the task location116, the associated poses, a packing/placement location, and/or other processing results. Details regarding the dynamic packing algorithm are described below.

In some embodiments, for example, the sensors216can include position sensors224(e.g., position encoders, potentiometers, etc.) configured to detect positions of structural members (e.g., the robotic arms and/or the end-effectors) and/or corresponding joints of the robotic system100. The robotic system100can use the position sensors224to track locations and/or orientations of the structural members and/or the joints during execution of the task.

Discretization Models

FIG.3AandFIG.3Bare illustrations of discretized data used to plan and pack objects in accordance with one or more embodiments of the present technology.FIG.3Aillustrates discretized objects andFIG.3Billustrates discretized packing platform for the object packing.

In some embodiments, the robotic system100ofFIG.1can include predetermined discretized models/representations of the expected objects stored in the master data252ofFIG.2. In some embodiments, the robotic system100(via, e.g., the processors202ofFIG.2) can dynamically generate the discretized models by mapping continuous surfaces/edges of real-world objects (e.g., packages, pallets, and/or other objects associated with the task) into discrete counterparts (e.g., unit lengths and/or unit areas). For example, the robotic system100can discretize image data (e.g., a top view image and/or point cloud data) of the target object112and/or a pallet top surface captured by the one or more imaging devices222ofFIG.2. In other words, the robotic system100can discretize the image data of the start location114ofFIG.1, a location before the start location114on a conveyor, and/or the task location116ofFIG.1. The robotic system100can discretize based on identifying an outer perimeter of the object/pallet in the image data and then dividing the area within the outer perimeter according to a unit dimension/area. In some embodiments, the unit dimension/area can be scaled or mapped for the image data based on a size and/or a location of the object/pallet relative to the imaging devices222according to a coordinate scheme and/or a predetermined adjustment factor/equation.

As illustrated inFIG.3A, some embodiments of the robotic system100can use discretized object models302to plan/derive placement locations of objects (e.g., the target object112). The discretized object models302(shown using dotted lines) can represent exterior physical dimensions, shapes, edges, surfaces, or a combination thereof (shown using dash lines) for arriving or incoming objects (e.g., packages, boxes, cases, etc.) according to a discretization unit (e.g., a unit length). The discretized object models302can represent expected/known objects and/or unexpected/unknown objects that have been imaged and discretized as described above.

As illustrated inFIG.3B, some embodiments of the robotic system100can use one or more discretized platform models304(e.g., discretized representations of the task locations116ofFIG.1) to plan/derive stacking placements of objects. The discretized platform models304can represent a placement area340(e.g., the physical dimension, shape, or a combination thereof of the task location116, such as a top surface of the task location116, a top surface of a package placed thereon, or a combination thereof) according to the discretization unit. In one or more embodiments, the discretized platform models304can represent real-time conditions of the placement area340, such as via real-time updates. For example, with respect to a top view, the discretized platform models304can initially represent a top surface of a pallet, an inside-bottom surface of a bin or a box, etc. that is to receive and directly contact the objects. As the robotic system100places the objects, the placement area340can change to include top surfaces of the placed packages (e.g., for stacking packages) and the discretized platform model304can be updated to reflect the changes.

In some embodiments, the discretized platform models304can be based on top views of one or more standard size pallets (e.g., 1.1 m by 1.1 m pallets). Accordingly, the discretized platform models304can correspond to pixelated 2D representations of the placement areas along horizontal planes (e.g., the x-y planes) according to a grid system utilized by the robotic system100. In some embodiments, the discretized object models302can include top views (e.g., x-y planes) of expected or arriving objects. Accordingly, the discretized object models302can correspond to pixelated 2D representations of the objects.

The discretization unit, used to generate discretized models, can include a length that is set by a system operator, a system designer, a predetermined input/setting, an order, or a combination thereof. In some embodiments, the robotic system100can use unit pixels310(e.g., polygons, such as squares, having one or more dimensions according to the discretization unit) to describe areas/surfaces of targeted objects (via, e.g., the discretized object models302) and loading platforms/surfaces (via, e.g., the discretized platform models304). Accordingly, the robotic system100can pixelate the objects and the loading platforms in 2D along the x-y axes. In some embodiments, the size of the unit pixels310(e.g., the discretization unit) can change according to dimensions of the objects and/or dimensions of the loading platforms. The size of the unit pixels310can also be adjusted (via, e.g., a preset rule/equation and/or operator selection) to balance required resources (e.g., computation times, required memory, etc.) with packing accuracy. For example, when the size of the unit pixels310decreases, the computation times and the packing accuracy can increase. Accordingly, discretization of the packing tasks (e.g., the target packages and the packing platforms) using the unit pixels310that are adjustable provides increased flexibility for palletizing the packages. The robotic system100can control a balance between the computation resources/time with the packing accuracy according to real-time demands, scenarios, patterns, and/or environments.

In some embodiments, the robotic system100can include for the discretized object models302instances of the unit pixels310that only partially overlap the object, such that the unit pixels310extend beyond the actual peripheral edges of the object. In other embodiments, the robotic system100can exclude partially overlapping instances of the unit pixels310from the discretized platform models304the actual dimensions of the platform surface such that the unit pixels310in the discretized object models302are overlapped and/or contained within the actual peripheral edges of the platform surface.

As an illustrative example,FIG.3Ashows a first model-orientation332and a second model-orientation334of a discretized object model representing the target object112. In some embodiments, the robotic system100can rotate one of the discretized model (i.e., that is captured/stored as the first model-orientation332) by a predetermined amount along the imaged plane. As illustrated inFIG.3A, the robotic system100can rotate the discretized object model302about a vertical axis (extending in-out or perpendicular to the plane of the illustration) and along a horizontal plane (e.g., represented along the x and y axes) by 90 degrees for the second model-orientation334. The robotic system100can use the different orientations to test/evaluate corresponding placements of the objects.

Based on the discretized data/representations, the robotic system100can dynamically derive a placement location350for the target object112. As illustrated inFIG.3B, the robotic system100can dynamically derive the placement location350, even after one or more objects (e.g., illustrated as objects with diagonal fills inFIG.3B) have been placed on the placement area340. Also, the dynamic derivation of the placement location350can occur after/while the target object112is unloaded/de-shelved, registered, scanned, imaged, or a combination thereof. For example, the robotic system100can dynamically derive the placement location350as the target object112is transported (via, e.g., a conveyor), after the imaging devices222ofFIG.2generate the image data of the target object112, or a combination thereof.

Dynamically deriving the placement location350of an object provides increased flexibility and reduced human labor for shipping/packaging environments. The robotic system100can use discretized real-time images/depth maps of objects and the pallet (i.e., including the already-placed objects) to test and evaluate different placement locations and/or orientation. Accordingly, the robotic system100can still pack objects without any human operator interventions even when the object is not recognizable (e.g., for new/unexpected objects and/or computer vision errors), when an arrival sequence/order of the objects is unknown, and/or when an unexpected event occurs (e.g., a piece-loss event, and/or a collision event).

For illustrative purposes the placement location350is shown inFIG.3Bas being adjacent to (i.e., placed on the same horizontal layer/height as) the already-placed objects, such as directly on/contacting the pallet. However, it is understood that the placement location350can be on top of the already-placed objects. In other words, the robotic system100can derive the placement location350for stacking the target object112over and/or on top of one or more objects already on the pallet. As described in detail below, the robotic system100can evaluate the heights of the already-placed objects in deriving the placement location350to ensure that the object is sufficiently supported when stacked on top of the already-placed objects.

In some embodiments, the robotic system100can identify object edges362in deriving the placement location350. The object edges362can include lines in the image data that represent edges and/or sides of the objects already placed on the pallet. In some embodiments, the object edges362can correspond to edges that are exposed (e.g., not directly contacting/adjacent to another object/edge), such that they define a perimeter of one or a group of objects (e.g., a layer of objects) placed on the task location116.

As described further in detail below, the robotic system100can derive the placement location350according to a set of placement rules, conditions, parameters, requirements, etc. In some embodiments, the robotic system100can derive the placement location350based on evaluating/testing one or more candidate positions360. The candidate positions360can correspond to the discretized object models302overlaid on top of the discretized platform models304at various locations and/or orientations. Accordingly, the candidate positions360can include potentially placing the target object112adjacent to one or more of the object edges362and/or potentially stacking target object112on one or more of the already-placed objects. The robotic system100can evaluate each of the candidate positions360according to various parameters/conditions, such as support measure/condition, supported weight in comparison to fragility ratings (e.g., maximum supported weight, such as for packages stacked thereon) of the supporting objects, space/packing implications, or a combination thereof. The robotic system100can further evaluate the candidate positions360using one or more placement rules, such as collision free requirement, stack stability, customer-specified rules/priorities, package separation requirements or the absence thereof, maximization of total loaded packages, or a combination thereof.

Real-Time Placement Surface Updates

FIGS.4A and4Billustrate various aspects of a support computation and a support metric, in accordance with one or more embodiments of the present technology. In some embodiments, as illustrated inFIG.4A, the robotic system100ofFIG.1can generate the candidate positions360ofFIG.3Bbased on overlapping the discretized object model302ofFIG.3Aof the target object112ofFIG.1over the discretized platform model304of the task location116ofFIG.1. Further, the robotic system100can iteratively move the discretized object model302across the discretized platform model304in generating the candidate positions360. For example, the robotic system100can generate an initial instance of the candidate position360by placing a corresponding discretized object model302according to one or more orientations (e.g., the first model-orientation332ofFIG.3Aand/or the second model-orientation334ofFIG.3A) at a predetermined initial location (e.g., a corner) of the discretized platform model304. For the next instance of the candidate position360, the robotic system100can move the discretized object model302, which corresponds to another/next object, by a predetermined distance (e.g., one or more unit pixels310ofFIG.3B) according to a predetermined direction/pattern.

When the candidate positions360overlap one or more objects already placed at the task location116, the robotic system100can calculate and evaluate a measure of support provided by the already-placed objects. To calculate and evaluate the measure of support, the robotic system100can determine heights/contour for the placement area340ofFIG.3Bin real-time using one or more of the imaging devices222ofFIG.2. In some embodiments, the robotic system100can use depth measures (e.g., point cloud values) from one or more of the imaging devices222located above the task location116. In some embodiments, the robotic system100can have predetermined height/position values that correspond to vertical position of the ground and/or the platform (e.g., pallet) surface, such as a height of the platform surface above the facility ground surface. In some embodiments, the robotic system100can use the depth measure to calculate the heights/contour of the exposed top surface(s) of the platform, the placed objects, or a combination thereof. In some embodiments, the robotic system100can image the task location116and update the heights of the exposed top surface(s) in real-time, such as after transporting the object to and/or placing the object on the platform.

In some embodiments, as illustrated inFIG.4A, the robotic system100can update the discretized platform model304to include height measures402. The robotic system100can determine the height measures402according to each of the discretized pixels (e.g., the unit pixels310) in the discretized platform model304. For example, the robotic system100can determine the height measures402as the maximum heights for the surface portions of the placement area340represented by the corresponding unit pixels310.

For each of the candidate positions360that overlap one or more of the already-placed objects, the robotic system100can evaluate the placement possibility based on the height measures402. In some embodiments, the robotic system100can evaluate the placement possibility based on identifying the highest value of the height measures402overlapped in each of the candidate positions360. The robotic system100can further identify other height measures402located in each of the candidate positions360with the height measures402within a limit of a difference threshold relative to the highest measure of the height measures402. The qualifying cells/pixels can represent locations that can provide support for the stacked object such that the stacked object rests essentially flat/horizontal.

As illustrated inFIG.4A, for the first of the candidate positions360(upper-left corner of the discretized platform model304), the highest height measure can be 0.3 (i.e., 300 millimeters (mm) tall). For the difference threshold predetermined as 0.02 (representing, e.g., 20 mm), the robotic system100can identify the top four discretized cells/pixels as satisfying the difference threshold. The robotic system100can use the identified/qualifying cells/pixels to evaluate/represent the degree of support.

FIG.4Billustrates a further example of the support computation.FIG.4Bshows one of the candidate positions360ofFIG.3with the discretized object model302(shown using solid thicker outline) overlaid in an upper-left corner of the discretized platform model304. The robotic system100can calculate/utilize various support parameters410, which are parameters used to evaluate the candidate position360. For example, the support parameters410can include discretized dimensions412, an overlapped area414, a height difference threshold416, a support threshold418, a maximum height420, a lower height limit422, a qualifying count424, a set of support area outlines426, a support area size428, a support ratio430, a center-of-mass (CoM) location432, or a combination thereof.

The discretized dimensions412can describe physical dimensions (e.g., length, width, height, circumference, etc.) of the target object112ofFIG.1according to the unit pixels310ofFIG.3A. For example, the discretized dimensions412can include quantities of the unit pixels310that form peripheral edges of the discretized object model302. The overlapped area414can describe an area (e.g., a footprint size along the horizontal plane) occupied by the target object112, which can similarly be represented according to the unit pixels310. In other words, the overlapped area414can correspond to a quantity of the unit pixels310within the discretized object model302. For the example illustrated inFIG.4B, the target object112can have the discretized dimension412of six pixels by seven pixels, which corresponds to the overlapped area414of 42 pixels.

The height difference threshold416and the support threshold418can correspond to limits used to process and/or validate the candidate positions360. The height difference threshold416, which can be predetermined and/or adjusted by an operator and/or an order, can represent allowed deviations from another reference height (e.g., the maximum height420corresponding to the highest instance of the height measures402in the area overlapped by the discretized object model302) for contacting and/or supporting packages placed on top. In other words, the height difference threshold416can be used to define a range of surface heights that can contact and/or support the package placed thereon. As such, relative to the maximum height420, the lower height limit422can correspond to a lower limit for heights within the overlapped area414that can provide support for the stacked package. For the example illustrated inFIG.4B, the height difference threshold416can be 0.02. When the maximum height420is 0.2, the lower height limit422can be 0.18. Accordingly, in placing the target object112at the candidate position360, the robotic system100can estimate that surfaces/pixels with heights greater than 0.18 will contact and/or provide support for the target object112.

Accordingly, in one or more embodiments, the robotic system100can categorize the unit pixels310within the overlapped area414according to the height difference threshold416. For example, the robotic system100can categorize the unit pixels310having heights satisfying the height difference threshold416(i.e., values greater than or equal to the lower height limit422) as supporting locations442(e.g., a grouping of unit pixels310that represent a surface capable of having objects stacked thereon, such as represented inFIG.4Bvia shaded pixels). The robotic system100can categorize the other unit pixels310as unqualified locations444(e.g., pixels with heights lower than the lower height limit422).

The support threshold418can represent a limit for evaluating the candidate positions360based on a sufficiency of the supporting locations442. For example, the support threshold418can be for evaluating an amount, a ratio, an area, a location, or a combination thereof associated with the supporting locations442. In some embodiments, the support threshold418can be used to determine whether the qualifying count424(e.g., an amount of the supporting locations442) for the candidate position360is sufficient for supporting the target object112.

In one or more embodiments, the support threshold418can be used to evaluate a supported area (e.g., the unit pixels360that can provide support to an object stacked thereon, as can be determined by the height threshold) associated with the supporting locations442. For example, the robotic system100can determine the support area outlines426based on extending edges and/or determining lines that extend across or around the unqualified locations444to connect corners of outermost/perimeter instances of the supporting locations442. Thus, the support area outlines426can exclude the unqualified locations444. Accordingly, the support area outlines426can define a perimeter for the supported area based on the perimeter instances of the supporting locations442. Since the support area outlines426can extend across and/or include the unqualified locations444, the support area size428(e.g., a quantity of the unit pixels310within the supported area) can be greater than the qualifying count424. As such, the support area size428effectively represents separations between the outermost edges/corners where the support is provided. Because wider supports are preferred (e.g., where portions of the support area outlines426are greater than the overlap area414of the object for reducing overhangs and/or improving stability), the support threshold418can correspond to a minimum number of the unit pixels310in the supported area (e.g., for evaluating the support area outlines426), thereby effectively evaluating a separation between the outermost edges/corners where the support is provided.

In some embodiments, the support threshold418can be for evaluating the support ratio430, which can be calculated based on comparing the qualifying count424and/or the support area size428to the overlapped area414. For example, the support ratio430can include a ratio between the qualifying count424and the overlapped area414for representing horizontal stability, supported weight concentration, or a combination thereof. Also, the support ratio430can include a ratio between the support area size428and the overlapped area414for representing relative widths between supporting edges/corners under the target object112.

Further, the robotic system100can further evaluate the candidate positions360based on the CoM location432of the target object112. In some embodiments, the robotic system100can access the CoM location432of the target object112from the master data252ofFIG.2and/or dynamically estimate the CoM location432based on gripping and/or lifting the target object112. Once accessed/estimated, the robotic system100can compare the CoM location432to the support area outlines426. The robotic system100can require the candidate position360to include the CoM location432within the support area outlines426and eliminate/disqualify the candidate positions360that fail to satisfy such requirement. In one or more embodiments, the robotic system100can calculate and evaluate a placement score based on separation distances (e.g., along the x and/or the y axes) between the CoM location432and the support area outlines426.

The robotic system100can use the support parameters410to evaluate constraints/requirements. For example, the robotic system100can eliminate/disqualify the candidate positions that do not satisfy the support threshold418, a CoM location threshold (e.g., a requirement to include the CoM location432within the support area outlines426), and/or other stacking rules. Also, the robotic system100can use the support parameters410to calculate the placement scores for the candidate positions360(e.g., the locations that satisfy the constraints) according to predetermined weights and/or equations. As described in detail below, the robotic system100can use the calculated placement score to rank the candidate positions360according to the predetermined preferences (e.g., as reflected by the weights/equations).

Object Placement Operation

FIG.5is a top view illustrating an example placement executed by the robotic system100in accordance with one or more embodiments of the present disclosure. In some embodiments, the robotic system100can include and/or communicate with a robotic arm502(e.g., a portion of the transfer unit104ofFIG.1, such as a palletizing robot) configured to transfer the target object112from the start location114and place it at the derived placement location350at the task location116. For example, the robotic system100can operate the robotic arm502to grip and pick up the target object112from a designated location/portion on a conveyor and place the target object112on a pallet.

The robotic system100can dynamically derive the placement location350, e.g., as the target object112arrives at the facility and/or the start location114and/or after initially starting one or more operations, such as the packing operation. The robotic system100can dynamically derive the placement location350based on or to account for one or more uncertainty factors, such as an absence of a packing plan (e.g. a plan for representing placement locations350derived for a set of objects, including the target object112, at the task location116), an error in arriving objects (e.g., when the object doesn't match an expected/known object or sequence), or a combination thereof. The robotic system100can also dynamically derive the placement location350based on or to account for one or more uncertainties at the destination, such as due to previously placed objects508(e.g., unrecognizable and/or unexpected packages on the pallet) and/or a shift in one or more of the previously placed objects508.

In some embodiments, the robotic system100can dynamically derive the placement location350based on data (e.g., image data and/or measurement data) dynamically gathered via one or more of the sensors216ofFIG.2(e.g., the imaging devices222ofFIG.2). For example, the robotic system100can include and/or communicate with a source sensor504(e.g., a 3D camera) located over the start location114and/or an incoming path (e.g., conveyor). The robotic system100can use the data from the source sensor504to generate and/or access the discretized object models302ofFIG.3A. In one or more embodiments, the robotic system100can image the objects and/or measure one or more dimensions of the objects using the source sensor504. The robotic system100can compare the images and/or the measurements to the master data252ofFIG.2to identify the incoming objects. Based on the identification, the robotic system100can access the discretized object models302associated with the objects. In one or more embodiments, the robotic system100can dynamically generate the discretized object models302based on dividing the images/dimensions according to the unit pixel310as described above.

Also, the robotic system100can include and/or communicate with a destination sensor506(e.g., a 3D camera) located over the task location116. The robotic system100can use the data from the destination sensor506to determine and dynamically update the discretized platform models304ofFIG.3B. In one or more embodiments, the robotic system100can image and/or measure one or more dimensions of the placement area (e.g., the task location116, such as a pallet). The robotic system100can use the image and/or measurements to identify, access, and/or generate the discretized platform models304, similarly as described above for the discretized object models302. Further, the robotic system100can use the data (e.g., the depth map) from the destination sensor506to determine the height measures402ofFIG.4A. Accordingly, the robotic system100can use the height measures402to update the placement area340and the discretized platform models304in real time. For example, the robotic system100can update the height measures402according to the previously placed objects508, such as after placing the target object112at the placement location350.

The robotic system100can derive an approach path510for transferring the target object112to the placement location350and/or each of the candidate positions360ofFIG.3B. The approach path510can correspond to a motion plan for manipulating/transferring the target object112across space from the start location114to the corresponding candidate position360. The approach path510can be in 3D extending across horizontal and/or vertical directions.

Approach Path Evaluation

FIGS.6A and6Bare profile views illustrating example approaches for placing the target object112ofFIG.1in accordance with one or more embodiments of the present disclosure.FIGS.6A and6Billustrate the approach paths510ofFIG.5for placing the target object112at the corresponding candidate position360ofFIG.3Bover one of the previously placed objects508on the task location116(e.g., a pallet).

The robotic system100ofFIG.1can derive the approach paths510based on approach increments602, which are illustrated as the dashed boxes of F-1to F-5. The approach increments602can include sequential positions of the target object112in 3D space along the corresponding approach path510. In other words, the approach increments602can correspond to sampled positions of the target object112for following the corresponding approach path510. The approach increments602can be aligned according to path segments604of the corresponding approach path510. The path segments604can correspond to linear segments/directions in the approach path510. The path segments604can include a final segment606for placing the target object112at the corresponding candidate position360. The final segment606can include a vertical (e.g., a downward) direction.

To derive the approach paths510, the robotic system100can identify any of the previously placed objects508that may potentially become an obstacle610(e.g. such as a potential obstacle when placing the target object112at the candidate position360). In one or more embodiments, the robotic system100can identify potential obstacle(s)610as instance(s) of the previously placed objects508overlapping a horizontal line611(e.g., a straight line along the x-y plane) connecting the start location114and the corresponding candidate position360. The robotic system100can further identify the potential obstacle(s)610as instance(s) of the previously placed objects508overlapping a lane613derived around the horizontal line, such as based on deriving the lane parallel to and overlapping the horizontal line and having a width based on one or more dimensions (e.g., a width, a length, and/or a height) of the target object112. As illustrated inFIGS.6A and6B, the start location114can be to the right of the candidate position360. Accordingly, the robotic system100can identify the previously placed object on the right as the potential obstacle610.

In some embodiments, the robotic system100can validate the potential obstacle610based on the height measures402ofFIG.4A. For example, the robotic system100can validate/identify the potential obstacles610with one or more of the height measures402greater than or equal to those of the candidate position360. The robotic system100can eliminate the previously placed objects508having the height measures402less than those of the candidate position360as the potential obstacles610. In one or more embodiments, the robotic system100can identify/eliminate the potential obstacles610based on an ambiguity associated with the height of the candidate position360and/or the height of the potential obstacles610.

In some embodiments, the robotic system100can derive the approach paths510in a reverse order, such as beginning from the candidate position360and ending at the start location114ofFIG.5. Accordingly, the robotic system100can derive the final segment606first (e.g., before other segments) to avoid the potential obstacles610. For example, the robotic system100can determine the approach increments602(e.g., ‘F-1’ first, then ‘F-2’, etc.) based on iteratively increasing the height of the approach increments602by a predetermined distance. For each iteration, the robotic system100can calculate and analyze a vector612between the determined approach increment602(e.g., a bottom surface/edge thereof) and the potential obstacles610(e.g., a top surface/edge thereof). The robotic system100can continue to increase the height of the approach increments602until the vector612indicates that the determined approach increment602is above the potential obstacles610and/or clears the potential obstacles610by a clearance threshold614(e.g., a requirement for a minimum vertical separation for the target object112above a highest point of the potential obstacles610to avoid contact or collision between the target object112and the potential obstacle610). When the determined approach increment602satisfies the clearance threshold614or for the following iteration, the robotic system100can adjust the corresponding approach increment602along a horizontal direction (e.g., toward the start location114) by a predetermined distance. Accordingly, the robotic system100can derive the final segment606and/or the subsequent path segments604based on the candidate position360and the approach increment602that satisfied the clearance threshold614to derive the approach paths510.

Once derived, the robotic system100can use the approach paths510to evaluate the corresponding candidate positions360. In some embodiments, the robotic system100can calculate the placement score according to the approach paths510. For example, the robotic system100can calculate the placement score according to a preference (e.g., according to one or more weights that correspond to predetermined placement preferences) for a shorter length/distance for the final/vertical segment606. Accordingly, in comparing the approach paths510ofFIGS.6A and6B, the robotic system100can prefer the path illustrated inFIG.6B, which has a shorter length of the final/vertical segment606. In one or more embodiments, the robotic system100can include a constraint, such as a maximum limit, associated with the approach paths510(e.g., for the final/vertical segment606) used to eliminate or disqualify candidate positions360.

In some embodiments, the robotic system100can further evaluate the corresponding candidate positions360according to other collision/obstruction related parameters. For example, the robotic system100can evaluate the candidate positions360according to horizontal separations616between the candidate positions360and one or more of the previously placed objects508. Each of the horizontal separations616can be a distance (e.g., a shortest distance) along a horizontal direction (e.g., x-y plane) between the corresponding candidate position360and an adjacent instance of the previously placed objects508. The robotic system100can calculate the placement scores for the candidate positions360based on the horizontal separation616similarly as described above for the approach paths510. Also, the robotic system100can eliminate or disqualify candidate positions360based on the horizontal separation616, such as when the horizontal separation616fails a minimum requirement. Details regarding the placement score calculation and/or the constraints for eliminating the candidate positions360are discussed below.

Operational Flow

FIG.7is a flow diagram for a method700of operating the robotic system100ofFIG.1in accordance with one or more embodiments of the present technology. The method700can be for dynamically deriving the placement location350ofFIG.3Bon the task location116ofFIG.1for placing the target object112ofFIG.1. The method700can be implemented based on executing the instructions stored on one or more of the storage devices204ofFIG.2with one or more of the processors202ofFIG.2.

At block702, the robotic system100can identify real-time packaging conditions. For example, as illustrated at block732, the robotic system100can analyze incoming objects and/or the task location116ofFIG.1in real time. The robotic system100can receive and analyze sensor data from the sensors216ofFIG.2. In some embodiments, the robotic system100can receive (e.g., from the source sensor504ofFIG.5) and analyze source sensor data that represents the target object112ofFIG.1that is at or approaching the start location114ofFIG.1. Also, the robotic system100can receive (e.g., from the destination sensor506ofFIG.5) and analyze destination sensor data representing a placement area (e.g., the placement area340ofFIG.3Bthereon) associated with the task location116and/or the previously placed objects508ofFIG.5thereon.

In some embodiments, such as illustrated at block734, the robotic system100can analyze the sensor data to determine one or more uncertainty factors. For example, the robotic system100can compare the sensor data to a previously derived (via, e.g., an offline computation and/or a real-time computation at the applicable facility) packing plan that specifies placement locations for incoming objects, including the target object112. Accordingly, some instances of the uncertainties can be based on mismatches in the sensor data and the packing plan or an associated arrival sequence.

In analyzing the sensor data for uncertainties, as illustrated at block736, the robotic system100can process the sensor data (e.g., images and/or depth maps) to identify/estimate edges. For example, the robotic system100can process the sensor data, such as using Sobel filters, to recognize edges of the target object112, the task location116, the previously placed objects508, or a combination thereof. The robotic system100can use the edges to identify areas that represent separate objects and/or dimensions thereof.

In some instances, the mismatches can include source matching errors resulting from comparing the source sensor data to the master data252ofFIG.2, an access/arrival sequence associated with the packing plan, or a combination thereof. The source matching errors can result, for example, from misidentifying (e.g., when the source sensor data does not correspond to any objects in the packing plan and/or the master data252) the incoming object and/or from the incoming object being out of sequence and not matching the expected arrival/access sequence for the packing plan. Also, the mismatches can include destination matching errors resulting from comparing the destination sensor data to the packing plan. The destination matching errors can be caused by, for example, one or more of the previously placed objects508being at unanticipated locations (i.e., not matching the packing plan), such as due to a shift in the package. Other examples can include the container for the task location116not being fully open to receive the packages and/or having unexpected items therein upon arrival.

In one or more embodiments, the robotic system100can determine the uncertainties based on further triggers. For example, the robotic system100can determine the uncertainties based on an absence of the packing plan. Also, for example, the robotic system100can determine the uncertainties based on operational status or events, such as a collision event (e.g., when robotic units and/or objects collide), an object loss event (e.g., when objects are dropped during transport/manipulation), an object-shift event (e.g., when an object shifts after placement), or a combination thereof. As described in further detail below, the robotic system100can dynamically derive the placement location350ofFIG.3for the target object112in response to and/or to compensate for the uncertainties.

At block704, the robotic system100can generate and/or access discretized models (e.g., the discretized object models302ofFIG.3Aand/or the discretized platform models304ofFIG.3B) that represent the incoming packages (including, e.g., the target object112) and/or the task location116, such as the pallet and/or the cage.

The robotic system100can determine (e.g., generate and/or access) the discretized models (e.g. discretized object model302and/or the discretized platform models304) based on the real-time sensor data (e.g., the source sensor data and/or the destination sensor data). In some embodiments, the robotic system100can identify an object type (e.g., an identification or a category for the incoming object) for objects, such as the target object112, based the source sensor data. The robotic system100can search the master data252to match an imaged surface to surface images corresponding to the object type in checking for the uncertainties as described above. In some embodiments, the robotic system100can also estimate one or more dimensions or lengths of the sensed object (e.g., the incoming object, the target object112, the pallet, the cage, etc.) based on the sensor data (e.g., source sensor data) in checking for the uncertainties. The robotic system100can use the identifying information to access the discretized models stored in the storage devices ofFIG.2and/or another device (e.g., a storage device, a database, and/or a server of a package supplier accessed via the communication devices206ofFIG.2). For example, the robotic system100can search the master data252using the identifying information (e.g., the surface image and/or the estimated dimensions) to find and access matching discretized models.

In some embodiments, the robotic system100can generate the discretized models in real time, such as directly in response to receiving the source sensor data and/or determining the uncertainties. To dynamically generate the discretized models, the robotic system100can divide the sensor data and/or corresponding physical dimensions (e.g., for the incoming object, the pallet top surface, etc.) according to the unit pixel310ofFIG.3B. In other words, the robotic system100can generate the discretized models based on overlaying the unit pixels310over an area representative of the target object112and/or the task location116according to the corresponding sensor data. The unit pixel310can be predetermined (by, e.g., a manufacturer, an ordering customer, and/or an operator), such as at 1 mm or 1/16 inches (in) or greater (e.g., at 5 mm or 20 mm). In some embodiments, the unit pixel310can be based (e.g., a percentage or a fraction) on a dimension or a size of one or more of the packages and/or the platform.

At block706, the robotic system100can derive a set of candidate positions (e.g., the candidate position360ofFIG.3B) for placing the target object112at/over the task location116. The robotic system100can derive the candidate positions360based on overlapping the discretized object model302of the target object112over the discretized platform model304at corresponding locations in/over the task location116. The candidate positions360can correspond to locations of the discretized object models302along a horizontal plane and over/within the discretized platform model304. The robotic system100can derive the candidate positions360overlapping and/or adjacent to the previously placed objects508.

In some embodiments, the robotic system100can iteratively determine the locations of the discretized object model302based on determining an initial placement location (e.g., a predetermined location for an instance of the candidate position360, such as a designated corner of the placement area). The robotic system100can determine subsequent candidate positions360according to a predetermined direction for deriving the next candidate positions360, a separation requirement between the candidate positions360across iterations, a rule/condition governing the placement, a limit on the total number of the candidate positions360, one or more patterns thereof, or a combination thereof. Further, the robotic system100can include a set of preferences and/or rules for determining the candidate positions360relative to the previously placed objects508. For example, the robotic system100can be configured with preferences toward (e.g., for performing the function earlier than most other types/categories of the candidate position360) determining the candidate positions360where the discretized object model302is adjacent to or abutting one or more edges of the previously placed objects508and/or a peripheral boundary/edge of the placement area340. Also, the robotic system100can be configured with preferences toward determining the candidate positions360where the discretized object model302is over the previously placed objects508and fits within one of the objects and/or overlaps with one or more edges of the objects.

The robotic system100can derive the candidate positions360according to predetermined rules, patterns, limits, and/or sequences for placing the discretized object model302. For example, the robotic system100can derive the candidate positions360based on a preference for the object edges, such as adjacent to and/or within a predetermined distance limit from outer-most edges of the previously placed objects508. Also, the robotic system100can derive the candidate positions360based on a preference for outer edges/borders for the placement area340, such as where the discretized object model302is nearest to or abuts the borders/edges of the pallet, cage, etc. Also, the robotic system100can derive the candidate positions360overlapping the previously placed objects508.

At block708, the robotic system100can determine/update real-time conditions of the placement area340ofFIG.3B, such as for 3D stacking evaluations. For example, the robotic system100can use the destination sensor data to determine the height measures402ofFIG.4. The robotic system100can use the depth measures derived from the destination sensor data and known height of the task location116and/or the sensor to calculate heights of the top surface(s) at the task location116. The robotic system100can match the calculated heights to the unit pixels310in the discretized platform model304and assign the maximum calculated height within the unit pixel310as the corresponding height measure402. In some embodiments, the robotic system100can determine the height measures402for the unit pixels310overlapped by the discretized object model302in the candidate positions360.

At block710, the robotic system100can evaluate the candidate positions360. In some embodiments, the robotic system100can evaluate the candidate positions360according to real-time conditions, processing results, predetermined rules and/or parameters, or a combination thereof. For example, the robotic system100can evaluate the candidate positions360based on calculating corresponding placement scores, validating/qualifying the candidate positions360, or a combination thereof.

At block742, the robotic system100can calculate the placement score for each of the candidate positions360. The robotic system100can calculate the placement score according to one or more of the placement conditions. For example, the robotic system100can use placement preferences (via, e.g., multiplier weights) and/or equations to describe preferences for: separation distances between packages, differences in package dimensions/fragility ratings/package weights for horizontally adjacent packages, the collision probabilities (based on, e.g., the approach paths510ofFIG.5or a characteristic thereof and/or the horizontal separation616ofFIG.6), continuous/adjacent surfaces at the same height, a statistical result thereof (e.g., average, maximum, minimum, standard deviation, etc.), or a combination thereof. Other examples of the placement preferences can include a resulting height, a proximity measure, an edge-placement status, a maximum supportable weight, the object type, a supported weight ratio, or a combination thereof. Accordingly, in some embodiments, the robotic system100can include the processing weights/multipliers that represent preferences for lower maximum heights, for placing the target object112nearby a border of an already placed object or an edge of the placement platform, for minimizing a difference between heights and/or maximum supportable weights of adjacent objects, for reducing a ratio between supported weight and maximum supportable weight for objects overlapped by the target object112, for matching object types for adjacent objects, or a combination thereof. Each placement location can be scored according to the preference factors and/or the equations that are predefined by a system manufacturer, an order, and/or a system operator.

In some embodiments, for example, the robotic system100can calculate the placement scores based on support measures for the candidate positions360. The robotic system100can calculate the amount of support (e.g., in stacking objects) for one or more of the candidate positions360at least partially based on the height measures402. As an illustrative example, the robotic system100can calculate the amount of support based on identifying the maximum height420ofFIG.4Bfor each of the candidate positions360. Based on the maximum height420and the height difference threshold416ofFIG.4B, the robotic system100can calculate the lower height limit422ofFIG.4Bfor each of the candidate positions360. The robotic system100can compare the height measures402of the candidate positions360to the corresponding lower height limits422to identify the supporting locations442ofFIG.4Bfor each of the candidate positions360. The robotic system100can calculate the placement score for each of the candidate positions360based on the qualifying count424ofFIG.4Bof the corresponding supporting locations442.

In one or more embodiments, the robotic system100can calculate the placement scores based on deriving the support area outlines426ofFIG.4Bfor the candidate positions360. As described above, the robotic system100can derive the set of the support area outlines426for each of the candidate positions360based on extending outer edges and/or connecting corners of outermost/peripheral instances of the supporting locations442in the corresponding position. Based on the supporting locations442, the robotic system100can determine the support area size428ofFIG.4Band/or the support ratio430ofFIG.4Bfor calculating the placement score. Also, the robotic system100can calculate a lowest separation distance between the CoM location432and the support area outlines426. The robotic system100can use the support area size428, the support ratio430, the lowest separation distance, corresponding preference weights, or a combination thereof to calculate the placement score for the corresponding candidate position.

In one or more embodiments, the robotic system100can calculate the placement scores based on deriving the approach paths510for the candidate positions360as described above. The robotic system100can calculate the placement score for each of the candidate positions360according to the final segment606ofFIG.6(e.g., a length thereof), a quantity/length of one or more path segments604ofFIG.6, or a combination thereof. In some embodiments, the robotic system100can calculate the placement scores based on the horizontal separation616ofFIG.6for the candidate positions360.

In some embodiments, as illustrated at block744, the robotic system100can qualify the candidate positions360. The robotic system100can qualify the candidate positions360based on dynamically deriving a validated set of the candidate positions360according to one or more placement constraints. In deriving the validated set, the robotic system100can eliminate or disqualify instances of the candidate positions360that violate or fail to satisfy one or more of the placement constraints that are, at least partially, associated with the height measures402. In one or more embodiments, the robotic system100can derive the validated set first and then calculate the placement scores for the validated set. In one or more embodiments, the robotic system100can derive the validated set concurrently with calculating the placement scores.

In one or more embodiments, the placement constraints can be associated with comparing the qualifying count424, the set of support area outlines426, the support area size428, the support ratio430, the CoM location432, the approach paths510, the horizontal separation616, or a combination thereof to a threshold (e.g., the support threshold418ofFIG.4B) or a requirement. For example, the robotic system100can derive the validated set to include positions with the qualifying count424, the support area size428, and/or the support ratio430satisfying/exceeding a corresponding threshold. Also, the robotic system100can derive the validated set to include the positions having the CoM location432within/surrounded by the support area outlines426and/or satisfying a minimum separation distance from the support area outlines426. Also, the robotic system100can derive the validated set to include the positions having the approach path510(e.g., the final segment606therein) satisfying a maximum length threshold and/or having the horizontal separation616satisfying a minimum threshold.

At block712, the robotic system100can dynamically derive the placement location350for placing the target object112over/at the task location116. The robotic system100can dynamically derive the placement location350based on selecting one of the positions in the validated set or the candidate positions360according to the placement scores. In some embodiments, the robotic system100can track the candidate positions360using a heap structure. Accordingly, the robotic system100can remove positions from the heap structures when the positions violate constraints as described above. Further, the robotic system100can sequence or rank the tracked positions according to the corresponding placement scores. In some embodiments, the robotic system100can continuously sequence the tracked positions as the placement scores or iterative updates of the scores are being computed. As a result, the robotic system100can select the position at the designated location (e.g., first slot) in the heap structure as the placement location350when the score computations conclude.

At block714, the robotic system100can place the target object112at the derived placement location350. In placing the target object112at the placement location350, one or more components/devices of the robotic system100can communicate with and/or operate other components/devices. For example, one or more of the processors202and/or a stand-alone controller (such as, e.g., a warehouse/shipping center control device) can send information, such as the placement location350, a corresponding motion plan, a set of commands and/or settings for operating the actuation devices212ofFIG.2and/or the transport motor214ofFIG.2, or a combination thereof, to the other components/devices. The other components/devices, such as other instances of the processors202and/or the robotic arm502ofFIG.5, the actuation devices212, the transport motor214, and/or other external devices/systems, can receive the information and execute corresponding functions to manipulate (e.g., grip and pick up, transfer and/or reorient across space, place at destination, and/or release) the target object112and place it at the placement location.

In some embodiments, the robotic system100can update or re-identify real-time packaging conditions after placing the target object112. In other words, following block714, the control flow can move to block702. Accordingly, the robotic system100can update/identify the next incoming object as the target object112. The robotic system100can also update information for placement area340and/or the previously placed objects508thereon to include the recently placed object. In other embodiments, the robotic system100can recalculate or adjust the packing plan and/or resume according to the packing plan after placing the target object112.

Discretization of the tasks and the 2D/3D layering described above provides improved efficiency, speed, and accuracy for packing objects. Accordingly, the reduction in operator inputs and the increase in accuracy can further decrease human labor for the automated packing process. In some environments, the robotic system100as described above can eliminate the necessity of sequencing buffers, which can cost around or over $1 million US.

Further, the dynamic computation of the placement location350according to real-time conditions (e.g., as represented by the sensor data and other status/data) provides reduced operational errors. As described above, the robotic system can account for and resolve uncertainties introduced by unexpected conditions/events without requiring human intervention. Moreover, the embodiments described above can stack the objects in 3D without a pre-existing packing plan, such as by dynamically deriving the placement locations350as the objects arrive at the start location114. In comparison to traditional systems that are limited to 2D dynamic packing (i.e., placing objects directly on the platform as a single layer), the consideration of height can allow the embodiments described above to stack the objects on top of each other and increase the packing density.

CONCLUSION

The above Detailed Description of examples of the disclosed technology is not intended to be exhaustive or to limit the disclosed technology to the precise form disclosed above. While specific examples for the disclosed technology are described above for illustrative purposes, various equivalent modifications are possible within the scope of the disclosed technology, as those skilled in the relevant art will recognize. For example, while processes or blocks are presented in a given order, alternative implementations may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or sub-combinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed or implemented in parallel, or may be performed at different times. Further, any specific numbers noted herein are only examples; alternative implementations may employ differing values or ranges.

These and other changes can be made to the disclosed technology in light of the above Detailed Description. While the Detailed Description describes certain examples of the disclosed technology as well as the best mode contemplated, the disclosed technology can be practiced in many ways, no matter how detailed the above description appears in text. Details of the system may vary considerably in its specific implementation, while still being encompassed by the technology disclosed herein. As noted above, particular terminology used when describing certain features or aspects of the disclosed technology should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the disclosed technology with which that terminology is associated. Accordingly, the invention is not limited, except as by the appended claims. In general, the terms used in the following claims should not be construed to limit the disclosed technology to the specific examples disclosed in the specification, unless the above Detailed Description section explicitly defines such terms.

Although certain aspects of the invention are presented below in certain claim forms, the applicant contemplates the various aspects of the invention in any number of claim forms. Accordingly, the applicant reserves the right to pursue additional claims after filing this application to pursue such additional claim forms, in either this application or in a continuing application.