SCENE UNDERSTANDING USING LANGUAGE MODELS FOR ROBOTICS SYSTEMS AND APPLICATIONS

In various examples, 3D object knowledge can be developed to extract diverse knowledge from large language models, and a part-grounding model can be trained to ground part semantics in terms of local shape features and spatial relations between parts. For example, knowledge that “the opening part of a mug that affords the pouring action is located on the top of the mug body and is often circular” can be grounded by identifying a previously unknown “opening” part based on its spatial relation to the known “body” part and its circular shape. A robotic system, for example, may use a model to identify an unlabeled part of a 3D object in imaging data. The model may be generated using natural language descriptions of relationships between parts of 3D objects, with descriptions generated using a language model that produces text in response to queries related to spatial relationships between the parts.

BACKGROUND

Large-scale knowledge bases for robotic systems provide knowledge about objects. However, there may be knowledge missing in these knowledge bases, and so understanding a robotic system's surroundings may require making inferences based on known facts. Some methods, however, only perform reasoning at the textual level (e.g., relying on detection methods with predefined word labels), do not localize parts, and/or are not able to perform segmentation of objects into constituent parts-especially for objects that lack a uniquely distinct shape. Some methods involve learning visual representations for specific tasks, but these methods lack the necessary abstraction to transfer between tasks. In general, prior approaches model spatial relations and object properties of whole objects, and lack the ability to generalize to novel object categories and tasks.

SUMMARY

Embodiments of the present disclosure relate to grounding knowledge of objects in three dimensions for generalizable manipulation. Systems and methods are disclosed that can generate or deploy machine learning models to identify unlabeled regions of three-dimensional (3D) objects in imaging data, allowing for more effective interaction with the 3D objects.

In contrast to conventional systems, embodiments of the disclosed approach can develop 3D object knowledge to extract diverse knowledge from language models—such as large language models (LLMs)—and can learn a part-grounding model to ground part semantics in terms of local shape features and spatial relations between parts. For example, the knowledge that “the opening part of a mug that affords the pouring action is located on the top of the mug body and is often circular” can be grounded by identifying a previously unknown “opening” part based on its spatial relation to the known “body” part and its circular shape.

At least one aspect relates to a processor. In various embodiments, the processor can comprise, or can be, one or more circuits. The one or more circuits may segment, based at least on a query, a 3D object in imaging data, the 3D object comprising an unlabeled region. The segmenting may comprise providing, to a model, imaging data comprising the 3D object, and receiving, from the model, an identification of the unlabeled region of the 3D object in the imaging data.

In various embodiments, the identification comprises, or is, a segmentation mask. In various embodiments, the identification comprises, or is, a pointwise label. In various embodiments, the identification comprises, or is, a set of pixels. In various embodiments, the query comprises, or is, a task to be performed. In various embodiments, the query corresponds to, or is, an interaction with the 3D object. In various embodiments, the query corresponds to, or is, an interaction between the 3D object and a second 3D object. In various embodiments, the query is provided to the model to obtain the identification of the unlabeled region. In various embodiments, the model is updated using training data comprising natural language descriptions of relationships between a plurality of parts of the 3D object. In various embodiments, the plurality of parts of the 3D object are obtained using a dataset of 3D objects annotated with hierarchical 3D part information. In various embodiments, the natural language descriptions of the relationships are generated at least in part using a language model that produces human-like text. In various embodiments, the language model comprises a generative transformer network that provides the natural language descriptions in response to queries that are related to spatial relationships between the plurality of parts of the 3D object. In various embodiments, the one or more circuits are to generate an instruction to cause an interaction with the 3D object based at least on the identification of the unlabeled portion. In various embodiments, the one or more circuits are to receive, prior to segmenting the 3D object in the imaging data, an action to be performed with respect to the 3D object, and generate the query based at least on the action.

Another aspect relates to a processor. The processor can be, or can comprise, one or more circuits. The one or more circuits may generate, for a 3D object, training data comprising natural language descriptions of relationships between a plurality of parts of the 3D object, the natural language descriptions generated at least in part using a language model that produces human-like text in response to queries, the natural language descriptions generated by providing the language model queries related to spatial relationships between the plurality of parts of the 3D object. The one or more circuits may update, using the training data, a model to segment the 3D object in imaging data by receiving the imaging data and providing an identification of an unlabeled region of the 3D object in the imaging data.

In various embodiments, the language model comprises a generative transformer network. In various embodiments, the one or more circuits are to obtain the plurality of parts of the 3D object from a dataset of 3D objects annotated with hierarchical 3D part information. In various embodiments, the one or more circuits are to use the model to identify, in second imaging data, an unlabeled region of (i) the 3D object or (ii) a second 3D object. In various embodiments, the model is trained to segment, in second imaging data, the 3D object or a second 3D object. In various embodiments, the 3D object or the second 3D object is segmented based on a query corresponding to an interaction between at least two of: (i) the 3D object, (ii) the second 3D object, or (iii) a third 3D object. In various embodiments, the one or more circuits are to use the model by providing, to the model, second imaging data comprising a second 3D object, and receiving, from the model, at least one of a segmentation mask, a pointwise label, or a set of pixels corresponding to an unlabeled region of the second 3D object.

In various embodiments, the processors, systems, and/or methods described herein can be implemented by or via, or can be included in, at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for three-dimensional (3D) assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more large language models (LLMs)—which may process text, audio, sensor (e.g., image, LiDAR, etc.) data to generate one or more outputs, a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

DETAILED DESCRIPTION

Systems and methods are disclosed related to grounding knowledge of objects in three dimensions for generalizable manipulation.

Robotic systems, or other autonomous and/or semi-autonomous systems, can more effectively interact with their surroundings if they have a better understanding of objects in their environment. If a robotic system determines that a task is to be performed with respect to an object, or receives a command to perform the task from another system, the system might need to make inferences to perform the task. For example, if a robotic system decides that a cup is to be placed on a counter or in a dishwasher, or receives a command to place the cup on the counter or in the dishwasher, the system might use a vision system to look for and identify a cup, but if the system has not previously recognized a particular object (with its unique shape) in its surroundings as being a cup, the system might first need to make an inference that the particular object is a cup. Once the robotic system identifies the cup (or if it was previously identified as being a cup), the system may determine how to interact with the object. For example, the cup may need to be grasped, moved from its origin, and placed at its destination. However, and especially if the cup is currently not empty, the robotic system may not be able to simply grasp the cup at any part or in any orientation (without an unwanted consequence such as spilling the cup's contents). For example, depending on the center of gravity of the cup, the system might need to grasp the cup at a particular part of the cup. Even with an empty cup, the system might, for example, place the cup on the counter on its side rather than on its bottom, or place the cup in the dishwasher with the opening of the cup facing up rather than down. While placing the cup on the counter on its side, or placing the cup in the dishwasher with its opening up, might technically accomplish the task of moving the cup from one place to another, performing the tasks in this way might not be desirable for one or more reasons. For example, if the cup is placed on a counter on its side, the cup might roll off the counter, or if the cup is placed in a dishwasher with its opening up, the cup might not get cleaned by the water jets of the dishwasher, and may be left with sitting water at the end of the dishwasher cycle.

Some large-scale knowledge bases for robotic systems, such as “KnowRob” and “RoboBrain”, can provide rich knowledge about objects. Computational frameworks such as knowledge embeddings (which use representations of text in which the meanings of words are encoded such that words closer together in a vector space are expected to be more similar in meaning) and transformer networks (which use artificial neural networks to generalize knowledge from observations of objects) have been introduced to learn relations between object properties and infer missing knowledge based on known facts. Knowledge about object affordance (which can identify how an object is used) and geometries has been extracted from large language models (LLMs) (which involve natural language processing). LLMs have also facilitated task planning which requires understanding the states and functions of objects. These methods, however, only perform reasoning at the textual level or rely on detection methods with predefined word labels.

Representations of the different parts of objects have been introduced to 3D perception for retrieving object models given language descriptions, and conversely generating language descriptions of object models. However, these language reference tasks do not require localizing parts (e.g., subparts). Other methods examine discovering parts without regard for segmenting parts, such as the “PartNet” dataset that is often used to determine the names of the parts of objects. However, such segmenting of objects into constituent parts is limited when there are not distinct shape features that can be differentiated and segmented.

Other approaches seek to use structured sensorimotor representations for integrating perception with manipulation, which may combine semantic representations (the “what”) with the spatial indicators (the “where”) to aid manipulation of objects by robots. Vision-based object representations, such as affordance segmentation (which attempts to detect, simultaneously, multiple objects and their uses from images) and “keypoints” (which provide semantic representations of objects while ignoring details not relevant to a task), enable robots to generalize skills to newly-perceived objects based on what categories the objects belong to. Several task-oriented grasping methods have leveraged these visual representations to specify where to grasp an object and which part of the object to interact with the environment. However, a common issue of these methods is that the representations are learned for specific tasks without the necessary abstraction to easily transfer between tasks. Understanding spatio-semantic relations, such as “left” or “contain” (e.g., hold inside of) has also shown to be useful for many manipulation-based applications such as retrieval of objects and moving of objects. Perception that is “interactive” uses different sensor data (e.g., touch and sound) and exploratory actions (e.g., press and shake to better understand an object) to make sense of object properties such as “thin,” “rough,” and “compressible.” In general, however, such methods model spatial relations and object properties of whole objects rather than parts and subparts of objects, and lack the ability to generalize to novel object categories and tasks to other objects and tasks.

Embodiments of the disclosed approach may comprise developing 3D object knowledge by extracting diverse knowledge from LLMs and learning a part-grounding model to ground part semantics (e.g., the meaning and logic of the parts of objects) in terms of local shape features and spatial relations between parts of objects. For example, the knowledge that “the opening part of a mug that affords the pouring action is located on the top of the mug body and is often circular” can be grounded by identifying a previously unknown “opening” part based on its spatial relation to the known “body” part and its circular shape. Formally, the knowledge can be defined as “unary” facts specifying properties of local parts (e.g., (query part, circular)) and binary facts specifying spatial relations (e.g., (query part, on top of, anchor part)). Once there is an initial set of parts to anchor the system's understanding (which can be a small number of parts), new parts can be derived with spatial and geometric reasoning based on an understanding of the initial “anchor” parts.

Embodiments of the disclosed approach may involve building a knowledge base of objects with fine-grained correspondence between semantic properties and perceivable object features. Grounding of concepts on parts of 3D objects, and reasoning about spatial relations between these parts (e.g., identifying the opening part of a bottle located on the top part of the body), may be accomplished. 3D concepts can be used to define specific manipulation actions such as grasping a particular part of an object for a specific task (e.g., hand over a pair of scissors by grasping the blades) and placing an object in a specific pose or orientation (e.g., place the mug upside down in the dishwasher). 3D concepts can provide an abstraction for generalizing manipulation between object categories (e.g., a bowl should be placed in a dishwasher similarly to a mug with its opening facing down) and tasks (e.g., safely handing over a pair of scissors and using a pair of scissors rely on recognizing the same sharp part). Additional aspects, features, and advantages will be discussed below.

The systems and methods described herein may be used for a variety of purposes, by way of example and without limitation, for machine control, machine locomotion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environment simulation, object or actor simulation and/or digital twinning, data center processing, conversational AI, light transport simulation (e.g., ray-tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing and/or any other suitable applications.

Turning toFIG.1, depicted is an example process in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements, steps, and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) may be involved in addition to or instead of those shown, and certain elements or steps may be omitted altogether. Further, many of the elements employed to implement the disclosed approach described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities may be carried out by hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory.

Each block of method100inFIG.1, described herein, comprises a computing process that may be performed using any combination of hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory. The method may also be embodied as computer-usable instructions stored on computer storage media. The method may be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Method100may be executed by any one system, or any combination of systems, including, but not limited to, those described herein.

FIG.1is a flow diagram showing a method100for generating one or more models to identify unknown (e.g., unlabeled) or insufficiently-understood regions of 3D objects, in accordance with some embodiments of the present disclosure. The method100, at block110, includes training or otherwise updating one or more models in generating a model that is capable of detecting an unrecognized region (e.g., part) of a 3D object in, for example, imaging data obtained by a vision system of a robotic or other autonomous or semi-autonomous system. The models may be, or may comprise, one or more neural networks. At block140, method100includes receiving a query (e.g., a task to be performed, and/or an interaction by the robotic system with one or more objects in the system's environment or otherwise reachable by the robotic system). At block150, method100includes using the generated model(s) to detect a previously unknown region of the 3D object in imaging data as part of, or in furtherance of, the performance of the task.

In accordance with some embodiments of the present disclosure, method100, at block115, includes generating training data for training, updating, or otherwise generating one or more models. At120, as part of obtaining training data, a set of queries may be provided to one or more language models. The queries may be related to, for example, spatial relationships between parts of a 3D object. The language models may provide natural language descriptions of the 3D objects. The training data may comprise, or may be, the natural language descriptions from the language models. The names of the parts (between which spatial relationships are sought) may be obtained using one or more datasets of 3D objects. The datasets may include objects that are annotated with hierarchical 3D part information. As an example, such datasets may be, or may comprise, PartNet (https://partnet.cs.stanford.edu/). Table 1 provides example spatial relationships for parts of a bowl, bottle, and mug.

TABLE 1Spatial relationships for mugs, bowls, and bottlesPARTSSPATIAL RELATIONSHIPSBowl:bowl/bottom,1. first part is at the top of the bottle, andbowl/container,second part is at the bottom of the bottle.bowl/containing—2. first part is on the side of the bottle,thingsand second part is on the top of the bottle.Bottle:bottle/jug/body,3. first part is inside the mug, and secondbottle/jug/handle,part is attached to the mug.bottle/jug/mouth,4. first part is below second part.bottle/normal—5. first part is narrower than second part.bottle/body,6. first part is at the top of the second part,bottle/normal—and the second part is below the first part.bottle/closure,7. first part is the surface on which secondbottle/normal—part rests.bottle/handle,8. the first part of the jug is at the top,bottle/normal—and the second part is on the side.bottle/lid,9. first part is on the side of the secondbottle/normal—part, and the second part is on the top ofbottle/mouth,the first part.bottle/normal—10. first part is on the bottom, and secondbottle/neckpart is on the top.Mug:mug/body,11. first part is on top of second part.mug/containing—12. first part is on the top of second part.things,13. first part is surrounding second part.mug/handle14. first part is on the side of the mug, andsecond part is inside the mug.15. first part is on the side of the jug, andsecond part is on the top of the jug.16. first part is attached to second part.17. first part is inside second part.18. first part is at the top, and secondpart is at the bottom.19. first part is sitting on second part.20. first part is on the top of the bottle,and second part is on the bottom.21. first part is at the top of the bottle,and second part is on the side of the bottle.22. first part is in second part.23. first part is at the top of second part.

With reference also toFIG.2, in some embodiments, hierarchy of the parts of 3D objects (210) may be built from, or otherwise based on, PartNet annotations, and incomplete PartNet object models may be aligned with, for example, the ShapeNet V2 model. Annotations may indicate, for example (at220), that a mug240(“mug_0”) can be segmented into a body250(“body_1”) and a handle260(“handle_2”). Segmented point clouds (270) may be obtained, where each point (e.g., voxel) may correspond to a node in the hierarchy220, allowing for use of arbitrary levels in the hierarchy. At block125ofFIG.1, method100includes obtaining natural language descriptions of spatial relationships between parts of the 3D objects. These natural language descriptions can be obtained by querying a language model for spatial relations between parts, as further discussed below with respect toFIG.4. The language model may comprise, or may be, for example, Generative Pre-trained Transformer 3 (GPT-3). At block130of method100, the model may be trained, or otherwise updated, using the natural language descriptions obtained using one or more language models.

At block140of method100, the system may receive a query. The query may be received from, for example, a user of the robotic system, or from a device or from another system providing instructions to the robotic system. The query may be a task. For example, the task may involve interaction (e.g., by the robotic system) with one object (e.g., take an object to a destination), or interaction (e.g., by the robotic system) with more than one object, such as interaction between two objects (e.g., pour water out of a container into a cup), or interaction among more than two objects (e.g., wash a dish at a faucet using a sponge). Example tasks can be found inFIG.3, which provides, in particular, “pour from mug to bowl” at310, “pour from bowl to mug” at320, “put knife in mug” at330, “place mug under faucet” at340, and “place clock” at350, as example tasks. In various embodiments, the task to be performed is not received from a user, device, or system, but rather is identified as a task to be performed based, for example, (i) on other information available to, or obtained by, the robotic system, (ii) on other programming of the robotic system, and/or (iii) on outputs of other models involved in determining the behavior or functionality of the robotic system.

At block150, the robotic system may, based on a task, use the trained or otherwise updated model. The robotic system may comprise, or have access to, a vision system (e.g., one or more cameras, imaging optics, and/or other sensors) to obtain imaging data used to obtain information about the robotic system's surroundings. The imaging data may include an object with an unlabeled or otherwise unrecognized part. For example, the robotic system may recognize a cup in its field of view (FOV), but the robotic system may not have enough confidence that the task can be performed based only on the knowledge that the object within the FOV is a cup. The level of confidence can be determined based, for example, on past experiences of the robotic system and/or on how granular (e.g., fine-grained) the robotic system's knowledge of the 3D object is. The robotic system may, for example, not have performed the same task (or sufficiently similar task) with the same objects (or sufficiently similar objects) in the past.

At block155, the robotic system may provide the imaging data obtained via the vision system to a trained or otherwise updated model. At block160, the model may provide, in response to receiving the imaging data, an identification of a region of the 3D object in the imaging data as output to the robotic system. The region may have been previously unlabeled or otherwise not identified as a distinct part of the particular 3D object in the imaging data. The identification of the region may comprise, or may be, a segmentation mask, a pointwise label, and/or a set of pixels. The robotic system may have a better understanding of the object and/or the task because of the identification of the region from the model. The robotic system may take steps toward performing the task. The steps may include, for example, generating one or more commands or instructions to actuate one or more components of the robotic system to manipulate one or more objects in the robotic system's environment.

As suggested above, in accordance with various potential implementations, autonomous or semi-autonomous robotic systems that operate in diverse human environments need to interact with a wide range of objects while adapting to changes in task goals. Perceiving and understanding semantic properties of objects (e.g., a cup is ceramic, empty, located in kitchen, and used for drinking) has shown to enhance robot autonomy by inferring missing information in human instructions, efficiently searching for objects in homes, and manipulating objects based on their affordances and states. However, this textual representation of knowledge generalizes facts at the level of objects and lacks the granularity to localize object properties based on local geometry and functionalities (e.g., that the container part of a cup can be used to store liquid). The disclosed approach can provide a knowledge base of objects with more fine-grained (e.g., granular) correspondence between semantic properties and perceivable object features. Various embodiments provide grounding concepts on parts of 3D objects and reasoning about spatial relations between these parts (e.g., identifying the opening part of a bottle located on the top part of the body).

In various embodiments, concepts and spatial relations grounded on 3D objects can be used to define specific manipulation actions such as grasping a particular part of an object for a specific task and placing an object in a specific orientation. The symbolic representation of 3D objects also provides an abstraction for generalizing manipulation between object categories and tasks. Further, human users can leverage the vocabulary of 3D concepts to customize robot behavior and teach new knowledge about objects. Moreover, as robotic systems physically interact with objects in the environment, the symbolic knowledge can be further refined to reflect the capability of each robot and the constraint of each environment.

In various embodiments, 3D object knowledge can be defined as unary facts specifying properties of local parts (e.g., (query part, circular)) and binary facts specifying spatial relations (e.g., (query part, on top of, anchor part)). In some implementations, an assumption can be made that a small number of parts form the initial set of anchor parts. New parts can be derived, for example, via spatial and geometric reasoning. The disclosed approach can provide a part-level spatio-semantic representation for object manipulation. In various embodiments, a neural network model is used to localize object parts based on binary spatial relations and unary geometric attributes. In various embodiments, the model can comprise, or can be, one or more neural networks and/or other machine learning models. The disclosed framework can generalize part-aware pick-and-place actions and spatial interactions between objects to novel object categories and tasks.

In potential implementations, unary and binary facts grounded in 3D object models can be generated using a dataset (e.g., the PartNet dataset) and an LLM, as depicted inFIG.4, and in particular, block410(e.g., a mug handle and a mug body). At block420, given part segmentations from the dataset and their corresponding part labels (e.g., a bowl is segmented into the bottom of the bowl and a body of the bowl), unary facts can be extracted by querying the LLM for the geometric and/or functional properties of the parts (e.g., what is the shape of the bottom of a bowl?). At block430, to extract binary facts, the LLM can be queried for the prototypical spatial relation between two parts (e.g., where is the bottom of a bowl with respect to the body?).

The grounded relational facts can be used to train a part-grounding model, which takes as input the 3D representation (e.g., point cloud) of an object, the anchor object part in the form of a segmentation mask, and a set of facts specifying the query object part. These facts contain information about the geometric features of the query part and its spatial relation to the anchor object part. The model can be trained to predict the segmentation mask of the query object part. During inference, the LLM and part-grounding model may be used together to provide semantic knowledge about object parts and localize object parts that can be used to parameterize manipulation actions. Because the LLM does not have access to the object models, a latent model can be used in some implementations to account for the variance between object instances and the mismatch between language description of object parts and observation of object parts.

Example Computing Device

FIG.5is a block diagram of an example computing device(s)500suitable for use in implementing some embodiments of the present disclosure, such as generating models, providing commands (e.g., tasks) to a robotic system, etc. Computing device500may include an interconnect system502that directly or indirectly couples the following devices: memory504, one or more central processing units (CPUs)506, one or more graphics processing units (GPUs)508, a communication interface510, input/output (I/O) ports512, input/output components514, a power supply516, one or more presentation components518(e.g., display(s)), and one or more logic units520. In at least one embodiment, the computing device(s)500may comprise one or more virtual machines (VMs), and/or any of the components thereof may comprise virtual components (e.g., virtual hardware components). For non-limiting examples, one or more of the GPUs508may comprise one or more vGPUs, one or more of the CPUs506may comprise one or more vCPUs, and/or one or more of the logic units520may comprise one or more virtual logic units. As such, a computing device(s)500may include discrete components (e.g., a full GPU dedicated to the computing device500), virtual components (e.g., a portion of a GPU dedicated to the computing device500), or a combination thereof.

Although the various blocks ofFIG.5are shown as connected via the interconnect system502with lines, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component518, such as a display device, may be considered an I/O component514(e.g., if the display is a touch screen). As another example, the CPUs506and/or GPUs508may include memory (e.g., the memory504may be representative of a storage device in addition to the memory of the GPUs508, the CPUs506, and/or other components). In other words, the computing device ofFIG.5is merely illustrative. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “hand-held device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and/or other device or system types, as all are contemplated within the scope of the computing device ofFIG.5.

The interconnect system502may represent one or more links or busses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system502may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standards association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and/or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU506may be directly connected to the memory504. Further, the CPU506may be directly connected to the GPU508. Where there is direct, or point-to-point connection between components, the interconnect system502may include a PCIe link to carry out the connection. In these examples, a PCI bus need not be included in the computing device500.

The CPU(s)506may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device500to perform one or more of the methods and/or processes described herein. The CPU(s)506may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) that are capable of handling a multitude of software threads simultaneously. The CPU(s)506may include any type of processor, and may include different types of processors depending on the type of computing device500implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device500, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device500may include one or more CPUs506in addition to one or more microprocessors or supplementary co-processors, such as math co-processors.

In addition to or alternatively from the CPU(s)506, the GPU(s)508may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device500to perform one or more of the methods and/or processes described herein. One or more of the GPU(s)508may be an integrated GPU (e.g., with one or more of the CPU(s)506and/or one or more of the GPU(s)508may be a discrete GPU. In embodiments, one or more of the GPU(s)508may be a coprocessor of one or more of the CPU(s)506. The GPU(s)508may be used by the computing device500to render graphics (e.g., 3D graphics) or perform general purpose computations. For example, the GPU(s)508may be used for General-Purpose computing on GPUs (GPGPU). The GPU(s)508may include hundreds or thousands of cores that are capable of handling hundreds or thousands of software threads simultaneously. The GPU(s)508may generate pixel data for output images in response to rendering commands (e.g., rendering commands from the CPU(s)506received via a host interface). The GPU(s)508may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory may be included as part of the memory504. The GPU(s)508may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined together, each GPU508may generate pixel data or GPGPU data for different portions of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory, or may share memory with other GPUs.

In addition to or alternatively from the CPU(s)506and/or the GPU(s)508, the logic unit(s)520may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device500to perform one or more of the methods and/or processes described herein. In embodiments, the CPU(s)506, the GPU(s)508, and/or the logic unit(s)520may discretely or jointly perform any combination of the methods, processes and/or portions thereof. One or more of the logic units520may be part of and/or integrated in one or more of the CPU(s)506and/or the GPU(s)508and/or one or more of the logic units520may be discrete components or otherwise external to the CPU(s)506and/or the GPU(s)508. In embodiments, one or more of the logic units520may be a coprocessor of one or more of the CPU(s)506and/or one or more of the GPU(s)508.

The communication interface510may include one or more receivers, transmitters, and/or transceivers that enable the computing device500to communicate with other computing devices via an electronic communication network, included wired and/or wireless communications. The communication interface510may include components and functionality to enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating over Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and/or the Internet. In one or more embodiments, logic unit(s)520and/or communication interface510may include one or more data processing units (DPUs) to transmit data received over a network and/or through interconnect system502directly to (e.g., a memory of) one or more GPU(s)508.

The I/O ports512may enable the computing device500to be logically coupled to other devices including the I/O components514, the presentation component(s)518, and/or other components, some of which may be built in to (e.g., integrated in) the computing device500. Illustrative I/O components514include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I/O components514may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs may be transmitted to an appropriate network element for further processing. An NUI may implement any combination of speech recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of the computing device500. The computing device500may be include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations of these, for gesture detection and recognition. Additionally, the computing device500may include accelerometers or gyroscopes (e.g., as part of an inertia measurement unit (IMU)) that enable detection of motion. In some examples, the output of the accelerometers or gyroscopes may be used by the computing device500to render immersive augmented reality or virtual reality.

The power supply516may include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply516may provide power to the computing device500to enable the components of the computing device500to operate.

The presentation component(s)518may include a display (e.g., a monitor, a touch screen, a television screen, a heads-up-display (HUD), other display types, or a combination thereof), speakers, and/or other presentation components. The presentation component(s)518may receive data from other components (e.g., the GPU(s)508, the CPU(s)506, DPUs, etc.), and output the data (e.g., as an image, video, sound, etc.).

Example Data Center

FIG.6illustrates an example data center600that may be used in at least one embodiments of the present disclosure. The data center600may include a data center infrastructure layer610, a framework layer620, a software layer630, and/or an application layer640.

As shown inFIG.6, the data center infrastructure layer610may include a resource orchestrator612, grouped computing resources614, and node computing resources (“node C.R.s”)616(1)-616(N), where “N” represents any whole, positive integer. In at least one embodiment, node C.R.s616(1)-616(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input/output (NW I/O) devices, network switches, virtual machines (VMs), power modules, and/or cooling modules, etc. In some embodiments, one or more node C.R.s from among node C.R.s616(1)-616(N) may correspond to a server having one or more of the above-mentioned computing resources. In addition, in some embodiments, the node C.R.s616(1)-6161(N) may include one or more virtual components, such as vGPUs, vCPUs, and/or the like, and/or one or more of the node C.R.s616(1)-616(N) may correspond to a virtual machine (VM).

In at least one embodiment, grouped computing resources614may include separate groupings of node C.R.s616housed within one or more racks (not shown), or many racks housed in data centers at various geographical locations (also not shown). Separate groupings of node C.R.s616within grouped computing resources614may include grouped compute, network, memory or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R.s616including CPUs, GPUs, DPUs, and/or other processors may be grouped within one or more racks to provide compute resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and/or network switches, in any combination.

The resource orchestrator612may configure or otherwise control one or more node C.R.s616(1)-616(N) and/or grouped computing resources614. In at least one embodiment, resource orchestrator612may include a software design infrastructure (SDI) management entity for the data center600. The resource orchestrator612may include hardware, software, or some combination thereof.

In at least one embodiment, as shown inFIG.6, framework layer620may include a job scheduler628, a configuration manager634, a resource manager636, and/or a distributed file system638. The framework layer620may include a framework to support software632of software layer630and/or one or more application(s)642of application layer640. The software632or application(s)642may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud and Microsoft Azure. The framework layer620may be, but is not limited to, a type of free and open-source software web application framework such as Apache Spark™ (hereinafter “Spark”) that may utilize distributed file system638for large-scale data processing (e.g., “big data”). In at least one embodiment, job scheduler628may include a Spark driver to facilitate scheduling of workloads supported by various layers of data center600. The configuration manager634may be capable of configuring different layers such as software layer630and framework layer620including Spark and distributed file system638for supporting large-scale data processing. The resource manager636may be capable of managing clustered or grouped computing resources mapped to or allocated for support of distributed file system638and job scheduler628. In at least one embodiment, clustered or grouped computing resources may include grouped computing resource614at data center infrastructure layer610. The resource manager636may coordinate with resource orchestrator612to manage these mapped or allocated computing resources.

In at least one embodiment, software632included in software layer630may include software used by at least portions of node C.R.s616(1)-616(N), grouped computing resources614, and/or distributed file system638of framework layer620. One or more types of software may include, but are not limited to, Internet web page search software, e-mail virus scan software, database software, and streaming video content software.

In at least one embodiment, application(s)642included in application layer640may include one or more types of applications used by at least portions of node C.R.s616(1)-616(N), grouped computing resources614, and/or distributed file system638of framework layer620. One or more types of applications may include, but are not limited to, any number of a genomics application, a cognitive compute, and a machine learning application, including training or inferencing software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and/or other machine learning applications used in conjunction with one or more embodiments.

In at least one embodiment, any of configuration manager634, resource manager636, and resource orchestrator612may implement any number and type of self-modifying actions based on any amount and type of data acquired in any technically feasible fashion. Self-modifying actions may relieve a data center operator of data center600from making possibly bad configuration decisions and possibly avoiding underutilized and/or poor performing portions of a data center.

Example Network Environments

Network environments suitable for use in implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and/or other device types. The client devices, servers, and/or other device types (e.g., each device) may be implemented on one or more instances of the computing device(s)500ofFIG.5—e.g., each device may include similar components, features, and/or functionality of the computing device(s)500. In addition, where backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may be included as part of a data center600, an example of which is described in more detail herein with respect toFIG.6.