THREE-DIMENSIONAL REASONING USING MULTI-STAGE INFERENCE FOR AUTONOMOUS SYSTEMS AND APPLICATIONS

In various examples, an autonomous system may use a multi-stage process to solve three-dimensional (3D) manipulation tasks from a minimal number of demonstrations and predict key-frame poses with higher precision. In a first stage of the process, for example, the disclosed systems and methods may predict an area of interest in an environment using a virtual environment. The area of interest may correspond to a predicted location of an object in the environment, such as an object that an autonomous machine is instructed to manipulate. In a second stage, the systems may magnify the area of interest and render images of the virtual environment using a 3D representation of the environment that magnifies the area of interest. The systems may then use the rendered images to make predictions related to key-frame poses associated with a future (e.g., next) state of the autonomous machine.

BACKGROUND

Robots and other autonomous systems often must perceive the environment in a three-dimensional (3D) manner to solve a variety of tasks, such as 3D object manipulation tasks. To do so, as opposed to explicitly reconstructing a 3D model of a scene, view-based methods of object manipulation may directly process input images from single or multiple cameras. When given adequate training, view-based methods may successfully complete various pick-and-place and object rearrangement tasks. To be useful in industrial, household, and other domains, view-based methods for autonomous system control should be capable of learning new tasks with few demonstrations, as well as solving them precisely. However, the success of view-based methods involving high-precision 3D reasoning has been limited. Thus, performing precise, 3D manipulation tasks from few demonstrations has proven to be challenging.

SUMMARY

Embodiments of the present disclosure relate to three-dimensional (3D) reasoning using multi-stage inference for autonomous systems and applications. Systems and methods are disclosed for, among other things, predicting key-frame poses with higher precision by using a multi-stage, view transformation process to solve 3D manipulation tasks. For example, during a first stage of the process the disclosed systems and methods may predict an area of interest in a three-dimensional (3D) representation of an environment. The area of interest may correspond to a predicted location of an object in the environment, such as an object that an autonomous machine is instructed to manipulate. In a second stage, the systems may magnify the area of interest and render virtual images representing the 3D representation of the environment within the area of interest. The systems may then apply the virtual images to one or more machine learning models to make predictions related to key-frame poses associated with a future (e.g., next) state of the autonomous machine.

In contrast to conventional systems, the systems of the present disclosure, in some examples, are able to achieve better task performance, precision, and speed with respect to predicting key-frame poses and solving 3D manipulation tasks. For instance, by using a multi-stage inference pipeline, the systems of the present disclosure are able to magnify a region of interest and predict key-frame poses for an autonomous machine with greater precision. Additionally, the systems of the present disclosure may use convex up-sampling techniques, which may save graphics processing unit (GPU) memory during training and improve processing speed. Furthermore, in contrast to the conventional systems that use global features to predict end-effector rotation, the systems of the present disclosure improve end-effector rotation predictions by using location-conditioned features instead of the conventional global features.

Additionally, by using magnified or zoomed-in 3D representations that have greater detail for a predicted region of interest, the systems of the present disclosure are able to make predictions using fewer virtual images than the conventional systems, while still achieving the various improvements described herein. By being able to use fewer virtual images, the systems of the present disclosure thereby reduce the number of images to be rendered, as well as a number of tokens to be processed by a multi-view transformer, which improves training and inference speed without any loss in performance.

DETAILED DESCRIPTION

Systems and methods are disclosed related to three-dimensional (3D) reasoning using multi-stage inference for autonomous systems and applications. Although the present disclosure may be described with respect to an example autonomous or semi-autonomous vehicle or machine800(alternatively referred to herein as “vehicle800,” “ego-vehicle800,” “ego-machine800,” or “machine800,” an example of which is described with respect toFIGS.8A-8D), this is not intended to be limiting. For example, the systems and methods described herein may be used by, without limitation, non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more adaptive driver assistance systems (ADAS)), piloted and un-piloted robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, flying vessels, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater craft, drones, and/or other vehicle types. In addition, although the present disclosure may be described with respect to a Robotic View Transformer (RVT) for precise 3D object manipulation in a virtual environment (e.g., an accurate, fast, and scalable multi-view transformer for direct 3D object manipulation), this is not intended to be limiting, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and/or any other technology spaces where object detection and/or map creation may be used.

For instance, a system(s) may generate one or more first virtual images of a 3D representation of an environment (e.g., a physical environment) using a virtual environment that includes the 3D representation and determine one or more first predictions corresponding to the environment using the one or more first virtual images and one or more machine learning models (MLMs). For example, rather than directly applying the 3D representation to the MLM(s), the virtual images corresponding to the 3D representation may be applied to the MLM(s). Thus, the inputs to the MLM(s) (e.g., a transformer-based neural network) can be made independent from and/or reduced relative to the resolution of the 3D representation of the environment-allowing for reduced computational resources for training and deploying the MLM.

In some examples, the system(s) may use the one or more first predictions to update the 3D representation of the environment. For instance, the system(s) may magnify or zoom-in on a location or space within the 3D representation corresponding to the one or more first predictions. The system(s) may generate one or more second virtual images of the updated 3D representation—or magnified portion thereof)—using the virtual environment or another virtual environment that includes the updated 3D representation. The system(s) may apply the one or more second virtual images to the MLM(s) to determine one or more second predictions corresponding to the environment, and then use the one or more second predictions for controlling an autonomous machine. For example, rather than using the first predictions to control the autonomous machine, which may be less accurate, the system(s) may refine or update the first predictions (as the second predictions) by zooming in on a more detailed representation of the environment in a region of interest and running inference on the region of interest.

In some examples, to generate the 3D representation of the environment, one or more images of the environment may be captured using one or more sensors, such as one or more cameras in the environment. For example, multiple images (e.g., two-dimensional (2D) images) may be captured with each image corresponding to a respective camera, or one or more of the images may be generated using multiple cameras. In at least one embodiment, at least one image of the one or more images include Red Green Blue Depth (RGBD) images. The one or more images may be used to determine and/or generate one or more portions of the 3D representation of the environment (e.g., a 3D point cloud, a voxel representation, etc.). For instance, pixels of the one or more images may be projected into 3D space using various projection techniques.

In some instances, the one or more images of the 3D representation may be generated using 3D rendering techniques. For example, one or more virtual sensors, such as virtual cameras, may be positioned in the virtual environment, and at least one image of the one or more images may be rendered using the one or more virtual sensors. In at least one embodiment, images of the 3D representation may be rendered from views or perspectives of the virtual sensors. The images may be rendered using any combination or projection techniques, such as perspective projection or orthographic projection. Thus, one or more of the images may be rendered using a projection (e.g., an orthographic projection) that is different than projections used by physical sensors to determine the 3D representation (e.g., perspective projections). In further respects, images of the 3D representation may have different (e.g., higher) resolutions than images (e.g., real-world images) used to determine the 3D representation, may be captured using a different number of sensors (e.g., virtual sensors), and/or may be captured using sensors (e.g., virtual sensors) that have different poses than the sensors (e.g., physical sensors) used to determine the 3D representation.

In various examples, the one or more images of the 3D representation may be generated with corresponding depth information (e.g., 3D coordinates associated with pixels). The depth information may be applied to the MLM(s) with the image(s) to generate the one or more predictions. Also in at least one embodiment, where multiple images and/or rendered views are applied to the MLM, correspondence information may be generated for the images or views. The correspondence information may indicate one or more correspondences between one or more 3D points in the virtual environment and one or more points in the images or views. The correspondence information may be applied to the MLM(s) with the image(s) to generate the one or more predictions.

As described herein, in some examples the MLM(s) may include a transformer neural network, such as a multi-view transformer model. Images may be applied to the MLM based at least on dividing the images into grids of patches, tokenizing the patches (e.g., using a Multilayer Perceptron (MLP)), and projecting the tokenized patches to generate inputs to the transformer neural network. In at least one embodiment, the transformer neural network may include one or more first layers to separately evaluate image patches for different images applied to the transformer neural network to generate self-attention information for the images. One or more second layers of the transformer neural network may use the self-attention information to jointly evaluate the images to generate joint attention information for the images. The one or more predictions determine using the MLM(s) may correspond to the joint attention information.

In some instances, the MLM may compute per-view and/or image outputs and/or predictions. For example, the MLM may be used to generate 2D space predictions for one or more images applied to the MLM. In at least one embodiment, the per-view and/or image outputs may be combined to generate one or more predictions corresponding to multiple views or images. For example, the 2D space predictions from different images or views may be back-projected into a 3D space to generate one or more 3D space predictions. One or more control operations may be performed for the machine using the 3D space prediction(s).

In various examples, the MLM(s) may be trained to generate predictions corresponding to a 3D object manipulation task. For example, the machine may include a robot and predictions generated using the MLM(s) may be used to perform one or more control operations for 3D manipulation of an object in the environment. In at least one embodiment, the MLM generates output data indicating one or more heatmaps for one or more images or views. A heatmap may indicate (e.g., represent) likelihoods or confidence scores for different points within a corresponding image or view being relevant for an action (e.g., 3D object manipulation) to be performed using the robot. The heatmaps may be used to predict an end-effector translation for the robot. For example, the system may identify the most likely location(s) for the robot's end-effector based on the heatmaps, and this location(s) may indicate where the robot should move or translate the robot's end-effector. To do so, the system may, for example, back-project the heatmaps to predict scores for a discretized set of 3D points that densely cover the robot's workspace and use the 3D points to determine the end-effector translation.

The end-effector translation is one example of control information that may be predicted for the robot. In at least one embodiment, the MLM (e.g., a transformer neural network) may additionally or alternatively be used to generate one or more predictions indicating other forms of control information, such as a gripper position, a gripper rotation, and/or a collision or contact state with respect to the object and/or environment. For example, another MLM of the MLMs (e.g., an MLP) may use local image features and/or output corresponding to the transformer neural network to generate output corresponding to at least some of the control information.

In at least one embodiment, in addition to one or more images or views of the 3D representation, textual data (e.g., representing natural language text) may be applied to the MLM (e.g., a transformer neural network). For example, the textual data may be tokenized and applied to the transformer neural network. The textual data may correspond to a structured language command. The MLM may use the structured language command, at least in part, to determine at least some of the control information. For example, the textual data may indicate a desired object configuration for a 3D object manipulation task.

With reference toFIG.1A,FIG.1Ais a data flow diagram illustrating an example process100for using multi-stage inference for 3D reasoning, in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted altogether. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities may be carried out by hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory. In some embodiments, the systems, methods, and processes described herein may be executed using similar components, features, and/or functionality to those of example autonomous vehicle800ofFIGS.8A-8D, example computing device900ofFIG.9, and/or example data center1000ofFIG.10.

The process100may be implemented using, amongst additional or alternative components, a virtual environment determiner(s)102, one or more image generators104A and104B, one or more machine learning models106A and106B, (e.g., “MLM(s)106”), a magnifier(s)108, and a control component(s)110.

As an overview, the virtual environment determiner(s)102may be configured to receive data obtained using one or more sensors corresponding to one or more views (e.g., perspective views) of an environment, e.g., sensor data120. The virtual environment determiner(s)102may obtain the sensor data120and use the sensor data120to determine a 3D representation122in a virtual environment150(FIG.1C). The image generator(s)104A may generate image data124(e.g., representing image(s)160A and160B through160N of the virtual environment150) from the 3D representation122, and the image data124may be applied to the MLM(s)106A. The MLM(s)106A may use input data126and/or the image data124corresponding to the 3D representation122to determine output data128indicating one or more first predictions corresponding to the environment (e.g., the physical environment). The magnifier(s)108may obtain and use the output data128to determine a magnified 3D representation130in a virtual environment152(FIG.1F). The image generator(s)104B-which may be the same as or different from the image generator(s)104A—may generate image data132(e.g., representing image(s)162A and162B through162N of the virtual environment152) from the magnified 3D representation130, and the image data132may be applied to the MLM(s)106B-which may be the same as or different from the MLM(s)106A. The MLM(s)106B may use the input data126and/or the image data132corresponding to the magnified 3D representation130to determine refined output data134indicating one or more second predictions corresponding to the environment (e.g., the physical environment). The control component(s)110may use the refined output data134to perform one or more control operations for the machine800, e.g., in the physical environment.

In some embodiments, rather than directly applying the 3D representation122and/or the magnified 3D representation130to the MLM(s)106, the corresponding image(s)160and/or162generated using the 3D representation122and/or the magnified 3D representation130may be applied to the MLM(s)106. Thus, the inputs to the MLM(s)106can be made independent from and/or reduced relative to the resolution of the 3D representation122and/or the magnified 3D representation130of the virtual environment150—allowing for reduced computational resources for training and deploying the MLM(s)106.

The sensor data120may be generated using one or more sensors, such as any combination of the various sensors described herein. In one or more embodiments, the sensors may include at least one of one or more physical sensors in a physical environment or one or more virtual sensors in a simulated or virtual environment. For example, the one or more sensors may correspond to a physical or simulated version of the machine800, as described herein, or another robot, ego-machine, and/or vehicle.FIGS.1B-1H and2-6are primarily described using examples where the machine800corresponds to a robotic arm, whereasFIGS.8A-8Drelate to an example where the machine800corresponds to a vehicle.

The sensor data120may include, without limitation, sensor data from any of the sensors of and/or surrounding the machine800(and/or other vehicles or objects, such as robotic devices, VR systems, AR systems, etc., in some examples). For example, and with reference toFIGS.8A-8D, the sensor data120may include data generated by or using, without limitation, global navigation satellite systems (GNSS) sensor(s)858(e.g., Global Positioning System sensor(s), differential GPS (DGPS), etc.), RADAR sensor(s)860, ultrasonic sensor(s)862, LIDAR sensor(s)864, inertial measurement unit (IMU) sensor(s)866(e.g., accelerometer(s), gyroscope(s), magnetic compass(es), magnetometer(s), etc.), microphone(s)896, stereo camera(s)868, wide-view camera(s)890(e.g., fisheye cameras), infrared camera(s)892, surround camera(s)894(e.g., 360 degree cameras), long-range and/or mid-range camera(s)898, speed sensor(s)844(e.g., for measuring the speed of the machine800and/or distance traveled), and/or other sensor types.

In some examples, the sensor data120may include sensor data generated using one or more forward-facing sensors, side-view sensors, downward-facing sensors, upward-facing sensors, and/or rear-view sensors. This sensor data may be useful for identifying, detecting, classifying, and/or tracking movement of objects around the machine800within the environment. In embodiments, any number of sensors may be used to incorporate multiple fields of view (e.g., the fields of view of the long-range cameras898, the forward-facing stereo camera868, and/or the forward facing wide-view camera890ofFIG.9B) and/or sensory fields (e.g., of a LIDAR sensor864, a RADAR sensor860, etc.). As used herein, the sensor data120or portions of sensor data may reference unprocessed sensor data, pre-processed sensor data, or a combination thereof.

The sensor data120may include image data representing an image136A or136B through136N (also referred to as “images136”) ofFIG.1B, image data representing a video (e.g., snapshots of video), data representing sensory fields of sensors (e.g., depth maps for LIDAR sensors, a value graph for ultrasonic sensors, etc.), and/or data representing measurements of sensors. Where the sensor data120includes image data, any type of image data format may be used, such as, for example, and without limitation, compressed images such as in Joint Photographic Experts Group (JPEG) or Luminance/Chrominance (YUV) formats, compressed images as frames stemming from a compressed video format such as H.264/Advanced Video Coding (AVC) or H.265/High-Efficiency Video Coding (HEVC), raw images such as originating from Red Clear Blue (RCCB), Red Clear (RCCC), or another type of imaging sensor, and/or other formats. In addition, in some examples, the sensor data120may be used without any pre-processing (e.g., in a raw or captured format), while in other examples, at least some of the sensor data120may undergo pre-processing (e.g., noise balancing, demosaicing, scaling, cropping, augmentation, white balancing, tone curve adjustment, etc., such as using a sensor data pre-processor (not shown)).

In the example ofFIG.1A, the virtual environment determiner(s)102may generate or determine one or more portions of the virtual environment150, including the 3D representation122of the environment. The sensor data120may capture one or more views of the environment (e.g., a physical environment). For example, the virtual environment determiner(s)102may generate the 3D representation122based on the sensor data120representing one or more images136of a physical environment corresponding to various views of sensors and/or cameras mounted on and/or about the machine800, examples of which are described herein.

The sensor(s) and/or camera(s) may be used to capture multiple image(s)136of the environment. For example, a first image (e.g., image136A inFIG.1B) may correspond to a respective position, perspective, and/or view of a sensor and/or camera in the environment. An image(s)136may be generated using any number of sensors or cameras oriented on or about the machine800in the environment. In at least one embodiment, at least one of the images136may include depth information, such as Red Green Blue Depth (RGB-D) information. The virtual environment determiner(s)102may use one or more of the images136to determine and/or generate one or more portions of the 3D representation122of the environment. For example, the 3D representation122may include, amongst other possibilities, a 3D point cloud or a voxel representation that is based at least on the depth information.

Referring now toFIG.1B,FIG.1Billustrates examples of the images136which may be used to determine the 3D representation122of an environment, in accordance with some embodiments of the present disclosure. For example,FIG.1Bshows three images136(e.g., image136A, image136B, and/or image136N) corresponding to different perspective views of the environment. The image(s)136can comprise RGB-D image(s), including one or more color channels140and one or more depth channels142. More or fewer image(s)136may be used to determine the 3D representation122, as indicated inFIG.1B. The image(s)136can provide different views and or perspectives of the environment. Different frames, or sets, of the images136may capture different regions of the environment and may correspond to or overlap with one or more portions of another frame or image or one or more of the images136may be non-overlapping.

In at least one embodiment, the images136and/or views may be configured to collectively capture one or more portions of a 360-degree field of view of the environment, and the views may or may not be at least partially overlapping. For example, the image(s)136shown inFIG.1B(including the image136A, image136B, and/or image136N) correspond to various perspectives of the environment and may be time-synchronized for use by the virtual environment determiner(s)102to generate the 3D representation122(e.g., over any number of timestamps and/or iterations of image captures).

The image(s)136of the physical environment can be captured using one or more sensors, for example, by orienting the camera(s) relative to the machine800in the environment. For example, multiple image(s)136may be captured with each image136corresponding to a respective camera, or one or more of the images136may be generated using multiple cameras. For example, multiple cameras or a single camera may be used to capture the image136A. Multiple cameras may be oriented throughout the environment to obtain different portions of the sensor data120. The cameras may be static/fixed or may move to capture different views. In at least one embodiment, at least one image (e.g., the image136A) of the one or more images136may include RGB-D information. As described herein, the virtual environment determiner(s)102may use the image(s)136to determine and/or generate one or more portions of the 3D representation122of the physical (or virtual in some examples) environment.

The virtual environment determiner(s)102may generate and/or determine the 3D representation122through various approaches. As an example, the virtual environment determiner(s)102may generate and/or determine one or more portions of the 3D representation122based at least in on the matching existing 3D models (e.g., from a library). In this approach, computer vision algorithms analyze the images136, identifying distinctive features that are then compared with a database of pre-existing 3D models. Additionally, or alternatively, the virtual environment determiner(s)102may generate one or more point clouds, for example, derived from the depth information in the images136and/or technologies such as LiDAR or stereo vision. Additionally, or alternatively, a photogrammetry-based approach can be used, where the virtual environment determiner(s)102analyzes multiple images of the same scene taken from different viewpoints. By extracting 3D information from the parallax and perspective shifts in these images, the virtual environment determiner(s)102may reconstruct one or more portions of the environment in three dimensions.

Referring now toFIG.1C,FIG.1Cillustrates an example of the 3D representation122of a virtual environment, in accordance with some embodiments of the present disclosure. One or more virtual cameras144(or more generally one or more virtual sensors) are shown as being oriented about the virtual environment150. The image generator104A may use the virtual camera(s)144to generate one or more images of the 3D representation122in the virtual environment150. Examples of the one or more images include an image160A, an image160B, and/or an image160N (also referred to as “images160”) illustrated inFIG.1D.

Various virtual cameras144may be virtually positioned/located throughout the virtual environment150to generate the images160to capture various perspectives of the virtual environment150. For example, the virtual cameras144can be used to obtain the images160from perspectives that are different than the perspectives of the cameras used to generate the 3D representation122.FIG.1Cshows positions and orientations of the virtual cameras144, a 3D representation127of the machine800, and a 3D representation125of an article(s) or object(s) in a coordinate space146(e.g., each of which may be generated and/or determined using the virtual environment determiner(s)102). In at least one embodiment, the control component(s)110performs control operations for the machine800to move or manipulate the article(s) with respect to the coordinate space146.

In at least one embodiment, the coordinate space146may be a cartesian coordinate space (e.g., including an X, Y, and Z axis). As various examples, the coordinate space(s)146may use one or more of cartesian coordinates, polar coordinates, spherical coordinates, and/or cylindrical coordinates to represent positions and/or orientations of 3D data with respect to the virtual environment150. Further examples include parabolic coordinates, bipolar coordinates, elliptical coordinates, toroidal coordinates, and/or generalized coordinates. In at least one embodiment, the coordinate space146may be used to orient and/or reference a location of the machine800with respect to the virtual environment150and/or the article(s) being manipulated. For example, the control component(s)110may use the coordinate space146to determine or track the location and/or orientation of various objects in the physical environment including the machine's location and/or position relative to those objects.

As described herein, the image generator104A may be configured to generate (or render) the image(s)160of the 3D representation122of the virtual environment150. The image generator104A may use one or more virtual sensors to render or generate the image(s)160, such as the virtual camera(s)144. In particular, the virtual camera(s)144in the virtual environment150may be used by the image generator104A to generate the image data124, corresponding to the image(s)160, which may be input to the MLM(s)106. In at least one embodiment, the one or more image(s)160of the 3D representation122are generated using 3D rendering techniques, such as light transport simulation (e.g., path tracing, ray tracing, etc.), rasterization, and/or other graphical rendering techniques. For example, one or more virtual sensors and/or the virtual camera(s)144may be positioned in the virtual environment150, and at least one image (e.g., the image160A) of the images160may be rendered using the one or more virtual cameras144(e.g., virtual sensors). The images160of the 3D representation122may be rendered from views or perspectives of the virtual cameras144.

The image generator104A may generate an image160of the virtual environment150that has a higher resolution than one or more of the images136used to determine the 3D representation122of the environment (e.g., a real or physical environment). For example, the images160(e.g., of the virtual environment150) may have a higher resolution than each of the images136of a physical environment. The image generator104A can modify the perspective, location, orientation, and/or one or more intrinsic or extrinsic properties of the virtual cameras144to generate the image data124. In at least one embodiment, one or more properties of the virtual cameras144may remain fixed across iterations of the process100to iteratively perform control operations for the machine800.

In some embodiments, the one or more images160of the 3D representation122are captured with corresponding depth information (e.g., 3D coordinates associated with pixels or pixel locations). For example, the image generator104may generate corresponding image(s)160with the corresponding depth information (e.g., stored in a depth channel). The depth information may be applied to the MLM(s)106with the image(s)160(e.g., color information thereof) to generate one or more predictions. In some embodiments, where multiple images160include rendered views that are applied to the MLM(s)106, correspondence information may be generated for the image(s)160or views generated using the image generator104and input into the MLM(s)106. The correspondence information may indicate one or more correspondences between one or more 3D points in the 3D representation122of the virtual environment150and points or pixels across the images160or views. In at least one embodiment, correspondence information may be provided for each pixel in each image160and may encode the coordinates (e.g., x, y, z) of one or more corresponding points in the virtual environment. The generated correspondence information may be applied to the MLM(s)106with the image(s)160and facilitate the determination of the generated predictions.

In some examples, the image generator(s)104(e.g., the image generators104A and104B), may render images using a projection-based point-cloud rendering technique. The image generator(s)104may perform various steps to render a point-cloud with N points to an RGB image and depth image of size (h, w). For instance, during a projection step(s), for each 3D point of index n∈{0, 1, . . . . N} and corresponding RGB value fn, the image generator(s)104may compute the depth dnand image pixel coordinate (xn, yn) using camera intrinsics and extrinsics. From the 2D pixel coordinate (xn, yn), the image generator(s)104may compute the linear pixel index in=(xn)(w)+yn. The projection operation may be accelerated using GPU matrix multiplications, in some examples. During a Z-ordering step(s), for each pixel of a linear-index j in the image, the image generator(s)104may find the point index with smallest depth dnamong the set of points that project to the pixel {n|in=j}. The image generator(s)104may assign that point's RGB value fnto pixel j of the RGB image and depth dnto pixel j of the depth image. To accelerate Z-ordering, the image generator(s)104may pack each point's depth and index into a single 64-bit integer, such that the most significant 32 bits encode depth, while the least significant bits encode the point index. Then, Z-ordering can be implemented with two kernels (e.g., CUDA kernels). First, a parallel loop over point cloud points may try to store each packed depth-index into a depth-independent image at the pixel j using the atomicMin operation. In some examples, the depth-index stored by the minimum-depth point at each pixel may survive. The second kernel, in a loop over pixels, may create depth and feature images by unpacking the depth-index and looking up the point feature. For instance, color point-clouds may be rendered by packing the 32-bit color, and the disclosed system(s) may extend this to images with arbitrary number of channels by packing the point index instead. During a screen-space splatting step(s), the image generator(s)104may model each point by some geometry of a finite size. The image generator(s)104may model each point as a disc of radius r facing the camera. This splatting may be computed in screen space after projection and z-ordering, thereby reducing the computation required in the projection and z-ordering. For each pixel j in the image, the image generator(s)104may search nearby for another pixel k of lowest depth. If the pixel k has depth dk<djand is closer than r·focal_length/dk, the feature may be replaced and depth of pixel j with that of pixel k.

In at least one embodiment, the virtual environment determiner(s)102, the image generator(s)104(e.g., the image generators104A and104B), and/or the magnifier108may be implemented using one or more Neural Radiance Fields (NeRFs). For example, the NeRF(s) may receive the image data representative of one or more of the images136and/or other sensor data to generate one or more portions of the image data124and/or132(e.g., one or more images160and/or162). In at least one embodiment, the NeRF(s) may receive one or more input parameters to control one or more of the views and/or aspects thereof captured by the image data124and/or132. In some examples, the image generator (s0104may render the images (e.g., virtual images) such that one or more sizes associated with the image(s) are rationally divisible by one or more patch sizes associated with the MLM(s)106.

The image generator104A outputs image data124corresponding to the images160that are applied to the one or more MLMs106(e.g., the MLMs106A and/or106B) trained to generate one or more predictions corresponding to the environment. The MLM(s)106can include one or more models for learning complex non-linear functions by adapting one or more internal parameters. The MLM(s)106and/or other MLMs described herein may be include any suitable MLM. For example and without limitation, any of the various MLMs described herein may include one or more of any type(s) of machine learning model(s), such as a machine learning model using linear regression, logistic regression, decision trees, support vector machines (SVM), Naïve Bayes, k-nearest neighbor (Knn), K means clustering, control barrier functions, random forest, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., one or more auto-encoders, convolutional, recurrent, transformer, perceptrons, Long/Short Term Memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid state machine, large language, etc. neural networks), and/or other types of machine learning model.

In at least one embodiment, the MLM(s)106can include a transformer model, such as a multi-view transformer model, such as inFIG.2. With reference toFIG.2, as described herein, one or more images may be generated for one or more perspectives in the virtual environment150. For example, the image160A and the images160B through160N may capture various perspectives and/or orientations within the virtual environment150, e.g., as shown inFIG.1D. One or more images210A, and210B through210N (which may correspond to the images160A-160N and/or the images162A-162N) may be applied to the MLM(s)106, as shown inFIG.2, to generate corresponding output data220(which may correspond to the output data128and/or the refined output data134) and to obtain corresponding predictions. For example, the output data220A may correspond to the image210A, the output data220B may correspond to the image210B, and the output data220N may correspond to the image210N.

In some embodiments, an image processor202can at least partially divide the images210into grids260of patches262in order to apply the images210to the MLM(s)106. For example, the image processor202can generate tokenized patches262(e.g., using a Multilayer Perceptron (MLP)). The image projector204can then project the tokenized patches262to generate inputs to the transformer neural network corresponding to the MLM(s)106.

For example, the image processor202can split each image210into smaller non-overlapping patches262that may be flattened and/or projected using the image projector204. The image projector204can project the tokenized patches262into a lower dimension by using a linear projection or a multilayer perceptron to generate a token264representing each patch262and capturing the visual and/or depth content in the image(s)210. In some embodiments, the image projector204can project the images210into a higher (or lower) resolution by using the multilayer perceptron or linear projection. In this way, the image(s)210applied to the MLM(s)106can have a higher or lower resolution than the related images136captured by a real sensor or camera in the physical environment.

In at least one embodiment, the MLM(s)106, e.g., the transformer neural network, may include one or more layers230of tokens264to separately evaluate the tokens correspond to the image patches262for different images210to generate self-attention information for the images210. One or more layers232of the transformer neural network may use the self-attention information to jointly evaluate the images to generate joint attention information for the images. The one or more predictions determined using the MLM(s)106may correspond to the joint attention information.

In at least one example, the MLM(s)106may remove the feature upsampling and directly predict heatmaps of shape hxw from features at the token resolution. Specifically, the MLM(s)106may use one or more convex upsampling layers to make predictions. For instance, the convex upsampling layer(s) may use a learned convex combination of features to make predictions in a higher resolution.

In at least one embodiment, the MLM(s)106compute per-view and/or image outputs and/or predictions, e.g., shown as output data220. Referring now toFIG.1EwithFIG.2,FIG.1Eshows predictions including examples of view-specific predictions, in accordance with some embodiments of the present disclosure. For example, the MLM(s)106may be used to generate 2D (or other dimensional) space predictions172(e.g., heatmaps), shown inFIG.1E, for the one or more images160applied to the MLM106. In at least one embodiment, the per-view and/or image outputs may be combined to generate one or more predictions180corresponding to multiple views or images210. For example, the 2D space predictions for different images160(e.g., images160A and160B) or views may be back-projected into 3D space to generate one or more 3D space predictions180. The control component(s)110may perform one or more control operations for the machine800using the 3D space prediction(s)180and/or other predictions.

In at least one embodiment, the MLM(s)106may be trained to generate predictions180corresponding to a 3D object manipulation task. For example, the machine800may include a robot and predictions generated using the MLM(s)106may be used to perform one or more control operations for 3D manipulation of an object in the environment. In at least one embodiment, the MLM106generates the output data220indicating one or more heat maps for one or more images or views. For example, each of the 2D space predictions shown inFIG.1Emay correspond to a respective heatmap. A heatmap may indicate (e.g., represent) likelihoods or confidence scores for different points within a corresponding image or view being relevant for an action (e.g., 3D object manipulation) to be performed using the robot. The heatmaps may be used to predict an end-effector translation for the robot. For example, the system may identify the 3D space prediction(s)180as the most likely location(s) for the robot's end-effector based on the heatmaps, and this location(s) may indicate where the robot should move or translate the robot's end-effector to in the environment. To do so, the system may, for example, back-project the heatmaps to predict scores for a discretized set of 3D points that densely cover the robot's workspace and use the 3D points to determine the end-effector translation, as indicated inFIG.1E.

The end-effector translation is one example of control information that may be predicted for the robot or machine800. In at least one embodiment, the MLM106(e.g., a transformer neural network) may additionally or alternatively be used to generate one or more predictions180indicating other forms of control information, such as a gripper position, a gripper rotation, and/or a collision or contact state with respect to the object and/or environment. For example, as indicated inFIG.2, another MLM106of the MLMs106(e.g., an MLP206) may usc output corresponding to the transformer neural network (e.g., one or more layers234) to generate output data220C corresponding to at least some of the control information indicated inFIG.1E.

In at least one embodiment, in addition to one or more images210, other input data126may be applied to the MLM(s)106. For example, the input data126shown inFIGS.1A and2may include textual data (e.g., representing natural language text) applied to the MLM(s)106(e.g., a transformer neural network). For example, the textual data may be tokenized and applied to the transformer neural network. In at least one embodiment, the textual data may correspond to a structured language command. The MLM106may use the structured language command, at least in part, to determine at least some of the control information. For example, the textual data may indicate a desired object configuration for a 3D object manipulation task. Examples of the textual data may include one or more commands to put a marker in a bowl to instruct the machine800to pick up a marker and place the marker inside the bowl. Another example may include one or more commands to stack blocks, which may instruct the machine800to stack two or more blocks on top of one another. A further example may include one or more commands to turn the tap, which may instruct the machine800to turn on or off a water tap.

Referring back to the example ofFIG.1, the magnifier108may obtain the output data128and determine the magnified 3D representation130of the virtual environment. That is, the magnifier108may use the initial predictions indicated in the output data128to identify a region of interest in the virtual environment, such as a region surrounding a predicted location to position the end-effector of the robot. The magnifier108may then zoom in on the region of interest. In some examples, to generate the magnified 3D representation130, the magnifier108may perform one or more operations similar to those performed by the virtual environment determiner(s)102to generate the 3D representation122. As one example, the magnifier108may crop one or more of the images160to capture the region of interest in each of the images160, and then use the cropped images to generate the magnified 3D representation130. In at least one embodiment, at least one of the images136may include depth information, such as Red Green Blue Depth (RGB-D) information. The magnifier108may crop and use one or more of the images136to determine and/or generate one or more portions of the magnified 3D representation130of the region of interest in the environment. For example, the magnified 3D representation130may include, amongst other possibilities, a 3D point cloud or a voxel representation that is based at least on the depth information.

For instance,FIG.1Fillustrates an example of enlarging a portion of the 3D representation122ofFIG.1Cbased at least on the 2D (or other dimensional) space predictions172(e.g., heatmaps) and/or view-specific predictions180ofFIG.1E, in accordance with some embodiments of the present disclosure. The virtual cameras144(or more generally one or more virtual sensors) are shown as being oriented about the virtual environment150. The image generator104B may use the virtual camera(s)144to generate one or more images of the magnified 3D representation130in the virtual environment150. Examples of the one or more images include an image162A, an image162B, and/or an image162N (also referred to as “images162”) illustrated inFIG.1G. As shown, the images162may be enlarged and/or have a higher zoom factor than the images160of the initial 3D representation122.

As above, the virtual cameras144may be virtually positioned/located throughout the virtual environment150to generate the images162to capture various perspectives of the virtual environment150including the magnified 3D representation130. For example, the virtual cameras144can be used to obtain the images162from perspectives that are different than the perspectives of the cameras used to generate the images136.FIG.1Fshows positions and orientations of the virtual cameras144, and a magnified 3D representation127of an article(s) or object(s) in a coordinate space148(e.g., each of which may be generated and/or determined using the magnifier108).

In some instances, the coordinate space148may be the same as or different from the coordinate space146ofFIG.1C, and used for the same and/or different purposes. For example, the coordinate space148may be used to orient and/or reference a location of the machine800with respect to the virtual environment150and/or the article(s) being manipulated. Additionally, in some instances, the control component(s)110may use the coordinate space148to determine or track the location and/or orientation of various objects in the physical environment including the machine's location and/or position relative to those objects.

As described herein, the image generator104B (which may be the same as or different from the image generator104A) may be configured to generate (or render) the image(s)162of the magnified 3D representation130of the virtual environment150. The image generator104B may use one or more virtual sensors to render or generate the image(s)162, such as the virtual camera(s)144. In particular, the virtual camera(s)144in the virtual environment150may be used by the image generator104B to generate the image data132, corresponding to the image(s)162, which may be input to the MLM(s)106B. In at least one embodiment, the one or more image(s)162of the magnified 3D representation130may be generated using 3D rendering techniques, such as light transport simulation (e.g., path tracing, ray tracing, etc.), rasterization, and/or other graphical rendering techniques. For example, one or more virtual sensors and/or the virtual camera(s)144may be positioned in the virtual environment150, and at least one image (e.g., the image162A) of the images162may be rendered using the one or more virtual cameras144(e.g., virtual sensors). The images162of the magnified 3D representation130may be rendered from views or perspectives of the virtual cameras144.

The image generator104B may generate the images162of the virtual environment150that have similar resolution to the images160of the 3D representation122, but with greater detail associated with the region of interest from being magnified or otherwise zoomed in. The image generator104B can modify the perspective, location, orientation, and/or one or more intrinsic or extrinsic properties of the virtual cameras144to generate the image data132. In at least one embodiment, one or more properties of the virtual cameras144may remain fixed across iterations of the process100to iteratively perform control operations for the machine800.

In some embodiments, the one or more images162of the magnified 3D representation130are captured with corresponding depth information (e.g., 3D coordinates associated with pixels or pixel locations). For example, the image generator104B may generate corresponding image(s)162with the corresponding depth information (e.g., stored in a depth channel). The depth information may be applied to the MLM(s)106with the image data132representative of the image(s)162(e.g., color information thereof) to generate one or more predictions. In some embodiments, where multiple images162include rendered views that are applied to the MLM(s)106, correspondence information may be generated for the image(s)162or views generated using the image generator104B and input into the MLM(s)106. The correspondence information may indicate one or more correspondences between one or more 3D points in the magnified 3D representation130of the virtual environment150and points or pixels across the images162or views. In at least one embodiment, correspondence information may be provided for each pixel in each image162and may encode the coordinates (e.g., x, y, z) of one or more corresponding points in the virtual environment. The generated correspondence information may be applied to the MLM(s)106with the image(s)162and facilitate the determination of the generated predictions.

Referring now toFIG.1H,FIG.1Hshows examples of refined predictions based at least on applying the virtual images162ofFIG.1Gto the MLM(s)106B, in accordance with some embodiments of the present disclosure. For example, the MLM(s)106may be used to generate updated 2D (or other dimensional) space predictions174(e.g., heatmaps), shown inFIG.1H, for the one or more images162applied to the MLM(s)106. In at least one embodiment, the per-view and/or image outputs may be combined to generate one or more refined 3D space predictions182corresponding to multiple views or images210. For example, the 2D space predictions for different images162(e.g., images162A and162B) or views may be back-projected into 3D space to generate the one or more refined 3D space predictions182. The control component(s)110may perform one or more control operations for the machine800using the refined 3D space prediction(s)182and/or other predictions.

In at least one embodiment, the MLM(s)106may be trained to generate the refined 3D space predictions182corresponding to the 3D object manipulation task. For example, the machine800may include a robot and predictions generated using the MLM(s)106may be used to perform one or more control operations for 3D manipulation of an object in the environment. In at least one embodiment, the MLM(s)106generates the output data220indicating one or more heat maps for one or more images or views. For example, each of the updated 2D space predictions174shown inFIG.1Hmay correspond to a respective heatmap. A heatmap may indicate (e.g., represent) likelihoods or confidence scores for different points within a corresponding image or view being relevant for an action (e.g., 3D object manipulation) to be performed using the robot. The heatmaps may be used to predict an end-effector translation for the robot. For example, the system may identify the refined 3D space prediction(s)182as the most likely location(s) to position the robot's end-effector based on the heatmaps, and this location(s) may indicate where the robot should move or translate the robot's end-effector to in the environment. To do so, the system may, for example, back-project the heatmaps to predict scores for a discretized set of 3D points that densely cover the robot's workspace and use the 3D points to determine the end-effector translation, as indicated inFIG.1H.

Referring now toFIG.3,FIG.3illustrates examples of projection techniques which may be used to generate images of a virtual environment, in accordance with some embodiments of the present disclosure. In at least one embodiment, the image generator(s)104may use one or more of the virtual cameras144to generate one or more of the images160and/or162using a perspective projection of the virtual environment150. Additionally, or alternatively, the image generator(s)104may use one or more of the virtual cameras144to generate one or more of the images160and/or162using an orthographic projection of the virtual environment150. Generally, the images160and/or162may be rendered using any combination of projection techniques for the different views and/or images. As indicated inFIG.3, a projection of three-dimensional objects onto a two-dimensional plane can may performed using a perspective projection300and/or an orthographic projection310. The perspective projection300may be used to approximate how objects appear based on their distance from the sensor or camera. In contrast, in the orthographic projection310, objects in the three-dimensional environment may retain their size regardless of their distance or depth.

The perspective projection300may be used to mimic a real-world camera perspective, where 3D objects become smaller with distance from the sensor. The perspective projection300may correspond to foreshortening or convergence lines302onto a perspective projection plane304. Orthographic projection310keeps the sizes constant on an orthographic plane314, e.g., without perspective distortion of convergence lines302. The orthographic projection310may be performed without foreshortening and using, for example, parallel projection lines312to project to an orthographic plane314. The orthographic projection310may also be referred to as engineering perspective or projection.

For example, and without limitation, the image generator(s)104may generate an image(s)160and/or162using a projection (e.g., the orthographic projection310) that is different from a projection(s) used to generate one or more of the images160and/or162. For example, physical sensors and/or cameras may not be capable of performing the orthographic projection310. However, the MLM(s)106may provide higher quality output using one or more orthographically projected images.

Referring now toFIG.4,FIG.4illustrates various examples of camera parameters, which may be used to generate images of a virtual environment, in accordance with some embodiments of the present disclosure. For example,FIG.4shows camera locations400,410, or and420, which may be used to render one or more of the images160and/or162. The camera locations400,410, or420are provided as examples in addition to the camera locations ofFIGS.1C and1Fwhere three cameras are used to produce three images160and/or162for input to the MLM(s)106—one on each end of an axis (e.g., x axis, y axis, z axis). The camera locations400are an example where three cameras are provided, however more or fewer cameras may be used. For example, 5 cameras may be used to generate an image for each side of a cubic area. The camera locations410are shown as corresponding to a rotated cube (e.g., rotated by 15 degrees or some other amount). The camera locations420are shown as corresponding to the locations of real cameras, which may have been used to obtain the sensor data120(e.g., corresponding to perspectives of the images136). In at least one embodiment, by varying the number, locations, and/or other parameters (e.g., pose) of the virtual cameras144with respect to the real cameras used to obtain the images136, the performance of the MLM(s)106can be improved. In at least one embodiment, the camera locations and/or parameters may be fixed for each iteration of the process100. In at least one embodiment, one or more of the camera locations and/or parameters may be varied across one of more iterations of the process100. In at least one embodiment, the camera locations and/or parameters may match or otherwise correspond to the camera locations and/or parameters used to train the MLM(s)106.

As indicated byFIGS.3and4, the image(s)160and/or162generated by the image generator(s)104may have different (e.g., higher) resolutions and/or perspectives than the image(s)136captured using the cameras in the environment. In some embodiments, one or more of the image(s)136can be captured using physical sensor(s) and/or camera(s) within a physical environment and may or may not have a higher resolution than one or more of the image(s)160and/or162generated using the image generator(s)104. Similarly, the images136may be generated using a different number of sensor(s) and/or camera(s) than the virtual camera(s)144or sensor(s) within the virtual environment150used to generate the image(s)160and/or162of the 3D representations. The image(s)160and/or162generated using the image generator(s)104may be captured using sensors (e.g., virtual camera(s)144) that have different orientations, perspectives, intrinsic or extrinsic properties, and/or views than the sensors (e.g., physical sensors) used to generate the sensor data120.

Referring now toFIG.5,FIG.5is a block diagram of an example system suitable for use in implementing multi-stage inference for 3D reasoning, in accordance with some embodiments of the present disclosure. As shown, the system502(which may represent, and/or include, the example computing device(s)900and/or the example data center1000) may include one or more processors504(which may be similar to, and/or include, the CPUs906and/or the GPUs908) and memory506(which may be similar to, and/or include, the memory904). For instance, the memory506may store the virtual environment determiner(s)102, the image generator(s)104, the machine learning model(s)106, the magnifier(s)108, the control component(s)110, the image processor202, and/or the image projector204. Additionally, the processor(s)504may execute the virtual environment determiner(s)102, the image generator(s)104, the machine learning model(s)106, the magnifier(s)108, the control component(s)110, the image processor202, and/or the image projector204to perform one or more of the processes described herein. In some examples, one or more of the various components and/or modules stored in the memory506and executed using the processor(s)504may be stored and/or executed using other systems than the system502.

Additionally, as shown by the example ofFIG.5, the system502may receive the input data126, which may correspond to textual data, voice data, or other data. For instance, the input data126may include textual data (e.g., representing natural language text) and/or correspond to a structured language command. For example, the textual data may indicate a desired object configuration for a 3D object manipulation task. Examples of the textual data may include one or more commands to place a peg in a hole to instruct the machine800to pick up a peg and place the peg inside the hole. Another example may include one or more commands to stack blocks, which may instruct the machine800to stack two or more blocks on top of one another. A further example may include one or more commands to turn the tap, which may instruct the machine800to turn on or off a water tap.

FIG.6is a flow diagram illustrating an example method600for predicting locations in an environment using multi-stage inference, in accordance with some embodiments of the present disclosure. The method600, at block B602, may include rendering one or more virtual images depicting a magnified portion of a 3D representation of an environment from one or more perspectives. For instance, the image generator(s)104may render the image data132representative of the virtual images depicting the magnified 3D representation130of the virtual environment. In some examples, the magnified portion of the 3D representation may correspond to one or more first predicted locations in the environment.

The method600, at block B604, may include obtaining, based at least on applying the one or more virtual images to one or more machine learning models, one or more second predicted locations in the environment. For instance, the MLM(s)106may determine the second predicted location(s) in the environment based at least on processing the virtual image(s) represented by the image data132. The second predicted locations may be indicated in one or more heatmaps determined using the machine learning model(s).

The method600, at block B606, may include performing one or more control operations associated with a machine in the environment based at least on the one or more second predicted locations. For instance, the control component(s)110may cause performance of the control operation(s) associated with the machine in the environment based at least on the second predicted locations. In some examples, the control operation(s) may include causing an end-effector of an autonomous robotic system to move to a position corresponding to the second predicted location(s).

FIG.7is a flow diagram illustrating an example method700for using a multi-stage inference to perceive an environment for controlling a machine, in accordance with some embodiments of the present disclosure. The method700, at block B702, may include rendering, using a 3D representation of an environment, one or more first images of a virtual environment corresponding to the environment. For instance, the image generator104A may render the image data124representing the first image(s) of the virtual environment using the 3D representation122.

The method700, at block B704, may include determining, based at least on applying the first image(s) to one or more first machine learning models, one or more first predictions. For instance, the image data124representing the first image(s) may be applied to the MLM(s)106A, and the output data128may represent or include the first prediction(s). In some examples, the MLM(s)106A may determine the first prediction(s) based at least on the input data126which may represent an instruction associated with an autonomous machine, such as “place the blue block on the red square.”

The method700, at block B706, may include determining, based at least on the first prediction(s), an updated version of the 3D representation including a magnified portion of the 3D representation. For instance, the magnifier108may generate or determine the magnified 3D representation130based at least on the first prediction(s) in the output data128. The magnifier108may determine the magnified 3D representation130using the 3D representation122. In some examples, the magnifier108may zoom in on an area of interest in the 3D representation122based on the first prediction(s) to generate or determine the magnified 3D representation.

The method700, at block B708, may include apply, to one or more second machine learning models, one or more second images rendered using the magnified portion of the 3D representation. For instance, the image generator104B may generate the image data132representing the second image(s) using the magnified 3D representation130. The image data132may then be applied to the MLM(s)106B.

The method700, at block B710, may include perform one or more operations associated with a machine based at least on one or more second predictions obtained using the second machine learning model(s). For instance, the control component(s)110may cause the machine to perform the operation(s) based at least on the second prediction(s) included in the refined output data134. In various examples, the predictions of the refined output data134may be more precise or accurate than the predictions of the output data128. In some examples, the second prediction(s) may correspond to one or more refined versions of the first prediction(s) such that one or more first confidence scores associated with the first prediction(s) are less than one or more second confidences scores associated with the second prediction(s).

Example Autonomous Vehicle

The vehicle800may include components such as a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. The vehicle800may include a propulsion system850, such as an internal combustion engine, hybrid electric power plant, an all-electric engine, and/or another propulsion system type. The propulsion system850may be connected to a drive train of the vehicle800, which may include a transmission, to enable the propulsion of the vehicle800. The propulsion system850may be controlled in response to receiving signals from the throttle/accelerator852.

A steering system854, which may include a steering wheel, may be used to steer the vehicle800(e.g., along a desired path or route) when the propulsion system850is operating (e.g., when the vehicle is in motion). The steering system854may receive signals from a steering actuator856. The steering wheel may be optional for full automation (Level 5) functionality.

The brake sensor system846may be used to operate the vehicle brakes in response to receiving signals from the brake actuators848and/or brake sensors.

Controller(s)836, which may include one or more system on chips (SoCs)804(FIG.8C) and/or GPU(s), may provide signals (e.g., representative of commands) to one or more components and/or systems of the vehicle800. For example, the controller(s) may send signals to operate the vehicle brakes via one or more brake actuators848, to operate the steering system854via one or more steering actuators856, to operate the propulsion system850via one or more throttle/accelerators852. The controller(s)836may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals, and output operation commands (e.g., signals representing commands) to enable autonomous driving and/or to assist a human driver in driving the vehicle800. The controller(s)836may include a first controller836for autonomous driving functions, a second controller836for functional safety functions, a third controller836for artificial intelligence functionality (e.g., computer vision), a fourth controller836for infotainment functionality, a fifth controller836for redundancy in emergency conditions, and/or other controllers. In some examples, a single controller836may handle two or more of the above functionalities, two or more controllers836may handle a single functionality, and/or any combination thereof.

The controller(s)836may provide the signals for controlling one or more components and/or systems of the vehicle800in response to sensor data received from one or more sensors (e.g., sensor inputs). The sensor data may be received from, for example and without limitation, global navigation satellite systems (“GNSS”) sensor(s)858(e.g., Global Positioning System sensor(s)), RADAR sensor(s)860, ultrasonic sensor(s)862, LIDAR sensor(s)864, inertial measurement unit (IMU) sensor(s)866(e.g., accelerometer(s), gyroscope(s), magnetic compass(es), magnetometer(s), etc.), microphone(s)896, stereo camera(s)868, wide-view camera(s)870(e.g., fisheye cameras), infrared camera(s)872, surround camera(s)874(e.g., 360 degree cameras), long-range and/or mid-range camera(s)898, speed sensor(s)844(e.g., for measuring the speed of the vehicle800), vibration sensor(s)842, steering sensor(s)840, brake sensor(s) (e.g., as part of the brake sensor system846), and/or other sensor types.

One or more of the controller(s)836may receive inputs (e.g., represented by input data) from an instrument cluster832of the vehicle800and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display834, an audible annunciator, a loudspeaker, and/or via other components of the vehicle800. The outputs may include information such as vehicle velocity, speed, time, map data (e.g., the High Definition (“HD”) map822ofFIG.8C), location data (e.g., the vehicle's800location, such as on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and status of objects as perceived by the controller(s)836, etc. For example, the HMI display834may display information about the presence of one or more objects (e.g., a street sign, caution sign, traffic light changing, etc.), and/or information about driving maneuvers the vehicle has made, is making, or will make (e.g., changing lanes now, taking exit34B in two miles, etc.).

The vehicle800further includes a network interface824which may use one or more wireless antenna(s)826and/or modem(s) to communicate over one or more networks. For example, the network interface824may be capable of communication over Long-Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile communication (“GSM”), IMT-CDMA Multi-Carrier (“CDMA2000”), etc. The wireless antenna(s)826may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.), using local area network(s), such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc., and/or low power wide-area network(s) (“LPWANs”), such as LoRaWAN, SigFox, etc.

FIG.8Bis an example of camera locations and fields of view for the example autonomous vehicle800ofFIG.8A, in accordance with some embodiments of the present disclosure. The cameras and respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and/or alternative cameras may be included and/or the cameras may be located at different locations on the vehicle800.

A variety of cameras may be used in a front-facing configuration, including, for example, a monocular camera platform that includes a complementary metal oxide semiconductor (“CMOS”) color imager. Another example may be a wide-view camera(s)870that may be used to perceive objects coming into view from the periphery (e.g., pedestrians, crossing traffic or bicycles). Although only one wide-view camera is illustrated inFIG.8B, there may be any number (including zero) of wide-view cameras870on the vehicle800. In addition, any number of long-range camera(s)898(e.g., a long-view stereo camera pair) may be used for depth-based object detection, especially for objects for which a neural network has not yet been trained. The long-range camera(s)898may also be used for object detection and classification, as well as basic object tracking.

Any number of stereo cameras868may also be included in a front-facing configuration. In at least one embodiment, one or more of stereo camera(s)868may include an integrated control unit comprising a scalable processing unit, which may provide a programmable logic (“FPGA”) and a multi-core micro-processor with an integrated Controller Area Network (“CAN”) or Ethernet interface on a single chip. Such a unit may be used to generate a 3D map of the vehicle's environment, including a distance estimate for all the points in the image. An alternative stereo camera(s)868may include a compact stereo vision sensor(s) that may include two camera lenses (one each on the left and right) and an image processing chip that may measure the distance from the vehicle to the target object and use the generated information (e.g., metadata) to activate the autonomous emergency braking and lane departure warning functions. Other types of stereo camera(s)868may be used in addition to, or alternatively from, those described herein.

Cameras with a field of view that include portions of the environment to the side of the vehicle800(e.g., side-view cameras) may be used for surround view, providing information used to create and update the occupancy grid, as well as to generate side impact collision warnings. For example, surround camera(s)874(e.g., four surround cameras874as illustrated inFIG.8B) may be positioned to on the vehicle800. The surround camera(s)874may include wide-view camera(s)870, fisheye camera(s), 360 degree camera(s), and/or the like. Four example, four fisheye cameras may be positioned on the vehicle's front, rear, and sides. In an alternative arrangement, the vehicle may use three surround camera(s)874(e.g., left, right, and rear), and may leverage one or more other camera(s) (e.g., a forward-facing camera) as a fourth surround view camera.

Cameras with a field of view that include portions of the environment to the rear of the vehicle800(e.g., rear-view cameras) may be used for park assistance, surround view, rear collision warnings, and creating and updating the occupancy grid. A wide variety of cameras may be used including, but not limited to, cameras that are also suitable as a front-facing camera(s) (e.g., long-range and/or mid-range camera(s)898, stereo camera(s)868), infrared camera(s)872, etc.), as described herein.

Each of the components, features, and systems of the vehicle800inFIG.8Care illustrated as being connected via bus802. The bus802may include a Controller Area Network (CAN) data interface (alternatively referred to herein as a “CAN bus”). A CAN may be a network inside the vehicle800used to aid in control of various features and functionality of the vehicle800, such as actuation of brakes, acceleration, braking, steering, windshield wipers, etc. A CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus may be read to find steering wheel angle, ground speed, engine revolutions per minute (RPMs), button positions, and/or other vehicle status indicators. The CAN bus may be ASIL B compliant.

Although the bus802is described herein as being a CAN bus, this is not intended to be limiting. For example, in addition to, or alternatively from, the CAN bus, FlexRay and/or Ethernet may be used. Additionally, although a single line is used to represent the bus802, this is not intended to be limiting. For example, there may be any number of busses802, which may include one or more CAN busses, one or more FlexRay busses, one or more Ethernet busses, and/or one or more other types of busses using a different protocol. In some examples, two or more busses802may be used to perform different functions, and/or may be used for redundancy. For example, a first bus802may be used for collision avoidance functionality and a second bus802may be used for actuation control. In any example, each bus802may communicate with any of the components of the vehicle800, and two or more busses802may communicate with the same components. In some examples, each SoC804, each controller836, and/or each computer within the vehicle may have access to the same input data (e.g., inputs from sensors of the vehicle800), and may be connected to a common bus, such the CAN bus.

The vehicle800may include one or more controller(s)836, such as those described herein with respect toFIG.8A. The controller(s)836may be used for a variety of functions. The controller(s)836may be coupled to any of the various other components and systems of the vehicle800, and may be used for control of the vehicle800, artificial intelligence of the vehicle800, infotainment for the vehicle800, and/or the like.

The vehicle800may include a system(s) on a chip (SoC)804. The SoC804may include CPU(s)806, GPU(s)808, processor(s)810, cache(s)812, accelerator(s)814, data store(s)816, and/or other components and features not illustrated. The SoC(s)804may be used to control the vehicle800in a variety of platforms and systems. For example, the SoC(s)804may be combined in a system (e.g., the system of the vehicle800) with an HD map822which may obtain map refreshes and/or updates via a network interface824from one or more servers (e.g., server(s)878ofFIG.8D).

The CPU(s)806may include a CPU cluster or CPU complex (alternatively referred to herein as a “CCPLEX”). The CPU(s)806may include multiple cores and/or L2 caches. For example, in some embodiments, the CPU(s)806may include eight cores in a coherent multi-processor configuration. In some embodiments, the CPU(s)806may include four dual-core clusters where each cluster has a dedicated L2 cache (e.g., a 2 MB L2 cache). The CPU(s)806(e.g., the CCPLEX) may be configured to support simultaneous cluster operation enabling any combination of the clusters of the CPU(s)806to be active at any given time.

The GPU(s)808may include an integrated GPU (alternatively referred to herein as an “iGPU”). The GPU(s)808may be programmable and may be efficient for parallel workloads. The GPU(s)808, in some examples, may use an enhanced tensor instruction set. The GPU(s)808may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96 KB storage capacity), and two or more of the streaming microprocessors may share an L2 cache (e.g., an L2 cache with a 512 KB storage capacity). In some embodiments, the GPU(s)808may include at least eight streaming microprocessors. The GPU(s)808may use compute application programming interface(s) (API(s)). In addition, the GPU(s)808may use one or more parallel computing platforms and/or programming models (e.g., NVIDIA's CUDA).

The GPU(s)808may include unified memory technology including access counters to allow for more accurate migration of memory pages to the processor that accesses them most frequently, thereby improving efficiency for memory ranges shared between processors. In some examples, address translation services (ATS) support may be used to allow the GPU(s)808to access the CPU(s)806page tables directly. In such examples, when the GPU(s)808memory management unit (MMU) experiences a miss, an address translation request may be transmitted to the CPU(s)806. In response, the CPU(s)806may look in its page tables for the virtual-to-physical mapping for the address and transmits the translation back to the GPU(s)808. As such, unified memory technology may allow a single unified virtual address space for memory of both the CPU(s)806and the GPU(s)808, thereby simplifying the GPU(s)808programming and porting of applications to the GPU(s)808.

In addition, the GPU(s)808may include an access counter that may keep track of the frequency of access of the GPU(s)808to memory of other processors. The access counter may help ensure that memory pages are moved to the physical memory of the processor that is accessing the pages most frequently.

The SoC(s)804may include any number of cache(s)812, including those described herein. For example, the cache(s)812may include an L3 cache that is available to both the CPU(s)806and the GPU(s)808(e.g., that is connected both the CPU(s)806and the GPU(s)808). The cache(s)812may include a write-back cache that may keep track of states of lines, such as by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4 MB or more, depending on the embodiment, although smaller cache sizes may be used.

The SoC(s)804may include an arithmetic logic unit(s) (ALU(s)) which may be leveraged in performing processing with respect to any of the variety of tasks or operations of the vehicle800—such as processing DNNs. In addition, the SoC(s)804may include a floating point unit(s) (FPU(s))—or other math coprocessor or numeric coprocessor types—for performing mathematical operations within the system. For example, the SoC(s)804may include one or more FPUs integrated as execution units within a CPU(s)806and/or GPU(s)808.

The SoC(s)804may include one or more accelerators814(e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC(s)804may include a hardware acceleration cluster that may include optimized hardware accelerators and/or large on-chip memory. The large on-chip memory (e.g., 4 MB of SRAM), may enable the hardware acceleration cluster to accelerate neural networks and other calculations. The hardware acceleration cluster may be used to complement the GPU(s)808and to off-load some of the tasks of the GPU(s)808(e.g., to free up more cycles of the GPU(s)808for performing other tasks). As an example, the accelerator(s)814may be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be amenable to acceleration. The term “CNN,” as used herein, may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and Fast RCNNs (e.g., as used for object detection).

The DLA(s) may perform any function of the GPU(s)808, and by using an inference accelerator, for example, a designer may target either the DLA(s) or the GPU(s)808for any function. For example, the designer may focus processing of CNNs and floating point operations on the DLA(s) and leave other functions to the GPU(s)808and/or other accelerator(s)814.

The SoC(s)804may include data store(s)816(e.g., memory). The data store(s)816may be on-chip memory of the SoC(s)804, which may store neural networks to be executed on the GPU and/or the DLA. In some examples, the data store(s)816may be large enough in capacity to store multiple instances of neural networks for redundancy and safety. The data store(s)812may comprise L2 or L3 cache(s)812. Reference to the data store(s)816may include reference to the memory associated with the PVA, DLA, and/or other accelerator(s)814, as described herein.

The SoC(s)804may include one or more processor(s)810(e.g., embedded processors). The processor(s)810may include a boot and power management processor that may be a dedicated processor and subsystem to handle boot power and management functions and related security enforcement. The boot and power management processor may be a part of the SoC(s)804boot sequence and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, assistance in system low power state transitions, management of SoC(s)804thermals and temperature sensors, and/or management of the SoC(s)804power states. Each temperature sensor may be implemented as a ring-oscillator whose output frequency is proportional to temperature, and the SoC(s)804may use the ring-oscillators to detect temperatures of the CPU(s)806, GPU(s)808, and/or accelerator(s)814. If temperatures are determined to exceed a threshold, the boot and power management processor may enter a temperature fault routine and put the SoC(s)804into a lower power state and/or put the vehicle800into a chauffeur to safe stop mode (e.g., bring the vehicle800to a safe stop).

The processor(s)810may further include a real-time camera engine that may include a dedicated processor subsystem for handling real-time camera management.

The processor(s)810may further include a high-dynamic range signal processor that may include an image signal processor that is a hardware engine that is part of the camera processing pipeline.

The video image compositor may also be configured to perform stereo rectification on input stereo lens frames. The video image compositor may further be used for user interface composition when the operating system desktop is in use, and the GPU(s)808is not required to continuously render new surfaces. Even when the GPU(s)808is powered on and active doing 3D rendering, the video image compositor may be used to offload the GPU(s)808to improve performance and responsiveness.

The SoC(s)804may further include a broad range of peripheral interfaces to enable communication with peripherals, audio codecs, power management, and/or other devices. The SoC(s)804may be used to process data from cameras (e.g., connected over Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LIDAR sensor(s)864, RADAR sensor(s)860, etc. that may be connected over Ethernet), data from bus802(e.g., speed of vehicle800, steering wheel position, etc.), data from GNSS sensor(s)858(e.g., connected over Ethernet or CAN bus). The SoC(s)804may further include dedicated high-performance mass storage controllers that may include their own DMA engines, and that may be used to free the CPU(s)806from routine data management tasks.

The SoC(s)804may be an end-to-end platform with a flexible architecture that spans automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and makes efficient use of computer vision and ADAS techniques for diversity and redundancy, provides a platform for a flexible, reliable driving software stack, along with deep learning tools. The SoC(s)804may be faster, more reliable, and even more energy-efficient and space-efficient than conventional systems. For example, the accelerator(s)814, when combined with the CPU(s)806, the GPU(s)808, and the data store(s)816, may provide for a fast, efficient platform for level 3-5 autonomous vehicles.

In some examples, a CNN for facial recognition and vehicle owner identification may use data from camera sensors to identify the presence of an authorized driver and/or owner of the vehicle800. The always on sensor processing engine may be used to unlock the vehicle when the owner approaches the driver door and turn on the lights, and, in security mode, to disable the vehicle when the owner leaves the vehicle. In this way, the SoC(s)804provide for security against theft and/or carjacking.

The vehicle may include a CPU(s)818(e.g., discrete CPU(s), or dCPU(s)), that may be coupled to the SoC(s)804via a high-speed interconnect (e.g., PCIe). The CPU(s)818may include an X86 processor, for example. The CPU(s)818may be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and the SoC(s)804, and/or monitoring the status and health of the controller(s)836and/or infotainment SoC830, for example.

The vehicle800may include a GPU(s)820(e.g., discrete GPU(s), or dGPU(s)), that may be coupled to the SoC(s)804via a high-speed interconnect (e.g., NVIDIA's NVLINK). The GPU(s)820may provide additional artificial intelligence functionality, such as by executing redundant and/or different neural networks, and may be used to train and/or update neural networks based on input (e.g., sensor data) from sensors of the vehicle800.

The vehicle800may further include the network interface824which may include one or more wireless antennas826(e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface824may be used to enable wireless connectivity over the Internet with the cloud (e.g., with the server(s)878and/or other network devices), with other vehicles, and/or with computing devices (e.g., client devices of passengers). To communicate with other vehicles, a direct link may be established between the two vehicles and/or an indirect link may be established (e.g., across networks and over the Internet). Direct links may be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link may provide the vehicle800information about vehicles in proximity to the vehicle800(e.g., vehicles in front of, on the side of, and/or behind the vehicle800). This functionality may be part of a cooperative adaptive cruise control functionality of the vehicle800.

The vehicle800may further include data store(s)828which may include off-chip (e.g., off the SoC(s)804) storage. The data store(s)828may include one or more storage elements including RAM, SRAM, DRAM, VRAM, Flash, hard disks, and/or other components and/or devices that may store at least one bit of data.

The vehicle800may further include GNSS sensor(s)858. The GNSS sensor(s)858(e.g., GPS, assisted GPS sensors, differential GPS (DGPS) sensors, etc.), to assist in mapping, perception, occupancy grid generation, and/or path planning functions. Any number of GNSS sensor(s)858may be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet to Serial (RS-232) bridge.

The vehicle800may further include RADAR sensor(s)860. The RADAR sensor(s)860may be used by the vehicle800for long-range vehicle detection, even in darkness and/or severe weather conditions. RADAR functional safety levels may be ASIL B. The RADAR sensor(s)860may use the CAN and/or the bus802(e.g., to transmit data generated by the RADAR sensor(s)860) for control and to access object tracking data, with access to Ethernet to access raw data in some examples. A wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor(s)860may be suitable for front, rear, and side RADAR use. In some example, Pulse Doppler RADAR sensor(s) are used.

The vehicle800may further include ultrasonic sensor(s)862. The ultrasonic sensor(s)862, which may be positioned at the front, back, and/or the sides of the vehicle800, may be used for park assist and/or to create and update an occupancy grid. A wide variety of ultrasonic sensor(s)862may be used, and different ultrasonic sensor(s)862may be used for different ranges of detection (e.g., 2.5 m, 4 m). The ultrasonic sensor(s)862may operate at functional safety levels of ASIL B.

The vehicle800may include LIDAR sensor(s)864. The LIDAR sensor(s)864may be used for object and pedestrian detection, emergency braking, collision avoidance, and/or other functions. The LIDAR sensor(s)864may be functional safety level ASIL B. In some examples, the vehicle800may include multiple LIDAR sensors864(e.g., two, four, six, etc.) that may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

The vehicle may further include IMU sensor(s)866. The IMU sensor(s)866may be located at a center of the rear axle of the vehicle800, in some examples. The IMU sensor(s)866may include, for example and without limitation, an accelerometer(s), a magnetometer(s), a gyroscope(s), a magnetic compass(es), and/or other sensor types. In some examples, such as in six-axis applications, the IMU sensor(s)866may include accelerometers and gyroscopes, while in nine-axis applications, the IMU sensor(s)866may include accelerometers, gyroscopes, and magnetometers.

In some embodiments, the IMU sensor(s)866may be implemented as a miniature, high performance GPS-Aided Inertial Navigation System (GPS/INS) that combines micro-electro-mechanical systems (MEMS) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor(s)866may enable the vehicle800to estimate heading without requiring input from a magnetic sensor by directly observing and correlating the changes in velocity from GPS to the IMU sensor(s)866. In some examples, the IMU sensor(s)866and the GNSS sensor(s)858may be combined in a single integrated unit.

The vehicle may include microphone(s)896placed in and/or around the vehicle800. The microphone(s)896may be used for emergency vehicle detection and identification, among other things.

The vehicle may further include any number of camera types, including stereo camera(s)868, wide-view camera(s)870, infrared camera(s)872, surround camera(s)874, long-range and/or mid-range camera(s)898, and/or other camera types. The cameras may be used to capture image data around an entire periphery of the vehicle800. The types of cameras used depends on the embodiments and requirements for the vehicle800, and any combination of camera types may be used to provide the necessary coverage around the vehicle800. In addition, the number of cameras may differ depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and/or another number of cameras. The cameras may support, as an example and without limitation, Gigabit Multimedia Serial Link (GMSL) and/or Gigabit Ethernet. Each of the camera(s) is described with more detail herein with respect toFIG.8AandFIG.8B.

The vehicle800may further include vibration sensor(s)842. The vibration sensor(s)842may measure vibrations of components of the vehicle, such as the axle(s). For example, changes in vibrations may indicate a change in road surfaces. In another example, when two or more vibration sensors842are used, the differences between the vibrations may be used to determine friction or slippage of the road surface (e.g., when the difference in vibration is between a power-driven axle and a freely rotating axle).

The vehicle800may include an ADAS system838. The ADAS system838may include a SoC, in some examples. The ADAS system838may include autonomous/adaptive/automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warnings (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning systems (CWS), lane centering (LC), and/or other features and functionality.

The ACC systems may use RADAR sensor(s)860, LIDAR sensor(s)864, and/or a camera(s). The ACC systems may include longitudinal ACC and/or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of the vehicle800and automatically adjust the vehicle speed to maintain a safe distance from vehicles ahead. Lateral ACC performs distance keeping, and advises the vehicle800to change lanes when necessary. Lateral ACC is related to other ADAS applications such as LCA and CWS.

LKA systems are a variation of LDW systems. LKA systems provide steering input or braking to correct the vehicle800if the vehicle800starts to exit the lane.

The vehicle800may further include the infotainment SoC830(e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as a SoC, the infotainment system may not be a SoC, and may include two or more discrete components. The infotainment SoC830may include a combination of hardware and software that may be used to provide audio (e.g., music, a personal digital assistant, navigational instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and/or information services (e.g., navigation systems, rear-parking assistance, a radio data system, vehicle related information such as fuel level, total distance covered, brake fuel level, oil level, door open/close, air filter information, etc.) to the vehicle800. For example, the infotainment SoC830may radios, disk players, navigation systems, video players, USB and Bluetooth connectivity, carputers, in-car entertainment, Wi-Fi, steering wheel audio controls, hands free voice control, a heads-up display (HUD), an HMI display834, a telematics device, a control panel (e.g., for controlling and/or interacting with various components, features, and/or systems), and/or other components. The infotainment SoC830may further be used to provide information (e.g., visual and/or audible) to a user(s) of the vehicle, such as information from the ADAS system838, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and/or other information.

The infotainment SoC830may include GPU functionality. The infotainment SoC830may communicate over the bus802(e.g., CAN bus, Ethernet, etc.) with other devices, systems, and/or components of the vehicle800. In some examples, the infotainment SoC830may be coupled to a supervisory MCU such that the GPU of the infotainment system may perform some self-driving functions in the event that the primary controller(s)836(e.g., the primary and/or backup computers of the vehicle800) fail. In such an example, the infotainment SoC830may put the vehicle800into a chauffeur to safe stop mode, as described herein.

The vehicle800may further include an instrument cluster832(e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster832may include a controller and/or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster832may include a set of instrumentation such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicators, gearshift position indicator, seat belt warning light(s), parking-brakc warning light(s), engine-malfunction light(s), airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and/or shared among the infotainment SoC830and the instrument cluster832. In other words, the instrument cluster832may be included as part of the infotainment SoC830, or vice versa.

FIG.8Dis a system diagram for communication between cloud-based server(s) and the example autonomous vehicle800ofFIG.8A, in accordance with some embodiments of the present disclosure. The system876may include server(s)878, network(s)890, and vehicles, including the vehicle800. The server(s)878may include a plurality of GPUs884(A)-884(H) (collectively referred to herein as GPUs884), PCIe switches882(A)-882(H) (collectively referred to herein as PCIe switches882), and/or CPUs880(A)-880(B) (collectively referred to herein as CPUs880). The GPUs884, the CPUs880, and the PCIe switches may be interconnected with high-speed interconnects such as, for example and without limitation, NVLink interfaces888developed by NVIDIA and/or PCIe connections886. In some examples, the GPUs884arc connected via NVLink and/or NVSwitch SoC and the GPUs884and the PCIe switches882are connected via PCIe interconnects. Although eight GPUs884, two CPUs880, and two PCIc switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the server(s)878may include any number of GPUs884, CPUs880, and/or PCIe switches. For example, the server(s)878may each include eight, sixteen, thirty-two, and/or more GPUs884.

The server(s)878may receive, over the network(s)890and from the vehicles, image data representative of images showing unexpected or changed road conditions, such as recently commenced road-work. The server(s)878may transmit, over the network(s)890and to the vehicles, neural networks892, updated neural networks892, and/or map information894, including information regarding traffic and road conditions. The updates to the map information894may include updates for the HD map822, such as information regarding construction sites, potholes, detours, flooding, and/or other obstructions. In some examples, the neural networks892, the updated neural networks892, and/or the map information894may have resulted from new training and/or experiences represented in data received from any number of vehicles in the environment, and/or based on training performed at a datacenter (e.g., using the server(s)878and/or other servers).

In some examples, the server(s)878may receive data from the vehicles and apply the data to up-to-date real-time neural networks for real-time intelligent inferencing. The server(s)878may include deep-learning supercomputers and/or dedicated AI computers powered by GPU(s)884, such as a DGX and DGX Station machines developed by NVIDIA. However, in some examples, the server(s)878may include deep learning infrastructure that use only CPU-powered datacenters.

The deep-learning infrastructure of the server(s)878may be capable of fast, real-time inferencing, and may use that capability to evaluate and verify the health of the processors, software, and/or associated hardware in the vehicle800. For example, the deep-learning infrastructure may receive periodic updates from the vehicle800, such as a sequence of images and/or objects that the vehicle800has located in that sequence of images (e.g., via computer vision and/or other machine learning object classification techniques). The deep-learning infrastructure may run its own neural network to identify the objects and compare them with the objects identified by the vehicle800and, if the results do not match and the infrastructure concludes that the AI in the vehicle800is malfunctioning, the server(s)878may transmit a signal to the vehicle800instructing a fail-safe computer of the vehicle800to assume control, notify the passengers, and complete a safe parking maneuver.

For inferencing, the server(s)878may include the GPU(s)884and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-powered servers and inference acceleration may make real-time responsiveness possible. In other examples, such as where performance is less critical, servers powered by CPUs, FPGAs, and other processors may be used for inferencing.

Example Computing Device

FIG.9is a block diagram of an example computing device(s)900suitable for use in implementing some embodiments of the present disclosure. Computing device900may include an interconnect system902that directly or indirectly couples the following devices: memory904, one or more central processing units (CPUs)906, one or more graphics processing units (GPUs)908, a communication interface910, input/output (I/O) ports912, input/output components914, a power supply916, one or more presentation components918(e.g., display(s)), and one or more logic units920. In at least one embodiment, the computing device(s)900may comprise one or more virtual machines (VMs), and/or any of the components thereof may comprise virtual components (e.g., virtual hardware components). For non-limiting examples, one or more of the GPUs908may comprise one or more vGPUs, one or more of the CPUs906may comprise one or more vCPUs, and/or one or more of the logic units920may comprise one or more virtual logic units. As such, a computing device(s)900may include discrete components (e.g., a full GPU dedicated to the computing device900), virtual components (e.g., a portion of a GPU dedicated to the computing device900), or a combination thereof.

Although the various blocks ofFIG.9are shown as connected via the interconnect system902with lines, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component918, such as a display device, may be considered an I/O component914(e.g., if the display is a touch screen). As another example, the CPUs906and/or GPUs908may include memory (e.g., the memory904may be representative of a storage device in addition to the memory of the GPUs908, the CPUs906, and/or other components). In other words, the computing device ofFIG.9is merely illustrative. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “hand-held device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and/or other device or system types, as all are contemplated within the scope of the computing device ofFIG.9.

The interconnect system902may represent one or more links or busses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system902may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standards association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and/or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU906may be directly connected to the memory904. Further, the CPU906may be directly connected to the GPU908. Where there is direct, or point-to-point connection between components, the interconnect system902may include a PCIe link to carry out the connection. In these examples, a PCI bus need not be included in the computing device900.

The CPU(s)906may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device900to perform one or more of the methods and/or processes described herein. The CPU(s)906may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) that are capable of handling a multitude of software threads simultaneously. The CPU(s)906may include any type of processor, and may include different types of processors depending on the type of computing device900implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device900, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device900may include one or more CPUs906in addition to one or more microprocessors or supplementary co-processors, such as math co-processors.

In addition to or alternatively from the CPU(s)906, the GPU(s)908may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device900to perform one or more of the methods and/or processes described herein. One or more of the GPU(s)908may be an integrated GPU (e.g., with one or more of the CPU(s)906and/or one or more of the GPU(s)908may be a discrete GPU. In embodiments, one or more of the GPU(s)908may be a coprocessor of one or more of the CPU(s)906. The GPU(s)908may be used by the computing device900to render graphics (e.g., 3D graphics) or perform general purpose computations. For example, the GPU(s)908may be used for General-Purpose computing on GPUs (GPGPU). The GPU(s)908may include hundreds or thousands of cores that are capable of handling hundreds or thousands of software threads simultaneously. The GPU(s)908may generate pixel data for output images in response to rendering commands (e.g., rendering commands from the CPU(s)906received via a host interface). The GPU(s)908may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory may be included as part of the memory904. The GPU(s)908may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined together, each GPU908may generate pixel data or GPGPU data for different portions of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory, or may share memory with other GPUs.

In addition to or alternatively from the CPU(s)906and/or the GPU(s)908, the logic unit(s)920may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device900to perform one or more of the methods and/or processes described herein. In embodiments, the CPU(s)906, the GPU(s)908, and/or the logic unit(s)920may discretely or jointly perform any combination of the methods, processes and/or portions thereof. One or more of the logic units920may be part of and/or integrated in one or more of the CPU(s)906and/or the GPU(s)908and/or one or more of the logic units920may be discrete components or otherwise external to the CPU(s)906and/or the GPU(s)908. In embodiments, one or more of the logic units920may be a coprocessor of one or more of the CPU(s)906and/or one or more of the GPU(s)908.

The communication interface910may include one or more receivers, transmitters, and/or transceivers that enable the computing device900to communicate with other computing devices via an electronic communication network, included wired and/or wireless communications. The communication interface910may include components and functionality to enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating over Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and/or the Internet. In one or more embodiments, logic unit(s)920and/or communication interface910may include one or more data processing units (DPUs) to transmit data received over a network and/or through interconnect system902directly to (e.g., a memory of) one or more GPU(s)908.

The I/O ports912may enable the computing device900to be logically coupled to other devices including the I/O components914, the presentation component(s)918, and/or other components, some of which may be built in to (e.g., integrated in) the computing device900. Illustrative I/O components914include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I/O components914may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs may be transmitted to an appropriate network element for further processing. An NUI may implement any combination of speech recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of the computing device900. The computing device900may be include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations of these, for gesture detection and recognition. Additionally, the computing device900may include accelerometers or gyroscopes (e.g., as part of an inertia measurement unit (IMU)) that enable detection of motion. In some examples, the output of the accelerometers or gyroscopes may be used by the computing device900to render immersive augmented reality or virtual reality.

The power supply916may include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply916may provide power to the computing device900to enable the components of the computing device900to operate.

The presentation component(s)918may include a display (e.g., a monitor, a touch screen, a television screen, a heads-up-display (HUD), other display types, or a combination thereof), speakers, and/or other presentation components. The presentation component(s)918may receive data from other components (e.g., the GPU(s)908, the CPU(s)906, DPUs, etc.), and output the data (e.g., as an image, video, sound, etc.).

Example Data Center

FIG.10illustrates an example data center1000that may be used in at least one embodiments of the present disclosure. The data center1000may include a data center infrastructure layer1010, a framework layer1020, a software layer1030, and/or an application layer1040.

As shown inFIG.10, the data center infrastructure layer1010may include a resource orchestrator1012, grouped computing resources1014, and node computing resources (“node C.R.s”)1016(1)-1016(N), where “N” represents any whole, positive integer. In at least one embodiment, node C.R.s1016(1)-1016(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input/output (NW I/O) devices, network switches, virtual machines (VMs), power modules, and/or cooling modules, etc. In some embodiments, one or more node C.R.s from among node C.R.s1016(1)-1016(N) may correspond to a server having one or more of the above-mentioned computing resources. In addition, in some embodiments, the node C.R.s1016(1)-10161(N) may include one or more virtual components, such as vGPUs, vCPUs, and/or the like, and/or one or more of the node C.R.s1016(1)-1016(N) may correspond to a virtual machine (VM).

In at least one embodiment, grouped computing resources1014may include separate groupings of node C.R.s1016housed within one or more racks (not shown), or many racks housed in data centers at various geographical locations (also not shown). Separate groupings of node C.R.s1016within grouped computing resources1014may include grouped compute, network, memory or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R.s1016including CPUs, GPUs, DPUs, and/or other processors may be grouped within one or more racks to provide compute resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and/or network switches, in any combination.

The resource orchestrator1012may configure or otherwise control one or more node C.R.s1016(1)-1016(N) and/or grouped computing resources1014. In at least one embodiment, resource orchestrator1012may include a software design infrastructure (SDI) management entity for the data center1000. The resource orchestrator1012may include hardware, software, or some combination thereof.

In at least one embodiment, as shown inFIG.10, framework layer1020may include a job scheduler1033, a configuration manager1034, a resource manager1036, and/or a distributed file system1038. The framework layer1020may include a framework to support software1032of software layer1030and/or one or more application(s)1042of application layer1040. The software1032or application(s)1042may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud and Microsoft Azure. The framework layer1020may be, but is not limited to, a type of free and open-source software web application framework such as Apache Spark™ (hereinafter “Spark”) that may utilize distributed file system1038for large-scale data processing (e.g., “big data”). In at least one embodiment, job scheduler1033may include a Spark driver to facilitate scheduling of workloads supported by various layers of data center1000. The configuration manager1034may be capable of configuring different layers such as software layer1030and framework layer1020including Spark and distributed file system1038for supporting large-scale data processing. The resource manager1036may be capable of managing clustered or grouped computing resources mapped to or allocated for support of distributed file system1038and job scheduler1033. In at least one embodiment, clustered or grouped computing resources may include grouped computing resource1014at data center infrastructure layer1010. The resource manager1036may coordinate with resource orchestrator1012to manage these mapped or allocated computing resources.

In at least one embodiment, software1032included in software layer1030may include software used by at least portions of node C.R.s1016(1)-1016(N), grouped computing resources1014, and/or distributed file system1038of framework layer1020. One or more types of software may include, but are not limited to, Internet web page search software, e-mail virus scan software, database software, and streaming video content software.

In at least one embodiment, application(s)1042included in application layer1040may include one or more types of applications used by at least portions of node C.R.s1016(1)-1016(N), grouped computing resources1014, and/or distributed file system1038of framework layer1020. One or more types of applications may include, but are not limited to, any number of a genomics application, a cognitive compute, and a machine learning application, including training or inferencing software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and/or other machine learning applications used in conjunction with one or more embodiments.

In at least one embodiment, any of configuration manager1034, resource manager1036, and resource orchestrator1012may implement any number and type of self-modifying actions based on any amount and type of data acquired in any technically feasible fashion. Self-modifying actions may relieve a data center operator of data center1000from making possibly bad configuration decisions and possibly avoiding underutilized and/or poor performing portions of a data center.

The data center1000may include tools, services, software or other resources to train one or more machine learning models or predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model(s) may be trained by calculating weight parameters according to a neural network architecture using software and/or computing resources described above with respect to the data center1000. In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks may be used to infer or predict information using resources described above with respect to the data center1000by using weight parameters calculated through one or more training techniques, such as but not limited to those described herein.

Example Network Environments

Network environments suitable for use in implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and/or other device types. The client devices, servers, and/or other device types (e.g., each device) may be implemented on one or more instances of the computing device(s)900ofFIG.9—e.g., each device may include similar components, features, and/or functionality of the computing device(s)900. In addition, where backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may be included as part of a data center1000, an example of which is described in more detail herein with respect toFIG.10.

Compatible network environments may include one or more peer-to-peer network environments—in which case a server may not be included in a network environment- and one or more client-server network environments—in which case one or more servers may be included in a network environment. In peer-to-peer network environments, functionality described herein with respect to a server(s) may be implemented on any number of client devices.

Example Paragraphs

A. A method comprising: rendering one or more virtual images from one or more perspectives using a magnified portion of a three dimensional (3D) representation of an environment, the magnified portion of the 3D representation corresponding to one or more first predicted locations in the environment; obtaining, based at least on applying the one or more virtual images to one or more machine learning models, one or more second predicted locations in the environment; and performing one or more control operations associated with a machine in the environment based at least on the one or more second predicted locations.

B. The method as recited in paragraph A, further comprising applying, to the one or more machine learning models substantially contemporaneously with the one or more virtual images, one or more token embeddings corresponding to a structured language command, wherein the obtaining of the one or more second predicted locations is further based at least on the applying of the one or more token embeddings.

C. The method as recited in any one of paragraphs A-B, further comprising obtaining, based at least on the applying of the one or more virtual images to the one or more machine learning models, one or more heatmaps indicative of the one or more second predicted locations.

D. The method as recited in any one of paragraphs A-C, wherein the one or more second predicted locations correspond to one or more refined versions of the one or more first predicted locations such that one or more first confidence scores associated with the one or more first predicted locations are less than one or more second confidences scores associated with the one or more second predicted locations.

E. The method as recited in any one of paragraphs A-D, wherein the one or more machine learning models include one or more convex upsampling layers to increase one or more spatial dimensions of one or more feature maps corresponding to the one or more virtual images.

F. The method as recited in any one of paragraphs A-E, wherein the one or more virtual images are rendered such that one or more sizes associated with the one or more virtual images are rationally divisible by one or more patch sizes associated with the one or more machine learning models.

G. The method as recited in any one of paragraphs A-F, further comprising: determining, based at least on one or more local features corresponding to the one or more second predicted locations, a degree of rotation associated with manipulating an end effector of the machine; and wherein the one or more control operations include rotating the end-effector of the machine based at least on the degree of rotation.

H. The method as recited in any one of paragraphs A-G, wherein the one or more first predicted locations and the one or more second predicted locations correspond to at least one of: one or more objects in the environment; or one or more positions associated with one or more key poses of the machine.

I. The method as recited in any one of paragraphs A-H, further comprising: generating the 3D representation of the environment based at least on applying one or more images depicting the environment to a neural network; and obtaining the one or more first predicted locations in the environment based at least on applying, to one or more second machine learning models, one or more second virtual images depicting the 3D representation of the environment from one or more second perspectives.

J. The method as recited in any one of paragraphs A-I, wherein a first zoom factor associated with the one or more virtual images is greater than a second zoom factor associated with the one or more second virtual images.

K. The method as recited in any one of paragraphs A-J, wherein the one or more second predicted locations include one or more two-dimensional (2D) space predictions corresponding to virtual images of the one or more virtual images, the method further comprising: mapping the 2D space predictions into a 3D space; generating, based at least on the mapping, one or more 3D space predictions; and performing one or more second control operations associated with the machine based at least on the one or more 3D space predictions.

L. A system comprising: one or more processors to: generate an updated version of a 3D representation of an environment, the updated version including a magnified portion of the 3D representation based at least on one or more first predictions associated with the magnified portion; apply, to one or more machine learning models, one or more images depicting the magnified portion of the 3D representation; and perform one or more operations associated with a machine in the environment based at least on one or more second predictions obtained using the one or more machine learning models.

M. The system as recited in paragraphs L, the one or more processors further to obtain, based at least on the application of the one or more images to the one or more machine learning models, one or more heatmaps indicative of one or more locations corresponding to the one or more second predictions.

N. The system as recited in any one of paragraphs L-M, wherein the one or more second predictions correspond to one or more refined versions of the one or more first predictions.

O. The system as recited in any one of paragraphs L-N, wherein the one or more second predictions are associated with one or more greater confidence scores than the one or more first predictions.

P. The system as recited in any one of paragraphs L-O, wherein the one or more images are generated such that one or more sizes associated with the one or more images are rationally divisible by one or more patch sizes associated with the one or more machine learning models.

Q. The system as recited in any one of paragraphs L-P, The system as recited in any one of paragraphs M-, the one or more processors further to: determine, based at least on one or more local features corresponding to the one or more second predictions, a degree of rotation associated with manipulating an end effector of the machine; and wherein the one or more operations include rotating the end-effector of the machine based at least on the degree of rotation.

R. The system as recited in any one of paragraphs L-Q, wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using a large language model; a system for performing operations using one or more vision language models (VLMs); a system for performing operations using one or more multi-modal language models; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

S. At least one processor comprising: processing circuitry to perform one or more operations associated with a machine in an environment using one or more updated predictions, the one or more updated predictions generated based at least on applying, to one or more machine learning models, one or more images depicting a magnified portion of a 3D representation of the environment, the magnified portion corresponding to one or more locations associated with one or more initial predictions.

T. The processor as recited in paragraph S, wherein the processor is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing one or more simulation operations; a system for performing one or more digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing one or more deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing one or more generative AI operations; a system for performing operations using a large language model; a system for performing operations using one or more vision language models (VLMs); a system for performing operations using one or more multi-modal language models; a system for performing one or more conversational AI operations; a system for generating synthetic data; a system for presenting at least one of virtual reality content, augmented reality content, or mixed reality content; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

U. A method comprising: determining an area of interest surrounding an object in a virtual representation of an environment; generating, using a virtual camera and based at least on zooming-in the virtual camera to magnify a view of the area of interest, an image depicting a magnified view of the area of interest; determining, using a machine learning model to analyze the image, a location of the object in the environment; and causing a machine to manipulate the object based at least on the location.

V. The method as recited in paragraph U, wherein the determining the area of interest comprises: applying a second image depicting the virtual representation of the environment to a second machine learning model; determining, using the second machine learning model, a predicted location of the object in the environment; and determining the area of interest based at least on the predicted location.

W. The method as recited in any one of paragraphs U-V, further comprising: generating, using a second virtual camera and based at least on zooming-in the second virtual camera to magnify a second view of the area of interest, a second image depicting a second magnified view of the area of interest from a different perspective than the image; and wherein the determining the location of the object is based at least on using the machine learning model to analyze the image and the second image.