PROTECTING SENSITIVE DATA ASSOCIATED WITH TRAINING MACHINE LEARNING MODELS

Embodiments of the present disclosure relate to a method of performing one or more operations using a machine learning model. In some embodiments, the machine learning model may be trained, at least in part, using synthetic training data that may have been generated using one or more generative machine learning models. Further, the one or more generative machine learning models may be trained to generate the synthetic training data based at least on real training data designated for protection.

BACKGROUND

Some machine learning systems-such as those corresponding to supervised learning models, unsupervised learning models, semi-supervised learning models, reinforcement learning models, deep learning models, and others—may be trained using one or more labeled datasets. For example, a dataset may include images of different facial expressions where the images are labeled identifying one or more characteristics included in a facial expression—e.g., smile, frown, lifted eyebrows, dimples, eyes open, eyes closed, etc. Continuing the example, a machine learning model may be trained using the labeled dataset such that, when introduced to new input data including one or more facial expressions, the trained or deployed machine learning model may be configured to identify one or more facial expressions and/or characteristics of facial expressions in the new input data.

Some machine learning models may be trained to analyze and/or recognize one or more characteristics included in sensitive data, which may include any data designated for protection (e.g., medical imaging data, biometric data, etc.). Some traditional approaches to training machine learning models to recognize characteristics included in sensitive data may include using one or more federated learning techniques to gain access to one or more training datasets including sensitive data. One limitation of using federated learning techniques is that the quality of the dataset cannot be determined until after the machine learning model has already been trained. Furthermore, the machine learning model may need to be moved between organizations, entities, users, and/or other systems that may include differing datasets including data and/or information designated for protection.

Further, other techniques for training machine learning models to recognize characteristics included in sensitive data may include one or more techniques designed to modify sensitive data such that the data is no longer designated for protection. As an example technique to modify real data, in the context of using personal, medical data to train one or more machine learning models, the medical data may be scrubbed of any personal information associated with the data (e.g., metadata, labels, etc.).

However, such approaches may be time consuming and costly to ensure the data is scrubbed sufficiently to overcome the designation for protection. Further, medical data, even when scrubbed, may include personal identifying information that may still lead back to the individual(s) and/or group(s) whose data is/are in the data due to the unique characteristics associated with any one individual's medical history. Additionally or alternatively, different countries, states, or governing bodies may have differing regulations regarding protected information (e.g., HIPAA requirements in the United States as opposed to GDPR regulations in the European Union). Therefore, sufficiently scrubbing and/or encrypting certain data in one jurisdiction may not be sufficient for another jurisdiction.

SUMMARY

According to one or more embodiments of the present disclosure, synthetic training data may be aggregated. In particular, synthetic training data may have been generated using one or more generative machine learning models that may have been trained using real data designated for protection. In some embodiments, the real training data may be collected or measured in the real-world and the synthetic training data may be artificial in that the synthetic training data may be generated by the one or more generative machine learning models instead of being collected or measured in the real-world.

Additionally or alternatively, one or more analytical machine learning models may be trained using the aggregated synthetic training data. In particular, the one or more analytical machine learning models may be trained such that the one or more analytical machine learning models may be configured to analyze one or more characteristics corresponding to real input data that may include one or more characteristics included in the real training data.

The embodiments of the present disclosure may provide the benefits of using large datasets to train machine learning models that analyze sensitive data while also protecting sensitive data by using synthetic data for the training instead of sensitive data.

DETAILED DESCRIPTION

One or more embodiments of the present disclosure may relate to generating synthetic data using a generative machine learning model, where the generative machine learning model may generate the synthetic data based on sensitive data. Further, in some embodiments, the synthetic data generated using the generative machine learning model may be used to train an analytical machine learning model to analyze one or more characteristics of real input data.

In these and other embodiments, reference to “synthetic data” may include data and/or annotated information that may be generated using one or more computer simulations, machine learning models, machine learning systems, etc. rather than data collected and/or measured in a real-world environment. In some instances, synthetic data may include one or more characteristics included in data collected, measured, and/or generated in a real-world environment referred to herein as “real-world data” or “real data.”

In some embodiments, the generative model may be trained to generate synthetic data using sensitive data. In these and other embodiments, the sensitive data may be labeled and/or otherwise characterized in such a way that the generative model may generate synthetic data including one or more of the same characteristics as the sensitive data; however, the synthetic data generated using the generative model may not be designated for protection. Additionally or alternatively, the generative model may generate synthetic data that may not be traceable back to the sensitive data used to generate the synthetic data.

In some embodiments, the generative model may be configured to generate one or more synthetic datasets based on one or more corresponding datasets including sensitive data, referred to herein as “sensitive datasets.” For example, the generative machine learning model may be trained using a first sensitive dataset to generate first synthetic data. Continuing the example, the generative model may additionally be trained using a second sensitive dataset to generate second synthetic data. Further, the first synthetic data and the second synthetic data may be combined to create a single synthetic dataset that may be used to train the analytical machine learning model.

In some embodiments, the term “analytical machine learning model” or “analytical model,” may refer to any suitable algorithms, computer systems, neural networks, deep learning models, and/or other models that may be configured to analyze one or more characteristics corresponding to input data. In some embodiments of the present disclosure, the analytical model may be configured to analyze one or more characteristics corresponding to real input data based on being trained using the one or more synthetic datasets.

Embodiments of the present disclosure may help improve analysis and/or recognition of one or more characteristics included in sensitive data using a machine learning model. Further, embodiments of the present disclosure may provide the benefits of using large datasets to train machine learning models that analyze sensitive data while also protecting sensitive data by using synthetic, non-sensitive data for the training while also avoiding limitations associated with other techniques such as federated learning techniques and/or data scrubbing.

One or more of the embodiments disclosed herein may relate to generating synthetic data using a generative machine learning model trained using one or more sensitive datasets, and training one or more analytical machine learning models using the synthetic data. These trained analytic models may then be used in various applications, such as in any applicable machine or system for performing one or more autonomous or semi-autonomous operations, medical imaging operations, medical diagnostic operations, etc. Example autonomous or semi-autonomous machine (e.g., ego-machines) may include, but are not limited to, vehicles (land, sea, space, and/or air), robots, robotic platforms, etc. By way of example, the ego-machine computing applications may include one or more applications that may be executed by an autonomous vehicle or semi-autonomous vehicle, such as an example autonomous or semi-autonomous vehicle or machine400(alternatively referred to herein as “vehicle400” or “ego-machine400) described with respect toFIGS.4A-4D. In the present disclosure, reference to an “autonomous vehicle” or “semi-autonomous vehicle” may include any vehicle that may be configured to perform one or more autonomous or semi-autonomous navigation or driving operations. As such, such vehicles may also include vehicles in which an operator is required or in which an operator may perform such operations as well.

In some embodiments, one or more ego-machines may be configured to collect sensitive data and/or information. In some embodiments, the sensitive data may then be used to train a generative machine learning model that may generate synthetic data used to train one or more analytical machine learning models used in one or more ego-machines, autonomous vehicles, semi-autonomous vehicles, etc.

By way of example and not limitation, one or more ego-machines may include one or more sensors configured to generate data and/or information that may be used by insurance companies to determine insurance rates for individual drivers. Continuing the example, the data generated using one or more sensors may include personal information corresponding to one or more individual drivers of the ego-machine—e.g., location data corresponding to the location of the ego-machine including home addresses, work addresses, route habits, data and/or information that may incriminate the one or more individual drivers (speeding data, reckless driving etc.).

Continuing the example, the data and/or information collected for insurance companies may be designated for protection and therefore considered sensitive data. The sensitive data may then be used to train one or more generative machine learning models to generate synthetic data that may include one or more characteristics included in the sensitive data without being designated for protection. The synthetic data may then be used to train one or more analytical machine learning models that may be included in one or more ego-machines (e.g., autonomous, or semi-autonomous vehicles). The analytical machine learning models included in the ego-machines may be trained to recognize and/or analyze real input data (e.g., sensor data). Further, because the synthetic data may not be sensitive or otherwise designated for protection, the synthetic data may be used to train the one or more analytical machine learning models to improve, for example, one or more analytical machine learning models used by the ego machine.

Disclosed embodiments may be comprised in a variety of different systems such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot, aerial systems, medial systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems implemented using an edge device, systems incorporating one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems that implement one or more language models, such as one or more large language models (LLMs) that process textual, audio, image, sensor, and/or other data types to generate one or more outputs, systems for hosting real-time streaming applications, systems for presenting one or more of virtual reality content, augmented reality content, or mixed reality content, systems for performing light transport simulation, systems for performing collaborative content creation for 3D assets, systems implemented at least partially using cloud computing resources, and/or other types of systems.

These and other embodiments of the present disclosure will be explained with reference to the accompanying figures. It is to be understood that the figures are diagrammatic and schematic representations of such example embodiments, and are not limiting, nor are they necessarily drawn to scale. In the figures, features with like numbers indicate like structure and function unless described otherwise.

With respect toFIG.1A,FIG.1Aillustrates an example environment100for training a generative machine learning model104using a sensitive dataset102and generating a synthetic dataset106using the generative machine learning model104, in accordance with one or more embodiments of the present disclosure.

The sensitive dataset102may include real data—e.g., data collected, measured, and/or generated in a real-world environment in contrast to synthetic data that may not be collected and/or measured in a real-world environment. In some embodiments, the sensitive dataset102may include real data annotated using one or more labels describing one or more characteristics of the real data.

In some embodiments, the sensitive dataset102may include real data and/or annotated information that may be sensitive, protected, and/or otherwise designated for protection, which is referred to generally herein as “sensitive data.” For example, sensitive data may include medical images (e.g., data generating using computed tomography (CT) scans, magnetic resonance imaging (MRI), ultrasound, positron emission tomography (PET), mammography, thermography, and other medical imaging systems), diagnostic results, medical notes, and other data and/or information designated for protection. Other examples of sensitive data may include company secrets, intellectual property, sensor data, video streams, client-specific data, and other sensitive information that may be designated for protection from disclosure, substitution, and/or compromise.

In some embodiments, sensitive data and/or information designated for protection may include data designated for protection by a regulatory agency. For example, data including one or more specific characteristics (e.g., personal medical information) may be regulated by the health insurance portability and accountability act (HIPAA) in the United States, the general data protection regulation (GDPR) in the European Union, and/or other regulations or regulatory agencies. Continuing the example, any data regulated by HIPAA and/or GDPR may be considered sensitive data or data that may be designated for protection—e.g., data included in the sensitive dataset102.

In some embodiments, the sensitive dataset102may include data that may be arbitrarily designated for protection by the party, system, entity, etc. that may own, use, or otherwise control data corresponding to the sensitive dataset102. For example, an entity may control and/or own data that the entity may consider to be part of their intellectual property. Continuing the example, regardless of whether the data controlled by the entity may, in actuality, be part of the entity's intellectual property, the data may be included in the sensitive dataset102.

In some embodiments, the sensitive dataset102may include sensitive data that may have originated in one or more different locations and/or data owned and/or controlled by one or more entities. For example, a first entity may control a first amount of medical imaging data and a second entity may control a second amount of medical imaging data. Continuing the example, the first amount of medical imaging data and the second amount of medical imaging data may be included in the sensitive dataset102. In some embodiments, the sensitive dataset102including data owned and/or controlled by one or more entities may be further described and/or illustrated in the present disclosure, such as, for example with respect toFIG.2A.

In some embodiments, the sensitive dataset102may include data and/or information that may include one or more overlapping characteristics such that the sensitive dataset102may be used to train one or more generative machine learning models104. For example, the sensitive dataset102may include sensitive data and/or information corresponding to medical notes generated for one or more patients. Continuing the example, the data and/or information corresponding to the one or more patients may include one or more similar characteristics (e.g., diagnostic information, weight, height, dietary restrictions, etc.) such that the data may be used in the sensitive dataset102to train one or more machine learning models (e.g., the generative machine learning model104).

The generative machine learning model104may include one or more machine learning models that may be trained using the sensitive dataset102to generate synthetic data that may be included in the synthetic dataset106. In these and other embodiments, the generative machine learning model104may include any suitable algorithms, computer models, neural networks, deep learning models, or other systems and/or models that may be trained to generate synthetic data (e.g., synthetic data that may be included in the synthetic dataset106). In some embodiments, the generative machine learning model104may be trained by learning one or more distributions associated with the sensitive data102(e.g., real training data) such that one or more second distributions associated with the synthetic dataset106are within a threshold similarity to the one or more first distributions.

Additionally or alternatively, the generative machine learning model104may be configured and/or trained to augment the sensitive data included in the sensitive dataset102. In some embodiments, augmenting the sensitive data may include adding, taking away, and/or modifying one or more features, characteristics, qualities, traits, attributes, data, metadata, etc. that may have been included in the sensitive data. In some embodiments, the generative machine learning model104may be trained using the sensitive data in the sensitive dataset102to generate synthetic data and/or augment the sensitive data to generate synthetic and/or augmented data included in the synthetic dataset106.

By way of example and not limitation, the generative machine learning model104may include a generative adversarial network (GAN), generative stochastic networks (GSN), generative moment matching networks (GMMN), deep convolutional adversarial networks (DCGAN), Wasserstein GAN (WGAN), diffusion machine learning models, transformer machine learning models, and/or other machine learning models that may be configured to generate synthetic data.

In some embodiments, the generative machine learning model104may be trained to generate synthetic data included in the synthetic dataset106where the synthetic data may not be sensitive or otherwise designated for protection. In these and other embodiments, the sensitive data may be labeled and/or otherwise characterized in such a way that the generative machine learning model104may generate synthetic data including one or more of the same characteristics as the sensitive data. However, the synthetic data generated using the generative machine learning model104may not be designated for protection. Additionally or alternatively, the generative machine learning model104may generate synthetic data that may not be traceable back to one or more individuals corresponding to data in the sensitive dataset102and/or any characteristics or features of the sensitive data that may have made the sensitive data designated for protection.

For example, the sensitive data may include medical images that may be designated for protection, in part, because of the personal identifiers corresponding to the medical images. Continuing the example, the generative machine learning model104may generate synthetic data based on an amalgamation of the medical images such that the synthetic data is not the same as any of the different medical images. In this example, the synthetic data may include one or more characteristics included in the different medical images; however, the synthetic data may not be traceable back to the medical images as a dataset, any one of the medical images included in the dataset, and/or any of the individuals associated with the medical images of the sensitive dataset102. Further continuing the example, the synthetic data may not be designated for protection because the synthetic data may not include any of the characteristics included in the medical images that may have designated the medical images for protection (e.g., personal identifying information).

In some embodiments, the generative machine learning model104may be configured to generate one or more synthetic datasets106based at least on one or more corresponding sensitive datasets102. For example, the generative machine learning model104may be trained using a first sensitive dataset to generate first synthetic data. Continuing the example, the generative machine learning model104may additionally be trained using a second sensitive dataset to generate second synthetic data. Further, the first synthetic data and the second synthetic data may be combined to create a single synthetic dataset106.

The synthetic dataset106may include data and/or annotated information that may be generated using one or more computer simulations, machine learning models, machine learning systems, etc. rather than data collected and/or measured in a real-world environment. In some instances, synthetic data may include one or more characteristics included in data collected, measured, and/or generated in a real-world environment. In some embodiments, the synthetic data that may be included in the synthetic dataset106may include one or more characteristics included in data included in the sensitive dataset102. In some embodiments, the synthetic data included in the synthetic dataset106may include all, or substantially all, of the characteristics included in data included in the sensitive dataset102.

In some embodiments, the synthetic data included in the synthetic dataset106may be used to train one or more analytical machine learning models—e.g., the analytical machine learning model108. In some embodiments, the analytical machine learning model108may be trained to analyze real data that may include one or more characteristics corresponding to synthetic data included in the synthetic dataset106.

The analytical machine learning model108may include any suitable algorithms, computer systems, neural networks, deep learning models, and/or other models that may be configured to analyze one or more characteristics corresponding to input data. By way of example and not limitation, the analytical machine learning model108may include: a supervised model, an unsupervised model, a semi-supervised model, artificial neural networks, and/or other machine learning models, systems, and/or neural networks. In some embodiments of the present disclosure, the analytical machine learning model108may be configured to analyze one or more characteristics corresponding to real input data based on being trained using the synthetic dataset106.

In some embodiments, real input data, as used in this disclosure, may refer to data measured, collected, and/or generated in a real-world environment that may include one or more characteristics included in the real data included in the sensitive dataset102that may be used to train the generative machine learning model104. In some embodiments, the real input data may not include the real data used to train the generative machine learning model104.

For example, the analytical machine learning model108may include one or more algorithms, systems, and/or models that may receive and/or otherwise obtain synthetic data including one or more characteristics. Continuing the example, the analytical machine learning model108, after being trained using the synthetic data included in the synthetic dataset106, may be configured to identify corresponding characteristics in real input data. In these and other embodiments, using the analytical machine learning model108to analyze real input data may be described and illustrated further in the present disclosure, such as, for example, with respect toFIG.1B.

By way of example and not limitation, in the context of MRI data corresponding to an anterior cruciate ligament (ACL) tear, the generative machine learning model104may generate synthetic data using the MRI data; the MRI data may include images of healthy ACLs, partially torn ACLs, fully torn ACLs, etc. Continuing the example, the synthetic data generated using the generative machine learning model104may include one or more characteristics included in the medical imaging data (e.g., new images including healthy ACLs, partially torn ACLs, fully torn ACLs etc.). Further, the synthetic data may not be traceable back to the MRI data and/or any corresponding metadata designated for protection (e.g., patient information, time, date, location, etc.) thereby rendering the synthetic data not designated for protection. Continuing the example, the synthetic data may be used to train the analytical machine learning model108to recognize one or more characteristics of an ACL tear (e.g., a healthy ACL, partially torn ACL, fully torn ACL, etc.) in one or more datasets including real input data.

By way of another example and not limitation, in the context of data corresponding to echocardiograms, the generative machine learning model104may generate synthetic data using echocardiogram data; the echocardiogram data may include sound waves, tabular data, etc. Continuing the example, the synthetic data generated and/or augmented using the generative machine learning model104may include one or more characteristics included in the electrocardiogram data (e.g., new and/or augmented sound waves corresponding to one or more structures of one or more hearts and/or tabular data corresponding to one or more electrocardiogram tests). Further, the synthetic data may not be traceable back to the echocardiogram and/or any corresponding metadata designated for protection (e.g., patient information, time, date, location, etc.) thereby rendering the synthetic data not designated for protection. Continuing the example, the synthetic data may be used to train the analytical machine learning model108to recognize and/or analyze one or more characteristics of a heart (e.g., a healthy heart structure, unhealthy heart structure, good blood flow, poor blood flow, etc.) in one or more datasets including real input data.

FIG.1Billustrates an example environment150for using the analytical machine learning model108ofFIG.1Aas trained to analyze and/or recognize one or more characteristics in real input data110, in accordance with one or more embodiments of the present disclosure. In some embodiments, the environment150may be analogous to or the same as the environment100described in the present disclosure, such as, for example, with respect toFIG.1A.

Real input data110as used in this disclosure, may refer to data measured, collected, and/or generated in a real-world environment. In some embodiments, the real input data110may include one or more different amounts of data and/or datasets that may be analyzed by the analytical machine learning model108. In some embodiments, real input data110A may include a first amount of real input data110, real input data110B may include a second amount of real input data110, up to and including real input data110nthat may include an nth amount of real input data110.

In some embodiments, the real input data110may include one or more characteristics included in sensitive data that may be used to train one or more generative machine learning models—e.g., sensitive data corresponding to the sensitive dataset102and/or the generative machine learning model104described and illustrated further in the present disclosure, such as, for example, with respect toFIG.1A.

In some embodiments, the real input data110may include one or more similarities, features, properties, and/or characteristics that may be included in the data used to train the analytical machine learning model108—e.g., the synthetic dataset106used to train the analytical machine learning model108.

For example, the analytical machine learning model108may have been trained using synthetic data that may include one or more characteristics included in MRI images of one or more types of knee injuries. Continuing the example, the analytical machine learning model108may analyze real input data110that may include MRI images of different knee injuries and, based on the training data, the analytical machine learning model108may be configured to analyze, diagnose, or otherwise identify the one or more knee injuries presented in the real input data110. In contrast, continuing the example, real input data110including medical images of other injuries (e.g., brain images, heart images, etc.) may not be accurately analyzed by the analytical machine learning model108based on the characteristics in the synthetic data used to train the analytical machine learning model108.

In these and other embodiments, the analytical machine learning model108may be the same as and/or analogous to the analytical machine learning model108and/or the analytical machine learning model214described and illustrated further in the present disclosure, such as, for example, with respect toFIGS.1A and2B.

In some embodiments, the analytical machine learning model108may be configured to generate output data114that may correspond to the real input data110. For example, in some embodiments, the output data114may include data and/or information that may describe one or more characteristics associated with the real input data110. Additionally or alternatively, the output data114may include data and/or information included in the real input data110. Additionally or alternatively, the output data114may include data and/or information associated with one or more determinations, inferences, conclusions, diagnoses, recommendations, etc. that may correspond to the real input data110. In some embodiments, output data114A may correspond to real input data110A, output data114B may correspond to real input data110B, up to and including output data114nthat may correspond to real input data110n.

The output data114may include data and/or information that may correspond to analysis of the real input data110by the analytical machine learning model108. In some embodiments, the output data114may include one or more characteristics corresponding to the real input data110. For example, in the context of using written or typed data, the analytical machine learning model108may be trained to analyze real input data110that may include medical notes corresponding to a patient. Continuing the example, the output data114generated using the analytical machine learning model108may include the words included in the medical notes, important categories of information in the medical notes (e.g., patient health information, doctor's potential diagnoses, abnormalities in the notes, etc.), etc.

In some embodiments, the output data114may include one or more determinations generated using the analytical machine learning model108. For example, continuing in the context of the real input data110including medical notes of the patient, the analytical machine learning model108may be configured to determine one or more diagnoses, inferences, conclusions, and/or other determinations based on the medical notes corresponding to the patient. Continuing the example, the medical notes may include dietary habits, blood pressure readings, weight, height, family medical history, etc. corresponding to the patient. Further, the output data114may include a determination that the patient may have high blood pressure, recommendations including dietary changes, recommendations for one or more blood pressure medications, etc.

As an additional example, the analytical machine learning model108may be trained to analyze medical images (e.g., MRI, x-ray, etc.) corresponding to knees, including knee injuries. Continuing the example, the analytical machine learning model108may have been trained using synthetic data corresponding to a synthetic dataset (e.g., synthetic dataset106) generated using one or more generative machine learning models (e.g., generative machine learning model104). The one or more generative machine learning models may have been trained to generate synthetic data including medical images of one or more knees based on sensitive data (e.g., sensitive data corresponding to the sensitive dataset102) from one or more data sources. Further, the analytical machine learning model108may be configured to collect information using the real input data110(e.g., one or more medical images of different knees) and to analyze the real input data110to generate output data114. Further continuing the example, the output data114may include data and/or information corresponding to the real input data110—e.g., identifying anatomical features of the knee, identifying one or more abnormalities associated with the image, etc. Additionally or alternatively, the analytical machine learning model108may be configured to include one or more determinations in the output data114—e.g., diagnosing a torn ligament, broken kneecap, inflammation, etc. The analytical machine learning model108may additionally be configured to generate one or more recommendations that may be included in the output data114—e.g., medications to help reduce inflammation, recommending various procedures to remedy abnormalities, etc.

Modifications, additions, or omissions may be made toFIG.1Aand/orFIG.1Bwithout departing from the scope of the present disclosure. For example, the amount of sensitive data used and/or synthetic data generated may vary. Further, the number and/or types of generative machine learning models and/or analytical machine learning models may vary.

FIG.2Aillustrates an example environment200for training one or more generative machine learning models206using one or more sensitive datasets204corresponding to one or more data sources202and generating a synthetic dataset208using the one or more generative machine learning models206, in accordance with one or more embodiments of the present disclosure.

The one or more data sources202may include databases, hard drives, storage devices, and/or other integrated systems that may be configured to aggregate, organize, and/or otherwise store the one or more sensitive datasets204. In some embodiments, the one or more data sources202may correspond to one or more of the sensitive datasets204. For example, in some embodiments, a first data source202A may generate, aggregate, and/or otherwise store a first sensitive dataset204A, a second data source202B may generate, aggregate, and/or otherwise store a second sensitive dataset204B, up to and including an nth data source202nthat may generate, aggregate, and/or otherwise store an nth sensitive dataset204n.

In some embodiments, the one or more sensitive datasets204may include sensitive data that may include one or more labels. In some embodiments, the sensitive data may be labeled such that the sensitive data may be used to train one or more machine learning models and/or systems (e.g., the one or more generative machine learning models206). In some embodiments, labeled sensitive data may refer to sensitive data that may be marked up, annotated, tagged, classified, processed, and/or otherwise labeled such that one or more machine learning models may be trained using the data.

For example, in the context of medical data used as the sensitive data included in one or more sensitive datasets204, the sensitive data may include one or more brain scans or images. Further, some of the images may include one or more abnormalities (e.g., bleeding, tumors, inflammation, etc.). One or more of the images may then be labeled to include annotations that may indicate when an image may show a healthy brain, a brain with abnormal bleeding, a brain with inflammation, a brain with a tumor, etc. Continuing the example, the one or more machine learning models may be trained, for example, to generate one or more images including healthy brains, brains with inflammation, brains with tumors, etc.

In some embodiments, using well-labeled sensitive data to train the one or more generative machine learning models206may improve performance of the one or more generative machine learning models206. In these and other embodiments, identifying whether the one or more generative machine learning models206may have been trained using well-labeled sensitive datasets204may be described and illustrated further in the present disclosure, such as, for example, described with respect to the filtering module210inFIG.2B.

In some embodiments, one or more entities may own and/or otherwise control the one or more data sources202and the one or more corresponding sensitive datasets204. In some embodiments, a first entity may own and/or control the first data source202A and the first sensitive dataset204A, a second entity may own and/or control the second data source202B and the second sensitive dataset204B, and an nth entity may own and/or control the nth data source202nand the nth sensitive dataset204/1.

In some embodiments, the data sources202and the corresponding sensitive datasets204may be kept separate because the data, or a portion of the data, included in the sensitive datasets204may be designated for protection. For example, in the context of the sensitive datasets204including medical data, the first entity may include a hospital where the first sensitive dataset204A corresponding to the first data source202A may include personal identifying information and therefore may be prohibited from being shared by law, at least without the consent of every patient whose data may be included in the first sensitive dataset204A. Further, the second entity may include an insurance company where the second sensitive dataset204B corresponding to the second data source202B may include medical data including personal identifying information and therefore may be prohibited from being shared by law, at least without the consent of the insured whose data may be included in the second sensitive dataset204B. Because the sensitive datasets204may not be easily shared and/or prohibited by law from being shared between the hospital and the insurance company, the first sensitive dataset204A may correspond to the first data source202A and remain separate from the second sensitive dataset204B that may correspond to the second data source202B.

The one or more generative machine learning models206may include one or more algorithms, machine learning models, systems including machine learning models, etc. that may be trained using one or more sensitive datasets (e.g., sensitive datasets204). In some embodiments, the one or more generative machine learning models206may be configured to generate synthetic data corresponding to one or more synthetic datasets208based at least on being trained using the one or more sensitive datasets204. In some embodiments, a first generative machine learning model206A may be trained using data corresponding to the first sensitive dataset204A, a second generative machine learning model206B may be trained using data corresponding to the second sensitive dataset204B, up to and including an nth generative machine learning model206nthat may be trained using data corresponding to the nth sensitive dataset204n. In these and other embodiments, the one or more generative machine learning models206may be the same as and/or analogous to other generative machine learning models described and illustrated in the present disclosure, such as, for example, generative machine learning model104described and illustrated with respect toFIG.1A.

In some embodiments, the one or more generative machine learning models206may be trained using one or more sensitive datasets204. In some embodiments, the first sensitive dataset204A, the second sensitive dataset204B, up to and including the nth sensitive dataset204nmay be used to train the one or more generative machine learning models206.

In some embodiments, the first generative machine learning model206A may include a different type of generative machine learning model206than the second generative machine learning model206B—e.g., the first generative machine learning model206A may include a GAN and the second generative machine learning model206B may include a VAE. In some embodiments, the first generative machine learning model206A and the second generative machine learning model206B may include the same type of generative machine learning model206(e.g., both may include a GAN). In some embodiments, the first generative machine learning model206A and the second generative machine learning model206B may be the same machine learning model206.

In some embodiments, the one or more generative machine learning models206may generate synthetic data based at least on the sensitive data included in the one or more sensitive datasets204used to train the multiple machine learning models206. In some embodiments, continuing in the context of a first generative machine learning model206A, a second generative machine learning model206B, and an nth generative machine learning model206, the first generative machine learning model206A may be trained using a first sensitive dataset204A and may, based on being trained using the first sensitive dataset204A, generate a first amount of synthetic data. The second generative machine learning model206B may be trained using a second sensitive dataset204B and may, based on being trained using the second sensitive dataset204B, generate a second amount of synthetic data. The nth generative machine learning model206nmay be trained using an nth sensitive dataset204nand may, based on being trained using the nth sensitive dataset204n, generate an nth amount of synthetic data. Continuing the example, the first amount of synthetic data, the second amount of synthetic data, and the nth amount of synthetic data may be included in the synthetic dataset208.

In some embodiments, the first generative machine learning model206A may be trained using data corresponding to one or more of the sensitive datasets204. In some embodiments, the first generative machine learning model206A may be trained using the first sensitive dataset204A, the second sensitive dataset204B, up to and including the nth sensitive dataset204/1.

In some embodiments, because the sensitive data included in the sensitive datasets204may not be easily shared, the one or more generative machine learning models206may be trained by one or more entities, systems, users, etc. without sharing sensitive data included in the sensitive datasets204. For example, the first entity may train the first generative machine learning model206A using sensitive data corresponding to the first sensitive dataset204A, the second entity may additionally train the second generative machine learning model206B using sensitive data corresponding to the second sensitive dataset204B, up to and including an nth entity that may train the nth generative machine learning model206using sensitive data corresponding to the nth sensitive dataset204/1.

In some embodiments, one of the one or more generative machine learning models206may be trained by one or more entities that may own and/or control the one or more respective data sources202. For example, the first entity may train the first generative machine learning model206A using sensitive data corresponding to the first sensitive dataset204A. Further the second entity may train the first generative machine learning model206A using sensitive data corresponding to the second sensitive dataset204B, up to and including the nth entity may train the nth generative machine learning model206nusing sensitive data corresponding to the nth sensitive dataset204/1.

In some embodiments, the one or more generative machine learning models206may be configured to generate synthetic data that may be included in the synthetic dataset208. In some embodiments, the synthetic dataset208may include synthetic data that may include one or more traits, characteristics, properties, data, etc. that may have been included in the sensitive data included in the one or more sensitive datasets204. In these and/or other embodiments, the synthetic dataset208may include, may be the same as, and/or may be analogous to synthetic data and/or synthetic datasets described and illustrated further in the present disclosure, such as, for example the synthetic dataset106described and illustrated with respect toFIG.1.

By way of example and not limitation, a first entity and second entity may each include a pharmaceutical or biosciences company that may be in the business of discovering useful molecular structures, pharmaceuticals, and the like. The first entity may own and/or control a first data source202A that may include a first sensitive dataset204A including data and/or information corresponding to intellectual property of the first entity (e.g., particular properties, molecular structures, properties of pharmaceuticals, etc.). The second entity may own and/or control a second data source202B that may include a second sensitive dataset204B that may include data and/or information corresponding to intellectual property of the second entity (e.g., particular properties, molecular structures, properties of pharmaceuticals, etc.). Neither the first entity nor the second entity may be willing and/or able to share the sensitive data and/or information corresponding to intellectual property corresponding to either entity. Because data and/or information may not be easily shared, the first entity may train the first generative machine learning model206A using sensitive data corresponding to the first sensitive dataset204A and the second entity may additionally train the second generative machine learning model206B using sensitive data corresponding to the second sensitive dataset204B.

Continuing the example, the first generative machine learning model206A may generate a first amount of synthetic data and the second generative machine learning model206B may generate a second amount of synthetic data. Further, the first amount of synthetic data and the second amount of synthetic data may be included in the synthetic dataset208. The synthetic dataset208that may not include any of the intellectual property of either the first entity or the second entity. However, the synthetic dataset208may be used by the first entity, the second entity, and/or others as a training dataset for one or more other machine learning models—e.g., analytical machine learning models that may be configured to further analyze other data that may be associated with molecular structures, pharmaceutical properties, etc.

In some embodiments, the synthetic dataset208may include synthetic data that may be used to train one or more other machine learning models that may be described and/or illustrated further in the present disclosure, such as, for example, the analytical machine learning model108, the analytical machine learning model214described with respect toFIGS.1A,1B, and2B.

FIG.2Billustrates an example environment250for training an analytical machine learning model214based on the synthetic dataset208ofFIG.2A, in accordance with one or more embodiments of the present disclosure. In these or other embodiments, the example environment250may be the same as and/or an extension of the environment200described and illustrated further in the present disclosure, such as, for example, with respect toFIG.2A.

In some embodiments, the environment250may include a filtering module210. In these or other embodiments, the one or more modules corresponding to the error handling system102may be implemented using hardware including one or more processors, central processing units (CPUs) graphics processing units (GPUS), data processing units (DPUs), parallel processing units (PPUs), microprocessors (e.g., to perform or control performance of one or more operations), programmable vision accelerators (PVAs)—which may include one or more direct memory access (DMA) systems and/or one or more vector or vision processing units (VPUs), field-programmable gate arrays (FPGA), application-specific integrated circuits (ASICs), accelerators (e.g., deep learning accelerators (DLAs)), and/or other processor types. In some other instances, one or more of these modules may be implemented using a combination of hardware and software. In the present disclosure, operations described as being performed by the filtering module210may include operations that the filtering module210may direct a corresponding computing system to perform. In these or other embodiments, the filtering module210may be implemented by one or more computing devices, such as that described in further detail with respect toFIGS.4A-4D,5, and/or6.

The filtering module210may be configured to perform one or more operations that may take away synthetic data, add synthetic data, and/or modify existing synthetic data included in the synthetic dataset208. Further, the filtering module210may be configured to perform one or more operations that may modify the synthetic dataset208in such a way that may result in a filtered synthetic dataset212. For example, the filtering module210may take away a portion of synthetic data included in the synthetic dataset208, the remaining synthetic data together may be designated as the filtered synthetic dataset212.

In some embodiments, the filtering module210may be configured to determine whether the synthetic data generated using the generative machine learning model206may be similar to the sensitive data used to train the generative machine learning model206. By way of example and not limitation, in instances in which the sensitive data included in the first sensitive dataset204A is well-labeled and includes enough data, the generative machine learning model206may be trained to generate synthetic data that may be similar to the sensitive data included in the first sensitive dataset204A.

In some embodiments, “similar” as used in this context may mean that one or more of the variables in the sensitive dataset204may be present in the synthetic dataset208, the frequency of the variables present in the sensitive dataset204may be present in the synthetic dataset208, correlations between variables in the sensitive dataset204may also be present in the synthetic dataset208, etc. In some embodiments, the filtering module210may be configured to determine whether the synthetic data is similar to the sensitive data used to train the generative machine learning model206. Additionally or alternatively, in response to the sensitive data and the synthetic data being sufficiently different (e.g., not satisfying a similarity threshold), the filtering module210may be configured to filter synthetic data deemed as being sufficiently different out of the synthetic dataset208.

In some embodiments, the filtering module210may be configured to determine whether the generative machine learning model206may have been trained using well-labeled real data (e.g., real data included in the one or more sensitive datasets204) by evaluating one or more characteristics of the generated synthetic data against one or more quality considerations—e.g., randomness of the synthetic data, how closely the synthetic data resembles real data, similarity with training data, etc. In some embodiments, the evaluating the synthetic data may result in one or more values identifying the quality of the labeled synthetic data included in the synthetic dataset208and, correspondingly, the quality of the sensitive data used to train the generative machine learning model—e.g., identifying how well-labeled the synthetic data may be.

In some embodiments, the quality of the synthetic data included in the synthetic dataset208may be assessed based on whether information included in the real data (e.g., data included in the one or more sensitive datasets204) is also present in the synthetic data include in the synthetic dataset208. In some embodiments, the filtering module210may be configured to determine whether one or more variables, labels, and/or categories of information present in the sensitive data included in the sensitive datasets204are also present in synthetic data included in the synthetic dataset208. For example, distributions of the one or more variables, labels, and/or categories may be assessed in both the sensitive datasets204and the synthetic dataset208to determine whether the desired variables are present. If not, or if the distributions are different beyond a predetermined threshold, it may be determined that the one or more sensitive datasets204used to train the generative machine learning model206may not have been well-labeled and that data may be filtered out of the synthetic dataset208using the filtering module210.

In some embodiments, the filtering module210may be configured to perform one or more statistical correlation analyses where the statistical correlation analyses may determine a correlation between variables in the one or more sensitive datasets204and compare that correlation to a correlation between variables in the synthetic dataset208. In some embodiments, if the correlation between variables in the sensitive dataset is different from the correlation between variables by a predetermined amount, it may be determined that the sensitive data included in the sensitive dataset204may not have been well-labeled and the filtering module210may be configured to remove the corresponding data from the synthetic dataset208.

In some embodiments, the filtering module210may be configured to analyze a combination of variables and frequency of the variables in the one or more sensitive datasets204and in the synthetic data generated therefrom. In some embodiments, if the difference between the combination of variables and frequency of the variables is above a predetermined threshold, it may be determined that the sensitive data may not have been well-labeled and the filtering module210may be configured to remove the corresponding data from the synthetic dataset208.

Other examples of determining whether the sensitive data may be well-labeled based on the synthetic data may include an inception score, Frechet Inception Distance (FID) score, and other values that may identify the quality of the synthetic data generated using the generative machine learning model206. In some embodiments, if one or more of the values identifying the quality of the synthetic data is not within a predetermined threshold, the synthetic data may be removed from the synthetic dataset208and may not be included in the filtered synthetic dataset212.

In some embodiments, the filtered synthetic dataset212may include synthetic data that may have been generated using one or more generative machine learning models206using one or more sensitive datasets204. In some embodiments, the filtered synthetic dataset212may include, be the same as, and/or analogous to synthetic datasets described and illustrated further in the present disclosure, such as, for example, with respect toFIGS.1A and2A.

In some embodiments, the filtered synthetic dataset212may be used to train the analytical machine learning model214to analyze one or more characteristics included in real input data. In these and other embodiments, the analytical machine learning model214may be the same as and/or analogous to other analytical machine learning models described and illustrated in the present disclosure, such as, for example, the analytical machine learning model108described with respect toFIGS.1A and1B.

In some embodiments, the analytical machine learning model214may be trained using the filtered synthetic dataset212. In some embodiments, the filtered synthetic dataset212may include only synthetic training data generated by the generative machine learning model206in instances in which the generative machine learning model206may have been trained by well-labeled real data. In these and other embodiments, the analytical machine learning model214trained using the filtered synthetic dataset212may be faster, more efficient, and/or more accurate at recognizing one or more characteristics in real input data than an analytical machine learning model that may not have been trained using the filtered synthetic dataset212.

Modifications, additions, or omissions may be made toFIG.2Aand/orFIG.2Bwithout departing from the scope of the present disclosure. For example, the number of data sources202and corresponding sensitive datasets that may be used to train one or more generative machine learning models206may vary. Further, in some embodiments, the filtering module210and its corresponding operations may be omitted.

FIG.3is a flow diagram showing a method300for training an analytical machine learning model using synthetic data that is used as a substitute for sensitive data, in accordance with one or more embodiments of the present disclosure. One or more operations of the method300may be performed by any suitable system, apparatus, or device such as, for example the generative machine learning model104, the analytical machine learning model108ofFIG.1, the generative machine learning model206, the filtering module210, the analytical machine learning module214ofFIG.2, the autonomous vehicle system(s) described with respect toFIGS.4A-4D, computing device(s) described with respect toFIG.5, and/or the data system(s) described with respect toFIG.6in the present disclosure.

The method300may include one or more blocks302,304,306, and308. Although illustrated with discrete blocks, the operations associated with one or more of the blocks of the method300may be divided into additional blocks, combined into fewer blocks, or eliminated, depending on the particular implementation.

In some embodiments, the method300may include block302. At block302, synthetic training data may be generated using one or more generative machine learning models. In some embodiments, the one or more generative machine learning models may be trained using real data that may be designated for protection referred to also herein as “real training data.” In some embodiments, the real training data may include sensitive data described and illustrated in the present disclosure, such as, for example, with respect toFIGS.1A and2A. In some embodiments, the real training data may be collected and measured in the real-world whereas synthetic data may be generated by one or more generative machine learning models instead of being collected and measured in the real-world. In some embodiments, real training data may include at least one of medical data, personal data, sensor data, secrets, intellectual property, sensor data, video streams, client specific data, other data that may be designated for protection. The data designated for protection may also be referred to herein as data corresponding to one or more protected data sources. In some embodiments, the synthetic data generated using the one or more machine learning models may include one or more characteristics, attributes, traits, labels, etc. that may be included in the real training data. In some embodiments, the synthetic data may be generated using one or more generative machine learning models that may perform one or more operations on real training data corresponding to one or more data sources. In these and other embodiments, one or more generative machine learning models may generate synthetic data based on being trained using real training data as described and illustrated further in the present disclosure, such as, for example, with respect toFIGS.1A and2A.

At block304, the synthetic training data may be aggregated. In these and other embodiments, the synthetic data that may be generated using one or more generative machine learning models may be aggregated in a synthetic dataset. For example, a first generative machine learning model may generate a first amount of synthetic data based on being trained using a first sensitive dataset that may include real training data designated for protection. Continuing the example, a second generative machine learning model may generate a second amount of synthetic data based on being trained using a second sensitive dataset that may include real training data designated for protection. Further, the first amount of synthetic data and the second amount of synthetic data may be aggregated and/or combined to be included in a synthetic dataset. In some embodiments, the synthetic data included in the synthetic dataset may be used to train an analytical machine learning model. In these and other embodiments, aggregating synthetic data may be described and/or illustrated further in the present disclosure, such as, for example, with respect toFIGS.1A and2A

At block306, an amount of synthetic training data may be filtered out of the synthetic dataset. In some embodiments, the synthetic training data may be filtered based at least on a determination as to whether the real training data used to train the one or more generative machine learning models is properly labeled and/or annotated. In some embodiments, the determination as to whether real training data is properly labeled may be made by evaluating one or more characteristics of the generated synthetic data against one or more quality considerations. For example, quality considerations may include randomness of the synthetic data, how closely the synthetic data resembles real training data, similarity of the synthetic data with training data, etc. In some embodiments, synthetic training data may be filtered out based on whether a first distribution corresponding to the synthetic training data is different by more than a threshold compared to a second distribution corresponding to the real training data. In some embodiments, the threshold may be determined using one or more heuristic analyses. In these and other embodiments, filtering out an amount of synthetic data from the synthetic dataset may be described and illustrated further in the present disclosure, such as, for example, with respect toFIG.2B.

At block308, an analytical machine learning model may be trained. In some embodiments, the analytical machine learning model may be trained using the synthetic training data that may be generated using one or more generative machine learning models. In some embodiments, the analytical machine learning model may be trained to analyze one or more characteristics corresponding to real input data. In some embodiments, the real input data may include one or more similar characteristics, traits, features, etc. that may be included and/or present in the real training data designated for protection used to train the one or more generative machine learning models. In these and other embodiments, training the analytical machine learning model using synthetic data generated using the generative machine learning model may be described and illustrated further in the present disclosure, such as, for example, with respect toFIGS.1B and2B.

Modifications, additions, or omissions may be made to one or more operations included in the method300without departing from the scope of the present disclosure. For example, the operations of method300may be implemented in differing order. Additionally or alternatively, two or more operations may be performed at the same time. Furthermore, the outlined operations and actions are only provided as examples, and some of the operations and actions may be optional, combined into fewer operations and actions, or expanded into additional operations and actions without detracting from the essence of the described embodiments.

Example Autonomous Vehicle

The vehicle400may include components such as a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. The vehicle400may include a propulsion system450, such as an internal combustion engine, hybrid electric power plant, an all-electric engine, and/or another propulsion system type. The propulsion system450may be connected to a drive train of the vehicle400, which may include a transmission, to enable the propulsion of the vehicle400. The propulsion system450may be controlled in response to receiving signals from the throttle/accelerator452.

A steering system454, which may include a steering wheel, may be used to steer the vehicle400(e.g., along a desired path or route) when the propulsion system450is operating (e.g., when the vehicle is in motion). The steering system454may receive signals from a steering actuator456. The steering wheel may be optional for full automation (Level 5) functionality.

The brake sensor system446may be used to operate the vehicle brakes in response to receiving signals from the brake actuators448and/or brake sensors.

Controller(s)436, which may include one or more CPU(s), system on chips (SoCs)404(FIG.4C) and/or GPU(s), may provide signals (e.g., representative of commands) to one or more components and/or systems of the vehicle400. For example, the controller(s) may send signals to operate the vehicle brakes via one or more brake actuators448, to operate the steering system454via one or more steering actuators456, and/or to operate the propulsion system450via one or more throttle/accelerators452. The controller(s)436may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals, and output operation commands (e.g., signals representing commands) to enable autonomous driving and/or to assist a human driver in driving the vehicle400. The controller(s)436may include a first controller436for autonomous driving functions, a second controller436for functional safety functions, a third controller436for artificial intelligence functionality (e.g., computer vision), a fourth controller436for infotainment functionality, a fifth controller436for redundancy in emergency conditions, and/or other controllers. In some examples, a single controller436may handle two or more of the above functionalities, two or more controllers436may handle a single functionality, and/or any combination thereof.

The controller(s)436may provide the signals for controlling one or more components and/or systems of the vehicle400in response to sensor data received from one or more sensors (e.g., sensor inputs). The sensor data may be received from, for example and without limitation, global navigation satellite systems sensor(s)458(e.g., Global Positioning System sensor(s)), RADAR sensor(s)460, ultrasonic sensor(s)462, LIDAR sensor(s)464, inertial measurement unit (IMU) sensor(s)466(e.g., accelerometer(s), gyroscope(s), magnetic compass(es), magnetometer(s), etc.), microphone(s)496, stereo camera(s)468, wide-view camera(s)470(e.g., fisheye cameras), infrared camera(s)472, surround camera(s)474(e.g., 360 degree cameras), long-range and/or mid-range camera(s)498, speed sensor(s)444(e.g., for measuring the speed of the vehicle400), vibration sensor(s)442, steering sensor(s)440, brake sensor(s)446(e.g., as part of the brake sensor system446), and/or other sensor types.

One or more of the controller(s)436may receive inputs (e.g., represented by input data) from an instrument cluster432of the vehicle400and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display434, an audible annunciator, a loudspeaker, and/or via other components of the vehicle400. The outputs may include information such as vehicle velocity, speed, time, map data (e.g., the HD map422ofFIG.4C), location data (e.g., the location of the vehicle400, such as on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and status of objects as perceived by the controller(s)436, etc. For example, the HMI display434may display information about the presence of one or more objects (e.g., a street sign, caution sign, traffic light changing, etc.), and/or information about driving maneuvers the vehicle has made, is making, or will make (e.g., changing lanes now, taking exit34B in two miles, etc.).

The vehicle400further includes a network interface424, which may use one or more wireless antenna(s)426and/or modem(s) to communicate over one or more networks. For example, the network interface424may be capable of communication over LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The wireless antenna(s)426may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.), using local area network(s), such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc., and/or low power wide-area network(s) (LPWANs), such as LoRaWAN, SigFox, etc.

FIG.4Bis an example of camera locations and fields of view for the example autonomous vehicle400ofFIG.4A, in accordance with some embodiments of the present disclosure. The cameras and respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and/or alternative cameras may be included and/or the cameras may be located at different locations on the vehicle400.

A variety of cameras may be used in a front-facing configuration, including, for example, a monocular camera platform that includes a CMOS (complementary metal oxide semiconductor) color imager. Another example may be a wide-view camera(s)470that may be used to perceive objects coming into view from the periphery (e.g., pedestrians, crossing traffic or bicycles). Although only one wide-view camera is illustrated inFIG.4B, there may any number of wide-view cameras470on the vehicle400. In addition, long-range camera(s)498(e.g., a long-view stereo camera pair) may be used for depth-based object detection, especially for objects for which a neural network has not yet been trained. The long-range camera(s)498may also be used for object detection and classification, as well as basic object tracking.

One or more stereo cameras468may also be included in a front-facing configuration. The stereo camera(s)468may include an integrated control unit comprising a scalable processing unit, which may provide a programmable logic (FPGA) and a multi-core micro-processor with an integrated CAN or Ethernet interface on a single chip. Such a unit may be used to generate a 3-D map of the vehicle's environment, including a distance estimate for all the points in the image. An alternative stereo camera(s)468may include a compact stereo vision sensor(s) that may include two camera lenses (one each on the left and right) and an image processing chip that may measure the distance from the vehicle to the target object and use the generated information (e.g., metadata) to activate the autonomous emergency braking and lane departure warning functions. Other types of stereo camera(s)468may be used in addition to, or alternatively from, those described herein.

Cameras with a field of view that include portions of the environment to the side of the vehicle400(e.g., side-view cameras) may be used for surround view, providing information used to create and update the occupancy grid, as well as to generate side impact collision warnings. For example, surround camera(s)474(e.g., four surround cameras474as illustrated inFIG.4B) may be positioned to on the vehicle400. The surround camera(s)474may include wide-view camera(s)470, fisheye camera(s), 360-degree camera(s), and/or the like. For example, four fisheye cameras may be positioned on the vehicle's front, rear, and sides. In an alternative arrangement, the vehicle may use three surround camera(s)474(e.g., left, right, and rear), and may leverage one or more other camera(s) (e.g., a forward-facing camera) as a fourth surround-view camera.

Cameras with a field of view that include portions of the environment to the rear of the vehicle400(e.g., rear-view cameras) may be used for park assistance, surround view, rear collision warnings, and creating and updating the occupancy grid. A wide variety of cameras may be used including, but not limited to, cameras that are also suitable as a front-facing camera(s) (e.g., long-range and/or mid-range camera(s)498, stereo camera(s)468), infrared camera(s)472, etc.), as described herein.

Each of the components, features, and systems of the vehicle400inFIG.4Cis illustrated as being connected via bus402. The bus402may include a Controller Area Network (CAN) data interface (alternatively referred to herein as a “CAN bus”). A CAN may be a network inside the vehicle400used to aid in control of various features and functionality of the vehicle400, such as actuation of brakes, acceleration, braking, steering, windshield wipers, etc. A CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus may be read to find steering wheel angle, ground speed, engine revolutions per minute (RPMs), button positions, and/or other vehicle status indicators. The CAN bus may be ASIL B compliant.

Although the bus402is described herein as being a CAN bus, this is not intended to be limiting. For example, in addition to, or alternatively from, the CAN bus, FlexRay and/or Ethernet may be used. Additionally, although a single line is used to represent the bus402, this is not intended to be limiting. For example, there may be any number of busses402, which may include one or more CAN busses, one or more FlexRay busses, one or more Ethernet busses, and/or one or more other types of busses using a different protocol. In some examples, two or more busses402may be used to perform different functions, and/or may be used for redundancy. For example, a first bus402may be used for collision avoidance functionality and a second bus402may be used for actuation control. In any example, each bus402may communicate with any of the components of the vehicle400, and two or more busses402may communicate with the same components. In some examples, each SoC404, each controller436, and/or each computer within the vehicle may have access to the same input data (e.g., inputs from sensors of the vehicle400), and may be connected to a common bus, such the CAN bus.

The vehicle400may include one or more controller(s)436, such as those described herein with respect toFIG.4A. The controller(s)436may be used for a variety of functions. The controller(s)436may be coupled to any of the various other components and systems of the vehicle400and may be used for control of the vehicle400, artificial intelligence of the vehicle400, infotainment for the vehicle400, and/or the like.

The vehicle400may include a system(s) on a chip (SoC)404. The SoC404may include CPU(s)406, GPU(s)408, processor(s)410, cache(s)412, accelerator(s)414, data store(s)416, and/or other components and features not illustrated. The SoC(s)404may be used to control the vehicle400in a variety of platforms and systems. For example, the SoC(s)404may be combined in a system (e.g., the system of the vehicle400) with an HD map422which may obtain map refreshes and/or updates via a network interface424from one or more servers (e.g., server(s)478ofFIG.4D).

The CPU(s)406may include a CPU cluster or CPU complex (alternatively referred to herein as a “CCPLEX”). The CPU(s)406may include multiple cores and/or L2 caches. For example, in some embodiments, the CPU(s)406may include eight cores in a coherent multi-processor configuration. In some embodiments, the CPU(s)406may include four dual-core clusters where each cluster has a dedicated L2 cache (e.g., a 2 MB L2 cache). The CPU(s)406(e.g., the CCPLEX) may be configured to support simultaneous cluster operation enabling any combination of the clusters of the CPU(s)406to be active at any given time.

The GPU(s)408may include an integrated GPU (alternatively referred to herein as an “iGPU”). The GPU(s)408may be programmable and may be efficient for parallel workloads. The GPU(s)408, in some examples, may use an enhanced tensor instruction set. The GPU(s)408may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96 KB storage capacity), and two or more of the streaming microprocessors may share an L2 cache (e.g., an L2 cache with a 512 KB storage capacity). In some embodiments, the GPU(s)408may include at least eight streaming microprocessors. The GPU(s)408may use compute application programming interface(s) (API(s)). In addition, the GPU(s)408may use one or more parallel computing platforms and/or programming models (e.g., NVIDIA's CUDA).

The GPU(s)408may include unified memory technology including access counters to allow for more accurate migration of memory pages to the processor that accesses them most frequently, thereby improving efficiency for memory ranges shared between processors. In some examples, address translation services (ATS) support may be used to allow the GPU(s)408to access the CPU(s)406page tables directly. In such examples, when the GPU(s)408memory management unit (MMU) experiences a miss, an address translation request may be transmitted to the CPU(s)406. In response, the CPU(s)406may look in its page tables for the virtual-to-physical mapping for the address and transmits the translation back to the GPU(s)408. As such, unified memory technology may allow a single unified virtual address space for memory of both the CPU(s)406and the GPU(s)408, thereby simplifying the GPU(s)408programming and porting of applications to the GPU(s)408.

In addition, the GPU(s)408may include an access counter that may keep track of the frequency of access of the GPU(s)408to memory of other processors. The access counter may help ensure that memory pages are moved to the physical memory of the processor that is accessing the pages most frequently.

The SoC(s)404may include any number of cache(s)412, including those described herein. For example, the cache(s)412may include an L3 cache that is available to both the CPU(s)406and the GPU(s)408(e.g., that is connected to both the CPU(s)406and the GPU(s)408). The cache(s)412may include a write-back cache that may keep track of states of lines, such as by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4 MB or more, depending on the embodiment, although smaller cache sizes may be used.

The SoC(s)404may include an arithmetic logic unit(s)(ALU(s)) which may be leveraged in performing processing with respect to any of the variety of tasks or operations of the vehicle400—such as processing DNNs. In addition, the SoC(s)404may include a floating point unit(s) (FPU(s))—or other math coprocessor or numeric coprocessor types—for performing mathematical operations within the system. For example, the SoC(s)104may include one or more FPUs integrated as execution units within a CPU(s)406and/or GPU(s)408.

The SoC(s)404may include one or more accelerators414(e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC(s)404may include a hardware acceleration cluster that may include optimized hardware accelerators and/or large on-chip memory. The large on-chip memory (e.g., 4 MB of SRAM), may enable the hardware acceleration cluster to accelerate neural networks and other calculations. The hardware acceleration cluster may be used to complement the GPU(s)408and to off-load some of the tasks of the GPU(s)408(e.g., to free up more cycles of the GPU(s)408for performing other tasks). As an example, the accelerator(s)414may be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be amenable to acceleration. The term “CNN,” as used herein, may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and Fast RCNNs (e.g., as used for object detection).

The DLA(s) may perform any function of the GPU(s)408, and by using an inference accelerator, for example, a designer may target either the DLA(s) or the GPU(s)408for any function. For example, the designer may focus processing of CNNs and floating point operations on the DLA(s) and leave other functions to the GPU(s)408and/or other accelerator(s)414.

The SoC(s)404may include data store(s)416(e.g., memory). The data store(s)416may be on-chip memory of the SoC(s)404, which may store neural networks to be executed on the GPU and/or the DLA. In some examples, the data store(s)416may be large enough in capacity to store multiple instances of neural networks for redundancy and safety. The data store(s)416may comprise L2 or L3 cache(s)412. Reference to the data store(s)416may include reference to the memory associated with the PVA, DLA, and/or other accelerator(s)414, as described herein.

The SoC(s)404may include one or more processor(s)410(e.g., embedded processors). The processor(s)410may include a boot and power management processor that may be a dedicated processor and subsystem to handle boot power and management functions and related security enforcement. The boot and power management processor may be a part of the SoC(s)404boot sequence and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, assistance in system low power state transitions, management of SoC(s)404thermals and temperature sensors, and/or management of the SoC(s)404power states. Each temperature sensor may be implemented as a ring-oscillator whose output frequency is proportional to temperature, and the SoC(s)404may use the ring-oscillators to detect temperatures of the CPU(s)406, GPU(s)408, and/or accelerator(s)414. If temperatures are determined to exceed a threshold, the boot and power management processor may enter a temperature fault routine and put the SoC(s)404into a lower power state and/or put the vehicle400into a chauffeur to safe-stop mode (e.g., bring the vehicle400to a safe stop).

The processor(s)410may further include an always-on processor engine that may provide necessary hardware features to support low power sensor management and wake use cases. The always-on processor engine may include a processor core, a tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I/O controller peripherals, and routing logic.

The processor(s)410may further include a real-time camera engine that may include a dedicated processor subsystem for handling real-time camera management.

The processor(s)410may further include a high dynamic range signal processor that may include an image signal processor that is a hardware engine that is part of the camera processing pipeline.

The video image compositor may also be configured to perform stereo rectification on input stereo lens frames. The video image compositor may further be used for user interface composition when the operating system desktop is in use, and the GPU(s)408is not required to continuously render new surfaces. Even when the GPU(s)408is powered on and active doing 3D rendering, the video image compositor may be used to offload the GPU(s)408to improve performance and responsiveness.

The SoC(s)404may further include a broad range of peripheral interfaces to enable communication with peripherals, audio codecs, power management, and/or other devices. The SoC(s)404may be used to process data from cameras (e.g., connected over Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LIDAR sensor(s)464, RADAR sensor(s)460, etc. that may be connected over Ethernet), data from bus402(e.g., speed of vehicle400, steering wheel position, etc.), data from GNSS sensor(s)458(e.g., connected over Ethernet or CAN bus). The SoC(s)404may further include dedicated high-performance mass storage controllers that may include their own DMA engines, and that may be used to free the CPU(s)406from routine data management tasks.

The SoC(s)404may be an end-to-end platform with a flexible architecture that spans automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and makes efficient use of computer vision and ADAS techniques for diversity and redundancy, provides a platform for a flexible, reliable driving software stack, along with deep learning tools. The SoC(s)404may be faster, more reliable, and even more energy-efficient and space-efficient than conventional systems. For example, the accelerator(s)414, when combined with the CPU(s)406, the GPU(s)408, and the data store(s)416, may provide for a fast, efficient platform for level 3-5 autonomous vehicles.

In some examples, a CNN for facial recognition and vehicle owner identification may use data from camera sensors to identify the presence of an authorized driver and/or owner of the vehicle400. The always-on sensor processing engine may be used to unlock the vehicle when the owner approaches the driver door and turn on the lights, and, in security mode, to disable the vehicle when the owner leaves the vehicle. In this way, the SoC(s)404provide for security against theft and/or carjacking.

The vehicle may include a CPU(s)418(e.g., discrete CPU(s), or dCPU(s)), that may be coupled to the SoC(s)404via a high-speed interconnect (e.g., PCIe). The CPU(s)418may include an X86 processor, for example. The CPU(s)418may be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and the SoC(s)404, and/or monitoring the status and health of the controller(s)436and/or infotainment SoC430, for example.

The vehicle400may include a GPU(s)420(e.g., discrete GPU(s), or dGPU(s)), that may be coupled to the SoC(s)404via a high-speed interconnect (e.g., NVIDIA's NVLINK). The GPU(s)420may provide additional artificial intelligence functionality, such as by executing redundant and/or different neural networks, and may be used to train and/or update neural networks based on input (e.g., sensor data) from sensors of the vehicle400.

The vehicle400may further include the network interface424which may include one or more wireless antennas426(e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface424may be used to enable wireless connectivity over the Internet with the cloud (e.g., with the server(s)478and/or other network devices), with other vehicles, and/or with computing devices (e.g., client devices of passengers). To communicate with other vehicles, a direct link may be established between the two vehicles and/or an indirect link may be established (e.g., across networks and over the Internet). Direct links may be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link may provide the vehicle400information about vehicles in proximity to the vehicle400(e.g., vehicles in front of, on the side of, and/or behind the vehicle400). This functionality may be part of a cooperative adaptive cruise control functionality of the vehicle400.

The vehicle400may further include data store(s)428, which may include off-chip (e.g., off the SoC(s)404) storage. The data store(s)428may include one or more storage elements including RAM, SRAM, DRAM, VRAM, Flash, hard disks, and/or other components and/or devices that may store at least one bit of data.

The vehicle400may further include GNSS sensor(s)458. The GNSS sensor(s)458(e.g., GPS, assisted GPS sensors, differential GPD (DGPS) sensors, etc.), to assist in mapping, perception, occupancy grid generation, and/or path planning functions. Any number of GNSS sensor(s)458may be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet to Serial (RS-232) bridge.

The vehicle400may further include RADAR sensor(s)460. The RADAR sensor(s)460may be used by the vehicle400for long-range vehicle detection, even in darkness and/or severe weather conditions. RADAR functional safety levels may be ASIL B. The RADAR sensor(s)460may use the CAN and/or the bus402(e.g., to transmit data generated by the RADAR sensor(s)460) for control and to access object tracking data, with access to Ethernet to access raw data, in some examples. A wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor(s)460may be suitable for front, rear, and side RADAR use. In some example, Pulse Doppler RADAR sensor(s) are used.

The vehicle400may further include ultrasonic sensor(s)462. The ultrasonic sensor(s)462, which may be positioned at the front, back, and/or the sides of the vehicle400, may be used for park assist and/or to create and update an occupancy grid. A wide variety of ultrasonic sensor(s)462may be used, and different ultrasonic sensor(s)462may be used for different ranges of detection (e.g., 2.5 m, 4 m). The ultrasonic sensor(s)462may operate at functional safety levels of ASIL B.

The vehicle400may include LIDAR sensor(s)464. The LIDAR sensor(s)464may be used for object and pedestrian detection, emergency braking, collision avoidance, and/or other functions. The LIDAR sensor(s)464may be functional safety level ASIL B. In some examples, the vehicle400may include multiple LIDAR sensors464(e.g., two, four, six, etc.) that may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

The vehicle may further include IMU sensor(s)466. The IMU sensor(s)466may be located at a center of the rear axle of the vehicle400, in some examples. The IMU sensor(s)466may include, for example and without limitation, an accelerometer(s), a magnetometer(s), a gyroscope(s), a magnetic compass(es), and/or other sensor types. In some examples, such as in six-axis applications, the IMU sensor(s)466may include accelerometers and gyroscopes, while in nine-axis applications, the IMU sensor(s)466may include accelerometers, gyroscopes, and magnetometers.

In some embodiments, the IMU sensor(s)466may be implemented as a miniature, high-performance GPS-Aided Inertial Navigation System (GPS/INS) that combines micro-electro-mechanical systems (MEMS) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor(s)466may enable the vehicle400to estimate heading without requiring input from a magnetic sensor by directly observing and correlating the changes in velocity from GPS to the IMU sensor(s)466. In some examples, the IMU sensor(s)466and the GNSS sensor(s)458may be combined in a single integrated unit.

The vehicle may include microphone(s)496placed in and/or around the vehicle400. The microphone(s)496may be used for emergency vehicle detection and identification, among other things.

The vehicle may further include any number of camera types, including stereo camera(s)468, wide-view camera(s)470, infrared camera(s)472, surround camera(s)474, long-range and/or mid-range camera(s)498, and/or other camera types. The cameras may be used to capture image data around an entire periphery of the vehicle400. The types of cameras used depends on the embodiments and requirements for the vehicle400, and any combination of camera types may be used to provide the necessary coverage around the vehicle400. In addition, the number of cameras may differ depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and/or another number of cameras. The cameras may support, as an example and without limitation, Gigabit Multimedia Serial Link (GMSL) and/or Gigabit Ethernet. Each of the camera(s) is described with more detail herein with respect toFIG.4AandFIG.4B.

The vehicle400may further include vibration sensor(s)442. The vibration sensor(s)442may measure vibrations of components of the vehicle, such as the axle(s). For example, changes in vibrations may indicate a change in road surfaces. In another example, when two or more vibration sensors442are used, the differences between the vibrations may be used to determine friction or slippage of the road surface (e.g., when the difference in vibration is between a power-driven axle and a freely rotating axle).

The vehicle400may include an ADAS system438. The ADAS system438may include a SoC, in some examples. The ADAS system438may include autonomous/adaptive/automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warnings (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning systems (CWS), lane centering (LC), and/or other features and functionality.

The ACC systems may use RADAR sensor(s)460, LIDAR sensor(s)464, and/or a camera(s). The ACC systems may include longitudinal ACC and/or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of the vehicle400and automatically adjust the vehicle speed to maintain a safe distance from vehicles ahead. Lateral ACC performs distance keeping, and advises the vehicle400to change lanes when necessary. Lateral ACC is related to other ADAS applications such as LCA and CWS.

LKA systems are a variation of LDW systems. LKA systems provide steering input or braking to correct the vehicle400if the vehicle400starts to exit the lane. BSW systems detects and warn the driver of vehicles in an automobile's blind spot. BSW systems may provide a visual, audible, and/or tactile alert to indicate that merging or changing lanes is unsafe. The system may provide an additional warning when the driver uses a turn signal. BSW systems may use rear-side facing camera(s) and/or RADAR sensor(s).

Conventional ADAS systems may be prone to false positive results, which may be annoying and distracting to a driver, but typically are not catastrophic, because the ADAS systems alert the driver and allow the driver to decide whether a safety condition truly exists and act accordingly. However, in an autonomous vehicle400, the vehicle400itself must, in the case of conflicting results, decide whether to heed the result from a primary computer or a secondary computer (e.g., a first controller436or a second controller436). For example, in some embodiments, the ADAS system438may be a backup and/or secondary computer for providing perception information to a backup computer rationality module. The backup computer rationality monitor may run a redundant diverse software on hardware components to detect faults in perception and dynamic driving tasks. Outputs from the ADAS system438may be provided to a supervisory MCU. If outputs from the primary computer and the secondary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

The vehicle400may further include the infotainment SoC430(e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be a SoC, and may include two or more discrete components. The infotainment SoC430may include a combination of hardware and software that may be used to provide audio (e.g., music, a personal digital assistant, navigational instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and/or information services (e.g., navigation systems, rear-parking assistance, a radio data system, vehicle-related information such as fuel level, total distance covered, brake fuel level, oil level, door open/close, air filter information, etc.) to the vehicle400. For example, the infotainment SoC430may include radios, disk players, navigation systems, video players, USB and Bluetooth connectivity, carputers, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free voice control, a heads-up display (HUD), an HMI display434, a telematics device, a control panel (e.g., for controlling and/or interacting with various components, features, and/or systems), and/or other components. The infotainment SoC430may further be used to provide information (e.g., visual and/or audible) to a user(s) of the vehicle, such as information from the ADAS system438, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and/or other information.

The infotainment SoC430may include GPU functionality. The infotainment SoC430may communicate over the bus402(e.g., CAN bus, Ethernet, etc.) with other devices, systems, and/or components of the vehicle400. In some examples, the infotainment SoC430may be coupled to a supervisory MCU such that the GPU of the infotainment system may perform some self-driving functions in the event that the primary controller(s)436(e.g., the primary and/or backup computers of the vehicle400) fail. In such an example, the infotainment SoC430may put the vehicle400into a chauffeur to safe-stop mode, as described herein.

The vehicle400may further include an instrument cluster432(e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster432may include a controller and/or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster432may include a set of instrumentation such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicators, gearshift position indicator, seat belt warning light(s), parking-brake warning light(s), engine-malfunction light(s), airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and/or shared among the infotainment SoC430and the instrument cluster432. In other words, the instrument cluster432may be included as part of the infotainment SoC430, or vice versa.

FIG.4Dis a system diagram for communication between cloud-based server(s) and the example autonomous vehicle400ofFIG.4A, in accordance with some embodiments of the present disclosure. The system476may include server(s)478, network(s)490, and vehicles, including the vehicle400. The server(s)478may include a plurality of GPUs484(A)-484(H) (collectively referred to herein as GPUs484), PCIe switches482(A)-482(H) (collectively referred to herein as PCIe switches482), and/or CPUs480(A)-480(B) (collectively referred to herein as CPUs480). The GPUs484, the CPUs480, and the PCIe switches may be interconnected with high-speed interconnects such as, for example and without limitation, NVLink interfaces488developed by NVIDIA and/or PCIe connections486. In some examples, the GPUs484are connected via NVLink and/or NVSwitch SoC and the GPUs484and the PCIe switches482are connected via PCIe interconnects. Although eight GPUs484, two CPUs480, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the server(s)478may include any number of GPUs484, CPUs480, and/or PCIe switches. For example, the server(s)478may each include eight, sixteen, thirty-two, and/or more GPUs484.

The server(s)478may receive, over the network(s)490and from the vehicles, image data representative of images showing unexpected or changed road conditions, such as recently commenced road work. The server(s)478may transmit, over the network(s)490and to the vehicles, neural networks492, updated neural networks492, and/or map information494, including information regarding traffic and road conditions. The updates to the map information494may include updates for the HD map422, such as information regarding construction sites, potholes, detours, flooding, and/or other obstructions. In some examples, the neural networks492, the updated neural networks492, and/or the map information494may have resulted from new training and/or experiences represented in data received from any number of vehicles in the environment, and/or based on training performed at a datacenter (e.g., using the server(s)478and/or other servers).

In some examples, the server(s)478may receive data from the vehicles and apply the data to up-to-date real-time neural networks for real-time intelligent inferencing. The server(s)478may include deep-learning supercomputers and/or dedicated AI computers powered by GPU(s)484, such as a DGX and DGX Station machines developed by NVIDIA. However, in some examples, the server(s)478may include deep learning infrastructure that use only CPU-powered datacenters.

The deep-learning infrastructure of the server(s)478may be capable of fast, real-time inferencing, and may use that capability to evaluate and verify the health of the processors, software, and/or associated hardware in the vehicle400. For example, the deep-learning infrastructure may receive periodic updates from the vehicle400, such as a sequence of images and/or objects that the vehicle400has located in that sequence of images (e.g., via computer vision and/or other machine learning object classification techniques). The deep-learning infrastructure may run its own neural network to identify the objects and compare them with the objects identified by the vehicle400and, if the results do not match and the infrastructure concludes that the AI in the vehicle400is malfunctioning, the server(s)478may transmit a signal to the vehicle400instructing a fail-safe computer of the vehicle400to assume control, notify the passengers, and complete a safe parking maneuver.

For inferencing, the server(s)478may include the GPU(s)484and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-powered servers and inference acceleration may make real-time responsiveness possible. In other examples, such as where performance is less critical, servers powered by CPUs, FPGAs, and other processors may be used for inferencing.

Example Computing Device

FIG.5is a block diagram of an example computing device(s)500suitable for use in implementing some embodiments of the present disclosure. Computing device500may include an interconnect system502that directly or indirectly couples the following devices: memory504, one or more central processing units (CPUs)506, one or more graphics processing units (GPUs)508, a communication interface510, input/output (I/O) ports512, input/output components514, a power supply516, one or more presentation components518(e.g., display(s)), and one or more logic units520. In at least one embodiment, the computing device(s)500may comprise one or more virtual machines (VMs), and/or any of the components thereof may comprise virtual components (e.g., virtual hardware components). For non-limiting examples, one or more of the GPUs508may comprise one or more vGPUs, one or more of the CPUs506may comprise one or more vCPUs, and/or one or more of the logic units520may comprise one or more virtual logic units. As such, a computing device(s)500may include discrete components (e.g., a full GPU dedicated to the computing device500), virtual components (e.g., a portion of a GPU dedicated to the computing device500), or a combination thereof.

Although the various blocks ofFIG.5are shown as connected via the interconnect system502with lines, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component518, such as a display device, may be considered an I/O component514(e.g., if the display is a touch screen). As another example, the CPUs506and/or GPUs508may include memory (e.g., the memory504may be representative of a storage device in addition to the memory of the GPUs508, the CPUs506, and/or other components). In other words, the computing device ofFIG.5is merely illustrative. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “hand-held device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and/or other device or system types, as all are contemplated within the scope of the computing device ofFIG.5.

The interconnect system502may represent one or more links or busses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system502may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standards association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and/or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU506may be directly connected to the memory504. Further, the CPU506may be directly connected to the GPU508. Where there is direct, or point-to-point, connection between components, the interconnect system502may include a PCIe link to carry out the connection. In these examples, a PCI bus need not be included in the computing device500.

The CPU(s)506may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device500to perform one or more of the methods and/or processes described herein. The CPU(s)506may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) that are capable of handling a multitude of software threads simultaneously. The CPU(s)506may include any type of processor, and may include different types of processors depending on the type of computing device500implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device500, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device500may include one or more CPUs506in addition to one or more microprocessors or supplementary co-processors, such as math co-processors.

In addition to or alternatively from the CPU(s)506, the GPU(s)508may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device500to perform one or more of the methods and/or processes described herein. One or more of the GPU(s)508may be an integrated GPU (e.g., with one or more of the CPU(s)506and/or one or more of the GPU(s)508may be a discrete GPU. In embodiments, one or more of the GPU(s)508may be a coprocessor of one or more of the CPU(s)506. The GPU(s)508may be used by the computing device500to render graphics (e.g., 3D graphics) or perform general purpose computations. For example, the GPU(s)508may be used for General-Purpose computing on GPUs (GPGPU). The GPU(s)508may include hundreds or thousands of cores that are capable of handling hundreds or thousands of software threads simultaneously. The GPU(s)508may generate pixel data for output images in response to rendering commands (e.g., rendering commands from the CPU(s)506received via a host interface). The GPU(s)508may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory may be included as part of the memory504. The GPU(s)508may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined together, each GPU508may generate pixel data or GPGPU data for different portions of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory, or may share memory with other GPUs.

In addition to or alternatively from the CPU(s)506and/or the GPU(s)508, the logic unit(s)520may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device500to perform one or more of the methods and/or processes described herein. In embodiments, the CPU(s)506, the GPU(s)508, and/or the logic unit(s)520may discretely or jointly perform any combination of the methods, processes and/or portions thereof. One or more of the logic units520may be part of and/or integrated in one or more of the CPU(s)506and/or the GPU(s)508and/or one or more of the logic units520may be discrete components or otherwise external to the CPU(s)506and/or the GPU(s)508. In embodiments, one or more of the logic units520may be a coprocessor of one or more of the CPU(s)506and/or one or more of the GPU(s)508.

The communication interface510may include one or more receivers, transmitters, and/or transceivers that enable the computing device500to communicate with other computing devices via an electronic communication network, include wired and/or wireless communications. The communication interface510may include components and functionality to enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating over Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and/or the Internet. In one or more embodiments, logic unit(s)520and/or communication interface510may include one or more data processing units (DPUs) to transmit data received over a network and/or through interconnect system502directly to (e.g., a memory of) one or more GPU(s)508.

The I/O ports512may enable the computing device500to be logically coupled to other devices including the I/O components514, the presentation component(s)518, and/or other components, some of which may be built in to (e.g., integrated in) the computing device500. Illustrative I/O components514include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I/O components514may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs may be transmitted to an appropriate network element for further processing. An NUI may implement any combination of speech recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail in the present disclosure) associated with a display of the computing device500. The computing device500may include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations of these, for gesture detection and recognition. Additionally, the computing device500may include accelerometers or gyroscopes (e.g., as part of an inertia measurement unit (IMU)) that enable detection of motion. In some examples, the output of the accelerometers or gyroscopes may be used by the computing device500to render immersive augmented reality or virtual reality.

The power supply516may include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply516may provide power to the computing device500to enable the components of the computing device500to operate.

The presentation component(s)518may include a display (e.g., a monitor, a touch screen, a television screen, a heads-up-display (HUD), other display types, or a combination thereof), speakers, and/or other presentation components. The presentation component(s)518may receive data from other components (e.g., the GPU(s)508, the CPU(s)506, etc.), and output the data (e.g., as an image, video, sound, etc.).

Example Data Center

FIG.6illustrates an example data center600that may be used in at least one embodiments of the present disclosure. The data center600may include a data center infrastructure layer610, a framework layer620, a software layer630, and/or an application layer640.

As shown inFIG.6, the data center infrastructure layer610may include a resource orchestrator612, grouped computing resources614, and node computing resources (“node C.R.s”)616(1)-616(N), where “N” represents any whole, positive integer. In at least one embodiment, node C.R.s616(1)-616(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input/output (NW I/O) devices, network switches, virtual machines (VMs), power modules, and/or cooling modules, etc. In some embodiments, one or more node C.R.s from among node C.R.s616(1)-616(N) may correspond to a server having one or more of the above-mentioned computing resources. In addition, in some embodiments, the node C.R.s616(1)-616(N) may include one or more virtual components, such as vGPUs, vCPUs, and/or the like, and/or one or more of the node C.R.s616(1)-616(N) may correspond to a virtual machine (VM).

In at least one embodiment, grouped computing resources614may include separate groupings of node C.R.s616housed within one or more racks (not shown), or many racks housed in data centers at various geographical locations (also not shown). Separate groupings of node C.R.s616within grouped computing resources614may include grouped compute, network, memory or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R.s616including CPUs, GPUs, DPUs, and/or other processors may be grouped within one or more racks to provide compute resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and/or network switches, in any combination.

The resource orchestrator612may configure or otherwise control one or more node C.R.s616(1)-616(N) and/or grouped computing resources614. In at least one embodiment, resource orchestrator612may include a software design infrastructure (SDI) management entity for the data center600. The resource orchestrator612may include hardware, software, or some combination thereof.

In at least one embodiment, as shown inFIG.6, framework layer620may include a job scheduler632, a configuration manager634, a resource manager636, and/or a distributed file system638. The framework layer620may include a framework to support software632of software layer630and/or one or more application(s)642of application layer640. The software632or application(s)642may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud and Microsoft Azure. The framework layer620may be, but is not limited to, a type of free and open-source software web application framework such as Apache Spark™ (hereinafter “Spark”) that may utilize distributed file system638for large-scale data processing (e.g., “big data”). In at least one embodiment, job scheduler632may include a Spark driver to facilitate scheduling of workloads supported by various layers of data center600. The configuration manager634may be capable of configuring different layers such as software layer630and framework layer620including Spark and distributed file system638for supporting large-scale data processing. The resource manager636may be capable of managing clustered or grouped computing resources mapped to or allocated for support of distributed file system638and job scheduler632. In at least one embodiment, clustered or grouped computing resources may include grouped computing resource614at data center infrastructure layer610. The resource manager636may coordinate with resource orchestrator612to manage these mapped or allocated computing resources.

In at least one embodiment, software632included in software layer630may include software used by at least portions of node C.R.s616(1)-616(N), grouped computing resources614, and/or distributed file system638of framework layer620. One or more types of software may include, but are not limited to, Internet web page search software, e-mail virus scan software, database software, and streaming video content software.

In at least one embodiment, application(s)642included in application layer640may include one or more types of applications used by at least portions of node C.R.s616(1)-616(N), grouped computing resources614, and/or distributed file system638of framework layer620. One or more types of applications may include, but are not limited to, any number of a genomics application, a cognitive compute, and a machine learning application, including training or inferencing software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and/or other machine learning applications used in conjunction with one or more embodiments.

In at least one embodiment, any of configuration manager634, resource manager636, and resource orchestrator612may implement any number and type of self-modifying actions based on any amount and type of data acquired in any technically feasible fashion. Self-modifying actions may relieve a data center operator of data center600from making possibly bad configuration decisions and possibly avoiding underutilized and/or poor performing portions of a data center.

Example Network Environments

Network environments suitable for use in implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and/or other device types. The client devices, servers, and/or other device types (e.g., each device) may be implemented on one or more instances of the computing device(s)500ofFIG.5—e.g., each device may include similar components, features, and/or functionality of the computing device(s)500. In addition, where backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may be included as part of a data center600, an example of which is described in more detail herein with respect toFIG.6.

As used herein, a recitation of “and/or” with respect to two or more elements should be interpreted to mean only one element, or a combination of elements. For example, “element A, element B, and/or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, “at least one of element A and element B” may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Additionally, use of the term “based on” should not be interpreted as “only based on” or “based only on.” Rather, a first element being “based on” a second element includes instances in which the first element is based on the second element but may also be based on one or more additional elements.