Patent Description:
Hyper-personalization attempts to achieve the same goals as the personalization experience described above, but in a manner that is specifically tailored to a person based on that person's customer data. Delivering a hyper-personalized experience for a customer typically involves the application of a user's private data (e.g., buying history, usage information, financial information, demographics, biometrics, relationships/social connections) to sophisticated machine learning algorithms.

E-commerce vendors and other service providers typically invest a lot of money in creating and training machine learning models, and such models may thereafter embody a great deal of proprietary business intelligence. For this reason, such models are closely held secrets. That is, vendors simply cannot trust that their models will not be misused, and therefore choose not to distribute such models.

Accordingly, to receive a hyper-personalized experience based on the output of such a machine learning model, a user typically must be willing to provide all their private data to a vendor/service provider for them to apply the data to their model. Unfortunately, this means that the user simply must trust that the service provider will not mis-use the personal data (e.g., by selling access to the data to third-parties), and that the service provider is willing and able to safeguard the data (i.e., prevent hackers from stealing the data). <CIT> discloses apparatuses, methods and storage media that are associated with user profile selection using contextual authentication. <CIT> discloses a multi-party privacy-preserving cloud machine learning system which has a trusted execution environment that executes machine learning code to process confidential data.

The invention provides a method in a computing device according to claim <NUM>, a system according to claim <NUM>, and a computer program product according to claim <NUM>.

Methods, systems and computer program products are described herein that enable users to receive hyper-personalized experiences while retaining possession and control over all of their private data. Furthermore, service providers are enabled to deliver hyper-personalized experiences while maintaining the secrecy of proprietary machine learning models. Further embodiments may advantageously permit detection of abnormal user behaviors (e.g., by online service providers) and abnormal machine behavior (i.e., detection of malware, viruses, worms, root kits, and the like), and provide for the prediction of device failures and providing of automatic remediation measures to address same.

In an example aspect, a secured virtual container is maintained on a computing device, where the secured virtual container is isolated from an operating system executing on the computing device. The secured virtual container and operating system may each run in parallel through a shared hypervisor, with virtualization features of the hypervisor and underlying hardware enforcing the isolation of each. In alternative embodiments, the secured container may be implemented in a hardware container (i.e., not virtualized on the computing device) wholly separate from the computing device.

In further aspects, the secured virtual container is enabled to securely store personal data corresponding to a user, where such data is inaccessible to processes running outside the secured virtual container. Such data may partially or wholly comprise features and/or feature vectors suitable for use with a machine learning model. A set of features corresponding to an inference category may be selected from the data, and an inference value for the category may be generated. Such generation may be accomplished in various ways such as by a suitably trained machine learning model. Thereafter, information regarding the availability of one or more inference values for various inference categories may be published to a broker external to the secured virtual container. The broker, for instance, may comprise an application running in an operating system separate and isolated from the secured virtual container. Applications may thereafter query the broker for the availability of one or more inference values corresponding to particular inference categories, and upon receiving such inference values, perform hyper-personalized operations based at least in part thereon.

Further features and advantages, as well as the structure and operation of various examples, are described in detail below with reference to the accompanying drawings. It is noted that the ideas and techniques are not limited to the specific examples described herein. Such examples are presented herein for illustrative purposes only. Additional examples will be apparent to persons skilled in the relevant art(s) based on the teachings contained herein.

The features and advantages of embodiments will become more apparent from the detailed description set forth below when taken in conjunction with the drawings, in which like reference characters identify corresponding elements throughout.

In embodiments, secure and private hyper-personalization is enabled by applying an obfuscation process to data specific to a particular user, whereby the user's data is, for example, hashed, normalized, and/or feature engineered, and thereafter provided in digest form to applications and/or operating system services. In embodiments, the obfuscation process may comprise applying user data to a machine learning model. User data is accordingly input to the system for hyper-personalization.

Such user data may, for example, by stored in a protected data store that is not directly accessible to a computing device's operating system or applications. Likewise, the obfuscation process may run within a protected personalization container that includes the protected data store. For example, a machine learning model may be securely transferred to the protected personalization container and made operable to perform operations on the use data stored within the protected data store (with due care being taken that such operations do not leak the user data outside the container). The output of the machine learning model may comprise the above mentioned hashed/normalized/feature engineered digest form of the user data that may be provided to the operating system and/or applications. Using a protected personalization container in this manner safeguards both user data, and the machine learning models that operate on the data. The various types of user data usable by embodiments will now be described.

User data may be collected in various ways, such as with the help of multiple direct or indirect sensors, and may be subsequently processed and/or stored in various ways, such as the form of a graph (e.g., using a graph database system). Such a graph may be constructed in a logically layered manner. For example, the first layer may consist of a policy layer that embodies policies, rules, values, and/or other foundational operating principles of the system that rarely change. For example, hyper-personalized work environment may operate based in part on security rules set by the company. Such rules may be based on a user's preference, risk profile, social profile and the like. Corporate machines that implement embodiments of a protected personalization system as described herein may be configured to have certain corporate rules and exclusion principles related to usage of the machine. Such rules may be a very granular and targeted interpretation of corporate policy that system configuration ("SCCM") or mobile device management ("MDM") tools enforce. For example, SCCM/MDM rules may disable USB ports on a device, prevent sending emails with any attachments and/or prevent or limit the taking of screenshots.

A second graph layer may include a knowledge graph, in an embodiment. A knowledge graph contains slowly changing knowledge about a user. The knowledge graph layer may be regarded as a 'warm' data layer inasmuch as such data changes slowly over time (e.g., time scales greater than a day). For example, the knowledge graph may reflect the user's risk profile, financial profile, application usage profile(s) and/or personalization data (e.g., preferences), habits, relationships, demographics, psychographic information, health, education, hobbies, commitments, basic social and/or professional network. Demography information may include, for example, recent face image, skin color, hair color, eye color, name, age, income, profession, home location, office location, resume information, music taste, Wi-Fi names, passwords, family members details, and the like. A knowledge graph may also include information about the computing device in use such as, make and model of the computer or mobile device, machine/device health status, as well as identification of available sensors. As mentioned above, the information in the knowledge layer changes relatively infrequently and embodiments may update such information using batch algorithms during night/free times.

Embodiments may also implement a third graph layer described herein as the transient layer. A transient layer typically includes data created and/or updated during a recent pre-determined time interval (e.g., time scales less than a day). For example, embodiments may create the transient layer by processing a rolling <NUM>-minute window of signals captured by the sensors and running basic processing to get a basic view of the state of use of the personal computer. For example, use states, presence, flow of users, dwell, interaction, engagement, atmosphere and system states. Transient layer information may also include the lock state of a computing device, the identity of the at least one user of the computing device, the location of the computing device, policy violations on the computing device, the identity of persons physically present with the at least one user of the computing device, the task being performed on the computing device, reminders, SMS (short message service) or MMS (multimedia messaging service) messages, emails, memory and/or file access signals, application states and application specific data.

Data in a transient layer may also include data corresponding to some predetermined period of time into the future. For example, the transient layer could include a trailing <NUM> minutes of sensor and other data, as well as data regarding events that will happen, for example, in the near future. In embodiments, such future focused transient layer data may be at least partially gleaned from calendar and/or free/busy data of the user, and thus reflect near future time commitments or other promises made via email, social networks, and the like. Alternatively, embodiments may learn user habits over time and predict likely near future actions of the user, and include such in the transient layer.

Transient layer data may include temporary/ephemeral data because certain types of data are not useful beyond a certain limited timeframe. For example, many useful types of data are only of interest in real time (e.g., temperature or location). Transient layer data need not, however, be completely temporary. Some transient layer data may be persisted in, for example, the second layer described above. For example, activity and/or usage data related to the user may not only be of interest in the present moment (as reflected by the transient layer), but also may be of interest over a longer time frame for determining general usage patterns over time. In addition to these data layers, a graph may include a service/management layer that includes functions for managing, updating and querying the data layers as will discussed in more detail below.

Thus, in embodiments, the transient layer will have a constantly changing graph of data with who the user is, who else may be present with them, where the user is, whether that is a public location (i.e., a protected location or not), whether the user is in motion or at rest, how fast the user may be traveling. Accordingly, the transient layer may be regarded as 'hot' data that rapidly changes with the user states.

Each of the abovementioned layers may correspond to one or more processing layers, in embodiments. For example, "hot path" processing of transient layer data gathered from sensors may be cached, and such data quarriable via API (application programming interface) calls. Similarly, information in the knowledge graph layer may be handled via a batch processing layer that may create analytical outputs, in form or forecast, classifications, and generative data about the user and environment of the types discussed in detail above.

Hyper-personalization services are described as follows in the context of a specific example. In particular, consider a computer operating system configured to provide hyper-personalization of the user interface, as well as provide hyper-personalization services to applications running on the operating system. It should be understood, however, that the described computer and operating system are merely exemplary, and embodiments may readily be implemented on other types of computing devices such as mobile devices/smart phones, and as discussed further herein below.

In embodiments, and as discussed above, enabling hyper-personalization requires that a user agree to the collection and use of information regarding the user. When a user agrees to enable hyper-personalization, they agree that the system may gather and process information only for internal device level consumption, and not for sharing to third parties. Granting such permission allows, for example, a laptop or desktop running in hyper-personalization mode to connect to any of a variety of data gathering devices such as: cameras, microphones, gaming consoles (e.g., Microsoft Xbox), mobile phones, TVs, monitors, printers, Bluetooth peripherals and any other devices that the operating system may access. Various types of data may be collected such as, for example, audio, video, radio signals, images, ambient light readings, motion, location, vibrations, velocity, acceleration, inertial sensor readings, magnetism, pressure, temperature, voltage, current, moisture and/or any other sensor information that the computer may access/receive.

When a user attaches a peripheral to the computer, users typically expect a nearly "Plug and Play" experience. That is, the computer will have necessary access to the devices to connect and activate them using driver software. Similarly, a protected personalization system executing on the personal computer may act as the user agent, activate needed peripherals and/or sensors at different time intervals, and collect information about the user state and local environment. Embodiments of a protected personalization system may have strict user awareness through, for example, conventional identification and authentication mechanisms (i.e., the system customizes operation based on who is logged into the machine). Other embodiments of a protected personalization system may be configured to automatically identify the user through sensors.

Whether embodiments function through login dialogs, or through automatic sensor-based identification, it should be understood that the hyper-personalization experience may vary for the same user. That is, embodiments may track multiple personas for each person, wherein a persona corresponds to a particular usage context. For example, a user may use their office computer at home as well as in the office. In the "Office" persona of the user, the user mostly sits at a desk and a camera can pick up the background and items around users to detect the location (other means of detecting location are possible in embodiments, such as global positing system (GPS), etc.). Moreover, people often wear certain types of clothes when in an "office" or "workplace" persona. For example, they may where hats and shirts with company logos, scrubs, where a certain hair style, use different glasses, wear more or less make-up, and the like. Workplaces typically will also have relatively unique visual and audio characteristics (at least as compared to a home environment). For example, workplace infrastructure/furniture such as cubicles, desks, counters, chairs and the like generally are different from home infrastructure/furniture.

In addition to visual cues, audio at every location is different, and signatures may be identifiable in each. For example, workplace locations will have the hissing of the computers and fans, low frequency voice transmission through the walls, ringing phones, elevator noise, printers, drawers, coffee machine, industrial refrigerators, air conditioners and the like which all emit different sounds than may typically be present in a home environment. Besides audio and visual clues, there are other signals such as use of docking station, Wi-Fi, keyboard and mouse, printer connections etc. that may also tell us about the location of the user and what persona he or she will likely to have at any point. All the above described differences may be detected and stored (typically in the transient layer) and dictate which persona of the user should govern the hyper-personalization experience.

Enabling a secure personalization system to gather and store the above described user information, and to obfuscate such information in a secure modeling environment may be accomplished in numerous ways. For example, <FIG> depicts an example computing device <NUM> including a protected personalization system <NUM>, according to an embodiment. As shown in <FIG>, computing device <NUM> includes an application <NUM> and protected personalization system <NUM>. Application <NUM> includes a GUI <NUM> that includes personalized content/function <NUM>. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the following discussion regarding computing device <NUM> as depicted in <FIG>.

Embodiments of computing device <NUM> may include any type of mobile computer or computing device such as a handheld device (e.g., a Palm® device, a RIM Blackberry® device, a personal digital assistant (PDA)), a laptop computer, a notebook computer, a tablet computer (e.g., an Apple iPad™, a Microsoft Surface™, etc.), a netbook, a mobile phone (e.g., a smart phone such as an Apple iPhone, a Google Android™ phone, a Microsoft Windows® phone, etc.), a wearable device (e.g., virtual reality glasses, helmets, and visors, a wristwatch (e.g., an Apple Watch®)), and other types of mobile devices. In further embodiments, computing device <NUM> may be stationary computer or computing device, such as a desktop computer.

In embodiments, protected personalization system <NUM> is configured to securely store user information of the types described herein, and to securely process such information to produce digest forms of the user information. For example, protected personalization system <NUM> may be configured to accept a suitably trained machine learning model capable of accepting user information (whether in raw form or pre-processed into suitable features) and producing inferences <NUM> therefrom. Inferences <NUM> may comprise, for example, a score representing the probability that a given proposition about the user is true based upon the user information securely provided to the model. For example, inferences <NUM> may include the probability that the user is in the office, the probability that the user likes particular shows or genres of shows, the probability that the user has bought particular types of products in the last <NUM> months, or the probability that the user belongs to a particular demographic group. Note, the above described example inferences <NUM> are merely exemplary, and inferences <NUM> may include virtually any type of inference capable of being modeled based on the available user information.

Numerous ways exist of implementing protected personalization system <NUM> and interfacing protected personalization system <NUM> with an operating system and/or application. For example, <FIG> depicts an example protected personalization system <NUM>, according to an embodiment. Protected personalization system <NUM> includes a personalization broker <NUM> and a protected personalization container <NUM>. The PPC <NUM> includes a personalization data processor <NUM> and a personal data store <NUM>. The personalization data processor <NUM> includes a Machine Learning ("ML") Engine <NUM>. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the following discussion regarding protected personalization system <NUM> as depicted in <FIG>.

At a high level, embodiments of protected personalization system <NUM> may be configured to receive and store user data <NUM> (i.e., all the above described types user data) in policy, knowledge graph and transient data layers within personal data store <NUM> of protected personalization container <NUM>. Protected personalization container <NUM> prevents compromised applications or operating system components from directly accessing user data <NUM> as stored in personal data store <NUM>, and instead requires all access to go through personalization broker <NUM>. Personalization broker <NUM> is configured to securely interface with personalization data processor <NUM> to perform such indirect access to user data <NUM>.

Personalization data processor <NUM> may be configured to select features and/or labels from user data <NUM> as stored in personal data store <NUM> to train machine learning modules residing in ML engine <NUM>. Alternatively, pre-trained ML models may be received and stored by personalization data processor <NUM> for subsequent processing of features selected from or generated by personalization data processor <NUM> and/or personal data store <NUM> based upon user data <NUM> stored in personal data store <NUM>. Features to be selected may be determined based at least in part on which model or models is/are present in ML engine <NUM> inasmuch as different models generally generate different inferences, and depend on different types of underlying user data <NUM>. These general operations, among others, of protected personalization system <NUM> and components contained therein are now described in further detail.

In embodiments, ML engine <NUM> may interoperate with or employ various machine learning frameworks, converters, runtimes, compilers and visualizers as known to persons skilled in the relevant art(s). For example, ML engine <NUM> may be configured to include and/or operate models in the Open Neural Network Exchange ("ONNX") format. ONNX is an open format for machine learning models that allows models to be shared and adapted for use with various ML frameworks and tools. For example, Microsoft Windows® ML allows for rapid integration of pre-trained machine learning models into various applications, and embodiments may adapt Windows® ML for use inside the above described secure container. Alternative embodiments may, instead of or in addition to adapting a ML framework such as Microsoft Windows® ML, instantiate a short-lived data pack and access protocol enabling usage of ONNX models on short-lived data of user data <NUM>. Example machine learning models will be discussed in further detail herein below in conjunction with <FIG> and <FIG>.

In embodiments, protected personalization container <NUM> comprises a virtual container that is isolated from an operating system running the user system and applications. Such isolation prevents even the operating system from accessing user data <NUM>, which thereby prevents any malicious programs running therein from accessing such data. In embodiments, protected personalization container <NUM> may comprise a container such as a virtual sandbox that operates within the context of the operating system, but is sufficiently hardened to prevent direct operating system access to the user data <NUM> as stored in personal data store <NUM>.

Alternatively, and as described in more detail below, protected personalization container <NUM> may comprise a virtualized container that runs in parallel with and fully isolated from the operating system. Examples of such containers may include virtual secure mode ("VSM") containers in Windows <NUM> Enterprise, Intel Clear Containers, Kata containers and/or Google gVisor containers. Embodiments of protected personalization container <NUM> may be configured to incorporate personal data store <NUM> and personalization data processor <NUM> within the confines of the container thereby securely separating processes running in personalization data processor <NUM>, and user data <NUM> stored in personal data store <NUM>, from the operating system.

Embodiments of personalization data processor <NUM> are configured to act as the interface between user data <NUM> stored in personal data store <NUM>, and systems and processes that exist outside of protected personalization container <NUM>. Personalization data processor <NUM> is configured to support data obfuscation operations through ML engine <NUM>. In particular, ML engine <NUM> is configured to include, or to receive and incorporate, machine learning models that digest user data <NUM> retrieved from personal data store <NUM> to produce the above described inference values <NUM>, and provide such inference values 112a to personalization broker <NUM> for relaying as inference values 112b to external consumers.

Personalization data processor <NUM> may also be configured to keep track of the various types or categories of inferences that may be accessed through personalization broker <NUM>, and to provide inference categories <NUM> to personalization broker <NUM>. Personalization broker <NUM> is in turn configured to publish inference categories <NUM> to entities outside of protected personalization system <NUM> that may wish to retrieve such inferences to construct a hyper-personalization experience for the user. Personalization broker <NUM> may be further configured to accept inference queries/subscriptions <NUM> from outside entities. Inference queries/subscriptions <NUM> may comprise one or more direct queries to personalization broker <NUM> for desired inference values and may also comprise one or more subscriptions. In embodiments, inference values may be logically grouped together into topics. A topic may comprise, for example, a category or type of inference value that may be of interest. For example, inference values related to a user's hobbies may be logically grouped into a "hobbies" topic. Interested entities may subscribe to the "hobbies" topic and thereafter be notified of any new or changed inference values that have been tagged with the "hobbies" topic.

As discussed above, user data <NUM> stored in personal data store <NUM> is subject to change over time. In the case of transient layer data, such information may be subject to rapid change. Likewise, inferences based on such information must therefore change over time. Inference subscriptions permit outside entities to instruct personalization broker <NUM> to automatically detect changes to inferences of interest, and to send one or more notifications <NUM> when updated inference values 112a are available. Alternatively, personalization broker <NUM> may be configured to operate in a push mode whereby inference values 112b are automatically pushed to subscribers as changes to such inferences are made by personalization data processor <NUM> either alone or in conjunction with ML engine <NUM>.

As described above, protected personalization system <NUM> and protected personalization container <NUM> may be configured in various ways. For example, <FIG> depicts a stack view of an example computing device <NUM> including a protected personalization system, according to an embodiment. Computing device <NUM> includes a host operating system <NUM>, protected personalization container <NUM>, hypervisor <NUM> and hardware <NUM>. Other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the following discussion regarding computing device <NUM> as depicted in <FIG>.

In embodiments, host OS <NUM> and protected personalization container <NUM> are each virtual machines running atop hypervisor <NUM> which in turn is running on, and abstracts the underlying hardware <NUM>. Host OS <NUM> includes a kernel <NUM> for performing operating system functions and providing an application environment wherein application <NUM> and personalization broker <NUM> may execute. Protected personalization container <NUM> likewise includes its own kernel <NUM> that not only provides protected personalization container <NUM> specific system functions (e.g., retrieval of data from personal data store <NUM>), but also provides the operating environment wherein personalization data processor <NUM> may execute. Hypervisor <NUM> is configured to prevent processes in host OS <NUM> and protected personalization container <NUM> from directly accessing the resources of the other. In operation, personalization broker <NUM> of host OS <NUM> may be configured to communicate with protected personalization container <NUM> by, for example, a network connection thereby enabling communication of inference values 112b from protected personalization container <NUM> to host OS <NUM>. Other techniques of enabling communication between isolated containers may be employed as may become apparent to persons skilled in the relevant art(s) based on the teachings herein.

With continued reference to computing device <NUM> of <FIG>, personalization broker <NUM> running in host OS <NUM> may be configured to accept inference values 112a and relay same to running applications (e.g., application <NUM>) or operating system components elsewhere in host OS <NUM>, where such applications and/or operating system components may be configured to perform personalization operations based at least in part on inference values 112b. For example, application <NUM> may be configured to customize a user interface associated with application <NUM>. Such customization may be performed based on, for example, an inference values 112b that indicates a high likelihood that the user is currently located at work, and at a particular location on the work campus (i.e., by displaying notifications of events near the user).

Components of host OS <NUM> are configured to perform customization operations based on inference values 112b. According to the invention, host OS <NUM> is configured to alter display output characteristics to reduce blue light output based on a) the time of day, and b) inference values 112b that indicate a high probability that the user environment currently has reduced ambient lighting, and c) where other inference values 112b indicate a high probability that the user has a configuration preference for, or a habit of setting, a low blue light display setting in low ambient lighting at night. It should be noted that these examples are far from exhaustive, and various inference categories and values are limited only by availability of machine learning model(s) suitably configured to generate a desired inference value, and availability of sufficient user data <NUM> for use by such model(s).

Further operational aspects of computing device <NUM> of <FIG>, and protected personalization system <NUM> of <FIG> will now be discussed in conjunction with <FIG> which depicts a flowchart <NUM> of an example method an example method for providing secure hyper-personalization in a computing device, according to an embodiment. Flowchart <NUM> is described with continued reference to <FIG> and <FIG>. However, other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the following discussion regarding flowchart <NUM> of <FIG> and protected personalization system <NUM> of <FIG>.

Flowchart <NUM> begins at step <NUM>. At step <NUM>, feature data is stored in a secured virtual container executing on a computing device in parallel with and isolated from an operating system executing on the computing device. For example, and with reference to protected personalization system <NUM> of <FIG>, personal data store <NUM> within protected personalization container <NUM> (i.e., a "secured virtual container") may be configured to store feature data such as, for example, personal user data <NUM> of the types described herein above. Also as described above in conjunction with the description of <FIG> and <FIG>, protected personalization container <NUM> may comprise a virtual container executing on a computing device in parallel with and isolated from an operating system executing on the device. In particular, and with reference to <FIG>, protected personalization container <NUM> may be configured to execute atop hypervisor <NUM>, in parallel with host OS <NUM> and isolated therefrom. Flowchart <NUM> of <FIG> continues at step <NUM>.

In step <NUM>, a first set of features is selected from the stored feature data. For example, and with continued reference to protected personalization system <NUM> of <FIG> and <FIG>, in the manner described in detail above, in an embodiment. More specifically, normalized or otherwise feature engineered versions of user data <NUM> may be retrieved from personal data store <NUM> and subsequently provided to ML engine <NUM> for processing. Selection of such features depends on the specific data a given model requires for generating a particular inference. Moreover, though a given model may be capable of generating multiple inferences, not all such inferences may be of interest to external consumers at any given moment in time, and consequently, corresponding features need not be selected and retrieved. Flowchart <NUM> of <FIG> continues at step <NUM>.

In step <NUM>, a first inference value for a first inference category is generated in the secured virtual container based at least in part on the first set of features. For example, and with continued reference to protected personalization system <NUM> of <FIG> and <FIG>, ML engine <NUM> may be configured to include a suitably trained machine learning model configured to accept feature data (i.e., feature processed user data <NUM>) retrieved from personal data store <NUM>, and to generate one or more inference values in the manner described in detail above, in embodiments. Flowchart <NUM> continues at step <NUM>.

At step <NUM>, availability of the first inference value corresponding to the first inference category is notified to a broker external to the secured virtual container. For example, and with continued reference to protected personalization system <NUM> of <FIG> and <FIG>, personalization data processor <NUM> may be configured to generate notifications <NUM> in response to the generation of inference values by ML engine <NUM>, and to send same to personalization broker <NUM> in the general manner described in detail above, in embodiments. In the embodiment illustrated in <FIG>, personalization data processor <NUM> (as well as ML Engine <NUM> which is part of personalization data processor <NUM>) is included in protected personalization container <NUM>, whereas personalization broker <NUM> is external to the secured virtual container (i.e., protected personalization container <NUM>). Flowchart <NUM> continues at step <NUM>.

At step <NUM>, the first inference value is received at the broker from the secured virtual container. For example, and with continued reference to protected personalization system <NUM> of <FIG> and <FIG>, personalization broker <NUM> is configured to accept inferences <NUM> relayed from protected personalization container <NUM> after generation of such inferences by ML engine <NUM> of personalization data processor <NUM> in the same general manner as described in detail above, in embodiments. Flowchart <NUM> concludes at step <NUM>.

At step <NUM>, the first inference value is provided by the broker to at least one running process in the operating system, wherein the at least one running process is configured to perform a personalization operation based at least in part on the first inference value. For example, and with continued reference to protected personalization system <NUM> of <FIG> and <FIG>, personalization broker <NUM> inference values 112a as received from personalization data processor <NUM> of protected personalization container <NUM> may be provided as inference values 112b to processes outside of protected personalization system <NUM>. For example, personalization data processor <NUM> within protected personalization container <NUM> may be configured to communicate inference values 112a from protected personalization container <NUM> to personalization broker <NUM> executing in the context of host OS <NUM>, where personalization broker <NUM> in turn communicates inference values 112b to application <NUM> (also running on host OS <NUM>). As discussed above, application <NUM> may be configured to customize the user experience based at least in part on the inference values 112b received from personalization broker <NUM>.

In the foregoing discussion of steps <NUM>-<NUM> of flowchart <NUM>, it should be understood that at times, such steps may be performed in a different order or even contemporaneously with other steps. Other operational embodiments will be apparent to persons skilled in the relevant art(s). Note also that the foregoing general description of the operation of protected personalization system <NUM> is provided for illustration only, and embodiments of protected personalization system <NUM> may comprise different hardware and/or software, and may operate in manners different than described above. Indeed, steps of flowchart <NUM> may be performed in various ways.

For example, <FIG> depicts a flowchart <NUM> of an additional example method of generating event suggestions, according to an embodiment, and wherein flowchart <NUM> comprises refinements or additions to the method steps of flowchart <NUM> as depicted in <FIG>. Accordingly, flowchart <NUM> of <FIG> will also be described with continued reference to protected personalization system <NUM> of <FIG> and personalization broker <NUM>, protected personalization container <NUM>, personalization data processor <NUM> and ML engine <NUM> of <FIG> and <FIG>. However, other structural and operational embodiments will be apparent to persons skilled in the relevant art(s) based on the following discussion regarding flowchart <NUM>.

In step <NUM>, additional feature data is received at the secured virtual container at a time subsequent to receiving the feature data, the additional feature data at least reflecting changes in the first set of features. For example, and with continued reference to protected personalization system <NUM> of <FIG> and <FIG>, information corresponding to a user and/or the user's operating environment is, as described above, being constantly gathered over time. Accordingly, user data <NUM> is constantly being transmitted to personal data store <NUM> for storage. Such user data <NUM> comprises additional feature data inasmuch as such data is subject to normalization or other feature engineering operations as known in the art. Flowchart <NUM> continues at step <NUM>.

In step <NUM>, the additional feature data is merged with the first set of features to provide a second set of features. For example, and with continued reference to protected personalization system <NUM> of <FIG> and <FIG>, personal data store <NUM> may already contain user data <NUM> that was previously gathered by the system and received by personal data store <NUM> for storage. As discussed above, as new user data <NUM> is gathered and transmitted to personal data store <NUM> for storage, such user data <NUM> must be reconciled with or merged with existing data. For example, suppose an embodiment of computing device <NUM> of <FIG> is configured to send the lock state of computing device <NUM> to personal data store <NUM> for storage. In such an instance, the current lock state should always be persisted, but prior lock states and timestamps associated therewith may be useful for determining usage patterns of computing device <NUM>. Accordingly, historic lock state information may be maintained, and subsequently augmented with new lock state information as it is received at personal data store <NUM>. Further, when user data <NUM> has changed, embodiments of personal data store <NUM> and/or personalization data processor <NUM> may generate or receive updated features (i.e., "second set of features") based on feature engineering of such changed user data <NUM>. Flowchart <NUM> continues at step <NUM>.

In step <NUM>, the second set of features is provided to a first inference generation model included in the secured virtual container and configured to generate a second inference value based at least in part on the second set of features. For example, as described above in conjunction with step <NUM> of flowchart <NUM> of <FIG>, ML engine <NUM> may be configured to include a suitably trained machine learning model configured to accept feature data (i.e., feature engineered user data <NUM>) retrieved from personal data store <NUM>, and to generate one or more inference values in the manner described in detail above, in embodiments. ML engine <NUM> of personalization data processor <NUM> may likewise be configured to generate additional inference values as new or changed user data <NUM> is received and stored at personal data store <NUM>. Flowchart <NUM> concludes at step <NUM>.

In step <NUM>, in response to a request received from the broker for the inference value corresponding to the first inference category, and at a time subsequent to the receipt of the additional feature data, the second inference value is provided to the broker. For example, and with continued reference to protected personalization system <NUM> of <FIG> and <FIG>, embodiments may operate as described above, wherein personalization data processor <NUM> may be configured to generate and send notifications to external applications and/or components regarding changes or availability of inference values corresponding to one or more inference categories. More specifically, and also as described above, entities external to protected personalization system <NUM> may subscribe via personalization broker <NUM> to receive notifications regarding one or more inference values or inference categories, and personalization data processor <NUM> may be configured to generate such notifications in response to changes to feature values/categories of interest. Alternatively, personalization data processor <NUM> may also be configured to directly push updated inference values to subscribing components, where the updated inference values themselves serve as a notification.

In the foregoing discussion of steps <NUM>-<NUM> of flowchart <NUM>, it should be understood that at times, such steps may be performed in a different order or even contemporaneously with other steps. Other operational embodiments will be apparent to persons skilled in the relevant art(s). Note also that the foregoing general description of the operation of protected personalization system <NUM> is provided for illustration only, and embodiments of protected personalization system <NUM> may comprise different hardware and/or software, and may operate in manners different than described above.

As discussed above, embodiments may include and/or use various machine learning platforms and algorithms. For example, ONNX models, or other types of machine learning models that may be available or generated, may be adapted to generate inferences <NUM> from user data <NUM>. For example, a deep neural network ("DNN") may be constructed to generate one or more inferences <NUM> based on user data <NUM>. A DNN is a type of artificial neural network that conceptually is comprised of artificial neurons. For example, <FIG> depicts an example artificial neuron <NUM> suitable for use in a DNN, according to an embodiment. Neuron <NUM> includes an activation function <NUM>, a constant input CI <NUM>, an input In1 <NUM>, an input In2 <NUM> and output <NUM>. Neuron <NUM> of <FIG> is merely exemplary, and other structural or operational embodiments will be apparent to persons skilled in the relevant art(s) based on the following discussion regarding neuron <NUM> of <FIG>.

Neuron <NUM> operates by performing activation function <NUM> on weighted versions of inputs CI <NUM>, In1 <NUM> and In2 <NUM> to produce output <NUM>. Inputs to activation function <NUM> are weighted according to weights b <NUM>, W1 <NUM> and W2 <NUM>. Inputs In1 <NUM> and In2 <NUM> may comprise, for example, normalized or otherwise features processed data corresponding to user data <NUM>. Activation function <NUM> is configured to accept a single number (i.e., in this example, the linear combination of weighted inputs) based on all inputs, and perform a fixed operation. As known in the art, such operations may comprise, for example, sigmoid, tanh or rectified linear unit operations. Input CI <NUM> comprises a constant value which may typically be set to the value <NUM>, the purpose of which will be discussed further below.

A single neuron generally accomplishes very little, and a useful machine learning model typically includes the combined computational effort of a large number of neurons working in concert. For example, <FIG> depicts an example deep neural network ("DNN") <NUM> composed of neurons <NUM>, according to an embodiment. DNN <NUM> includes a plurality of neurons <NUM> assembled in layers and connected in a cascading fashion. Such layers include an input layer <NUM>, a <NUM>st hidden layer <NUM>, a <NUM>nd hidden layer <NUM> and an output layer <NUM>. DNN <NUM> depicts outputs of each layer of neurons being weighted according to weights <NUM>, and thereafter serving as inputs solely to neurons in the next layer. It should be understood, however, that other interconnection strategies are possible in other embodiments, and as is known in the art.

Neurons <NUM> of input layer <NUM> (labeled Ni1, Ni2 and Ni3) each may be configured to accept normalized or otherwise feature engineered or processed data corresponding to user data <NUM> as described above in relation to neuron <NUM> of <FIG>. The output of each neuron <NUM> of input layer <NUM> is weighted according to the weight of weights <NUM> that corresponds to a particular output edge, and is thereafter applied as input at each neuron <NUM> of <NUM>st hidden layer <NUM>. It should be noted that each edge depicted in DNN <NUM> corresponds to an independent weight, and labeling of such weights for each edge is omitted for the sake of clarity. In the same fashion, the output of each neuron <NUM> of <NUM>st hidden layer <NUM> is weighted according to its corresponding edge weight, and provided as input to a neuron <NUM> in <NUM>nd hidden layer <NUM>. Finally, the output of each neuron <NUM> of <NUM>nd hidden layer <NUM> is weighted and provided to the inputs of the neurons of output layer <NUM>. The output or outputs of the neurons <NUM> of output layer <NUM> comprises the output of the model. In the context of the descriptions above, such outputs comprise inferences <NUM>. Note, although output layer <NUM> includes two neurons <NUM>, embodiments may instead of just a single output neuron <NUM>, and therefore but a single discrete output. Note also, that DNN <NUM> of <FIG> depicts a simplified topology, and a producing useful inferences from a DNN like DNN <NUM> typically requires far more layers, and far more neurons per layer. Thus, DNN <NUM> should be regarded as a simplified example only.

Construction of the above described DNN <NUM> comprises the start of generating a useful machine learning model. The accuracy of the inferences generated by such a DNN require selection of a suitable activation function, and thereafter each and every one of the weights of the entire model are adjusted to provide accurate output. The process of adjusting such weights is called "training. " Training a DNN, or other type of neural network, requires a collection of training data with known characteristics. For example, where a DNN is intended to predict the probability that an input image of a piece of fruit is an apple or a pear, the training data would comprise many different images of fruit, and typically include not only apples and pears, but also plums, oranges and other types of fruit. Training requires that the image data corresponding to each image is pre-processed according to normalization and/or feature extraction techniques as known in the art to produce input features for the DNN, and such features thereafter are input to the network. In the example above, such features are input to the neurons of input layer <NUM>.

Thereafter, each neuron <NUM> of DNN <NUM> performs their respective activation function operation, the output of each neuron <NUM> is weighted and fed forward to the next layer, and so forth until outputs are generated by output layer <NUM>. The output(s) of the DNN may thereafter be compared to the known or expected value of the output. The output of the DNN may then be compared to the expected value and the difference fed backward through the network to revise the weights contained therein according to a backward propagation algorithm as known in the art. With the model including revised weights, the same image features may again be input to the model (e.g., neurons <NUM> of input layer <NUM> of DNN <NUM> described above), and new output generated. Training comprises iterating the model over the body of training data and updating the weights at each iteration. Once the model output achieves sufficient accuracy (or outputs have otherwise converged and weight changes are having little effect), the model is said to be trained. A trained model may thereafter be used to evaluate arbitrary input data, the nature of which is not known in advance, nor has the model previously considered (e.g., a new picture of a piece of fruit), and output the desired inference (e.g., the probability that the image is that of an apple).

In embodiments, ML Engine <NUM> as described above may be configured to enable generating and training machine learning models such as, for example, deep neural networks as described herein above. For example, various platforms such as Keras or TensorFlow may permit the construction of an untrained DNN suitable for use with ML Engine <NUM>, and such a model thereafter trained using user data <NUM>. Alternatively, pre-trained machine learning model (e.g., a DNN with optimal or near optimal weights for a given problem) may be imported to ML Engine <NUM>, and thereafter accept user data <NUM> as input for generation of inferences <NUM>.

<FIG> is a block diagram of an exemplary mobile device <NUM> that may implement embodiments described herein. For example, mobile device <NUM> may be used to implement protected personalization system <NUM>, protected personalization container <NUM>, personalization data processor <NUM>, ML engine <NUM>, personal data store <NUM> and/or personalization broker <NUM>, and/or any of the components respectively described therein and/or any of the steps of any of flowcharts <NUM> and/or <NUM>. As shown in <FIG>, mobile device <NUM> includes a variety of optional hardware and software components. Any component in mobile device <NUM> can communicate with any other component, although not all connections are shown for ease of illustration. Mobile device <NUM> can be any of a variety of computing devices (e.g., cell phone, smart phone, handheld computer, Personal Digital Assistant (PDA), etc.) and can allow wireless two-way communications with one or more mobile communications networks <NUM>, such as a cellular or satellite network, or with a local area or wide area network. Mobile device <NUM> can also be any of a variety of wearable computing device (e.g., a smart watch, an augmented reality headset, etc.).

Mobile device <NUM> can include a controller or processor <NUM> (e.g., signal processor, microprocessor, ASIC, or other control and processing logic circuitry) for performing such tasks as signal coding, data processing, input/output processing, power control, and/or other functions. An operating system <NUM> can control the allocation and usage of the components of mobile device <NUM> and provide support for one or more application programs <NUM> (also referred to as "applications" or "apps"). Application programs <NUM> may include common mobile computing applications (e.g., e-mail applications, calendars, contact managers, web browsers, messaging applications) and any other computing applications (e.g., word processing applications, mapping applications, media player applications).

Mobile device <NUM> can include memory <NUM>. Memory <NUM> can include non-removable memory <NUM> and/or removable memory <NUM>. Non-removable memory <NUM> can include RAM, ROM, flash memory, a hard disk, or other well-known memory devices or technologies. Removable memory <NUM> can include flash memory or a Subscriber Identity Module (SIM) card, which is well known in GSM communication systems, or other well-known memory devices or technologies, such as "smart cards. " Memory <NUM> can be used for storing data and/or code for running operating system <NUM> and application programs <NUM>. Example data can include web pages, text, images, sound files, video data, or other data to be sent to and/or received from one or more network servers or other devices via one or more wired or wireless networks. Memory <NUM> can be used to store a subscriber identifier, such as an International Mobile Subscriber Identity (IMSI), and an equipment identifier, such as an International Mobile Equipment Identifier (IMEI). Such identifiers can be transmitted to a network server to identify users and equipment.

Mobile device <NUM> can support one or more input devices <NUM>, such as a touch screen <NUM>, a microphone <NUM>, a camera <NUM>, a physical keyboard <NUM> and/or a trackball <NUM> and one or more output devices <NUM>, such as a speaker <NUM> and a display <NUM>. Other possible output devices (not shown) can include piezoelectric or other haptic output devices. Some devices can serve more than one input/output function. For example, touch screen <NUM> and display <NUM> can be combined in a single input/output device. Input devices <NUM> can include a Natural User Interface (NUI).

Wireless modem(s) <NUM> can be coupled to antenna(s) (not shown) and can support two-way communications between processor <NUM> and external devices, as is well understood in the art. Modem(s) <NUM> are shown generically and can include a cellular modem <NUM> for communicating with the mobile communication network <NUM> and/or other radio-based modems (e.g., Bluetooth <NUM> and/or Wi-Fi <NUM>). At least one of wireless modem(s) <NUM> is typically configured for communication with one or more cellular networks, such as a GSM network for data and voice communications within a single cellular network, between cellular networks, or between the mobile device and a public switched telephone network (PSTN).

Mobile device <NUM> can further include at least one input/output port <NUM>, a power supply <NUM>, a satellite navigation system receiver <NUM>, such as a Global Positioning System (GPS) receiver, an accelerometer <NUM>, and/or a physical connector <NUM>, which can be a USB port, IEEE <NUM> (FireWire) port, and/or RS-<NUM> port. The illustrated components of mobile device <NUM> are not required or all-inclusive, as any components can be deleted and other components can be added as would be recognized by one skilled in the art.

In an embodiment, mobile device <NUM> is configured to implement any of the above-described features of protected personalization system <NUM>, protected personalization container <NUM>, personalization data processor <NUM>, ML engine <NUM>, personal data store <NUM> and/or personalization broker <NUM>, and/or any of the components respectively described therein and/or any of the steps of any of flowcharts <NUM> and/or <NUM>. Computer program logic for performing the functions of these devices may be stored in memory <NUM> and executed by processor <NUM>.

Each of protected personalization system <NUM>, protected personalization container <NUM>, personalization data processor <NUM>, ML engine <NUM>, personal data store <NUM> and/or personalization broker <NUM>, and flowcharts <NUM> and/or <NUM> may be implemented in hardware, or hardware combined with software and/or firmware. For example, protected personalization system <NUM>, protected personalization container <NUM>, personalization data processor <NUM>, ML engine <NUM>, personal data store <NUM> and/or personalization broker <NUM>, and flowcharts <NUM> and/or <NUM> may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a computer readable storage medium. Alternatively, protected personalization system <NUM>, protected personalization container <NUM>, personalization data processor <NUM>, ML engine <NUM>, personal data store <NUM> and/or personalization broker <NUM>, and flowcharts <NUM> and/or <NUM> may be implemented as hardware logic/electrical circuitry.

For instance, in an embodiment, one or more, in any combination, of protected personalization system <NUM>, protected personalization container <NUM>, personalization data processor <NUM>, ML engine <NUM>, personal data store <NUM> and/or personalization broker <NUM>, and flowcharts <NUM> and/or <NUM> may be implemented together in a SoC. The SoC may include an integrated circuit chip that includes one or more of a processor (e.g., a central processing unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), one or more graphics processing units (GPUs), etc.), memory, one or more communication interfaces, and/or further circuits, and may optionally execute received program code and/or include embedded firmware to perform functions.

<FIG> depicts an exemplary implementation of a computing device <NUM> in which embodiments may be implemented. For example, user device <NUM> and server(s) <NUM> may be implemented in one or more computing devices similar to computing device <NUM> in stationary or mobile computer embodiments, including one or more features of computing device <NUM> and/or alternative features. The description of computing device <NUM> provided herein is provided for purposes of illustration, and is not intended to be limiting. Embodiments may be implemented in further types of computer systems, as would be known to persons skilled in the relevant art(s).

A number of program modules may be stored on the hard disk, magnetic disk, optical disk, ROM, or RAM. These programs include operating system <NUM>, one or more application programs <NUM>, other programs <NUM>, and program data <NUM>. Application programs <NUM> or other programs <NUM> may include, for example, computer program logic (e.g., computer program code or instructions) for implementing protected personalization system <NUM>, protected personalization container <NUM>, personalization data processor <NUM>, ML engine <NUM>, personal data store <NUM> and/or personalization broker <NUM>, and flowcharts <NUM> and/or <NUM> (including any suitable step of flowcharts <NUM> and/or <NUM>), and/or further embodiments described herein.

Other input devices (not shown) may include a microphone, j oy stick, game pad, satellite dish, scanner, a touch screen and/or touch pad, a voice recognition system to receive voice input, a gesture recognition system to receive gesture input, or the like.

Such computer programs, when executed or loaded by an application, enable computing device <NUM> to implement features of embodiments described herein.

Claim 1:
A method in a computing device (<NUM>) for providing secure hyper-personalization, comprising:
in a secured virtual container executing (<NUM>) on the computing device (<NUM>) and isolated from an operating system (<NUM>) executing on the computing device:
storing feature data comprising personal user data (<NUM>), the personal user data (<NUM>) comprising collected ambient light readings;
selecting a first set of features from the stored feature data and providing it to a machine learning engine (<NUM>) included in the secured virtual container (<NUM>);
generating, by the machine learning engine (<NUM>) comprising a suitably trained machine learning model, first inference values (<NUM>) for a first inference category based at least in part on the first set of features,
; and
notifying availability of the first inference values (<NUM>) corresponding to the first inference category to a broker (<NUM>) external to the secured virtual container (<NUM>), wherein the broker (<NUM>) is configured to securely interface with the secured virtual container (<NUM>); and
in the broker (<NUM>) in the computing device (<NUM>):
receiving the first inference values from the secured virtual container (<NUM>); and
providing, in response to a query for the availability of the first inference values (<NUM>) from at least one running process in the operating system (<NUM>), the first inference values to the at least one running process in the operating system (<NUM>), wherein the at least one running process is configured to perform a personalization operation based at least in part on the first inference values,
the at least one running process in the operating system (<NUM>) being configured to alter display output characteristics to reduce blue light output based on the time of day, inference values (<NUM>) that indicate a high probability that the user environment currently has reduced ambient lighting, and other inference values (<NUM>) that indicate a high probability that the user has a configuration preference for, or a habit of setting, a low blue light display setting in low ambient lighting at night.