Training and using a machine learning module to determine locales and augmented reality representations of information on locales to render in an augmented reality display

Provided are a computer program product, system, and method for training and using a machine learning module to determine locales and augmented reality representations of information on locales to render in an augmented reality display. A determination is made of qualifying locales accessible to a visiting user during a user available time from a current location. Values for locale features of the qualifying locales and values of user features in a user profile of the visiting user are provided as input to a locale attraction machine learning module to determine local attraction scores for the qualifying locales. A determination is made of qualifying locales based on locale attraction scores of the qualifying locales to transmit to the visiting user computing device to cause an augmented reality display to render augmented reality representations of information on the qualifying locales.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a computer program product, system, and method for training and using a machine learning module to determine locales and augmented reality representations of information on locales to render in an augmented reality display.

2. Description of the Related Art

Augmented reality (AR) smart glasses are wearable computer-capable glasses that generate digital information, such as three-dimensional images, text, animations, and videos, to overlay into the wearer's field of vision so the digital information is viewable along with real world scenes in the wearer's field of vision. Augmented reality is used to supplement information presented to users on items they are looking at, such as augmented reality controls to control items in the wearer's field of vision or information on locations in the field of vision.

There is a need in the art to provide improved techniques for deploying augmented reality representations to enhance user experiences.

SUMMARY

Provided are a computer program product, system, and method for training and using a machine learning module to determine locales and augmented reality representations of information on locales to render in an augmented reality display. A determination is made of qualifying locales accessible to a visiting user during a user available time from a current location of the visiting user. Values for locale features of the qualifying locales and values of user features in a user profile of the visiting user are provided as input to a locale attraction machine learning module to produce local attraction scores for the qualifying locales indicating measurements of attraction of the qualifying locales to the visiting user. The values for the locale features provide measurable properties of characteristics of the locales. A determination is made of qualifying locales based on locale attraction scores of the qualifying locales. Information on the qualifying locales is transmitted to the visiting user computing device to cause an augmented reality display to render augmented reality representations of information on the qualifying locales.

DETAILED DESCRIPTION

A person may look-up on their smartphone locations to visit when the have available time, such as venues, merchants, service providers, public spaces, etc., to determine what to engage. Described embodiments provide improvements to computer technology for generating augmented reality representations on suggesting locales for a user to engage. Described embodiments utilize databases of locale information, a machine learning module to determine locale attraction scores of nearby locales that are likely of interest and likely to provide a positive experience for the user, and patterns of behavior of interaction by similar users with locales to determine locales of interest for the user. Augmented reality representations of the locales likely to be of interest are rendered in an augmented reality display of the user to direct the user to the locales of likely interest and increase the likelihood of a serendipity moment.

FIG.1illustrates an embodiment of a locale attraction server100in communication with a visiting user computing device102over a network105. There may be multiple user or visiting user computing devices102, although only one is shown. The user computers102include a locale attraction client104to communicate a locale request106to the locale attraction server100for locales that would be of interest to the visiting user, including a current location, as provided by a Global Positioning Satellite (GPS) module108, and a user available time period during which the visiting user is available to visit locales. The visiting user is requesting the locale attraction server100to provide information on locales that the user would likely be interested in visiting during available time. The user computers102include an augmented realty generator110to generate augmented reality representations of locales of interest, determined by the locale attraction server100in response to the locale request106, in an augmented reality display112.

In one embodiment, the augmented reality display112may comprise a type of computer vision glasses to render augmented reality images. The augmented reality display112may further comprise a gaze tracking device to receive a gazed image detected by eye tracking cameras that acquire the gazed image on which the tracked eye is fixed and information on coordinates of an axis of a line-of-sight, also referred to as sightline, visual axis, the user is viewing within the field of vision captured by the gaze tracking device tracking. The local attraction client104may return user visit results114concerning whether the user visited or did not visit locales whose information was rendered in the augmented reality display112.

The locale attraction server100includes a locale analyzer116to receive the locale request106, including a user identifier, current GPS location, transportation mode, e.g., auto, walking, public transport, taxi, etc., and availability time period the user has to visit locales. For the initial locale request106, the locale analyzer116generates session information200(FIG.2) having information on the visiting user locale request106and determines qualifying locales118the user can visit given their available time, transportation mode, and current location. The local analyzer116generates machine learning (ML) input300(FIG.3), for each of the qualifying locales118, including locale feature values (FL) for features of the qualifying locale118from a locale information database400and user feature values (FU) of features of the user from a user personal information database500, and any other features having strong predictive qualities of locales a user would likely want to visit.

A locale may comprise a public area designated for a purpose, e.g., park with hiking trails, city park, golf course, tennis courts, a store, a shopping mall, an entertainment venue, service provider, such as doctor, etc., and any other locales a user can visit for which information is maintained in the locale information database400.

The machine learning input300is provided to the locale attraction machine learning module120to produce a locale attraction score122for the qualifying locale118for which the machine learning input300is provided. A locale packager124receives the qualifying locales118and the locale attraction scores122for the qualifying locales118and packages them into qualifying groups600of one or more locales having highest locale attraction scores122that the user can visit within the user available time.

The local packager124may utilize a fastest path algorithm128to determine a fastest path among the qualifying locales in a qualifying group600, wherein the qualifying locales packaged in the qualifying group600are ordered according to their position on the fastest path. Fastest paths may be computed as minimal-cost paths in a weighted directed graph of the qualifying locales in a group600. The minimal-cost path algorithms may be a variant of the Dijkstra algorithm, the A* algorithm, and other path algorithms that further optimize the calculation. A qualifying group of locales600may be returned with information on the locales to the visiting user computing device102to use to render augmented reality representations of information on the locales in the group600and an order to visit them, based on the fastest path, in the augmented reality display112. This augmented reality information may lure the user to the likely locales of interest in the qualifying group600in the fastest order.

Upon receiving the user visit results114from the locale attraction client104at the visiting user computing device102, indicating the visited and not visited locales for which augmented reality information was presented, a visit collector130generates locale visit information700ihaving information on the visit to a locale, such as duration of visit, to store in the locale visit database700. A training set builder132processes the user visit results114to generate a training set800ifor each locale indicated in the results, including the machine learning input used to generate the attraction score for the locale and an adjusted attraction score comprising the original calculated attraction score122adjusted to reflect whether the user visited or did not visit the locale for which augmented reality representations were generated. For instance, if the user visited the locale, then the adjusted attraction score would indicate a higher level of user attraction to reflect the visit. If the user did not visit the highlighted locale, then the attraction score is adjusted lower to reflect less user attraction to the locale. The training set800iis stored in the training set database800.

A machine learning trainer134may periodically access training sets800ifrom the training set database800to use to train the locale attraction machine learning module120by forming an input set, such as a matrix, of the machine learning inputs used to produce the previously calculated attraction scores, and form an output set of the adjusted locale attraction scores from the training sets700i, such as a vector. The locale attraction machine learning module120is then trained with the input set to produce the output set of adjusted locale attraction scores reflecting whether the user decided to visit or not visit the locale for which augmented reality representations were presented.

The network105may comprise a network such as a Storage Area Network (SAN), Local Area Network (LAN), Intranet, the Internet, Wide Area Network (WAN), peer-to-peer network, wireless network, arbitrated loop network, etc.

The arrows shown inFIG.1between the components and objects in the locale attraction server100and the user computing device102represent a data flow between the components.

In certain embodiments, the locale attraction machine learning module120may use machine learning and deep learning algorithms, such as decision tree learning, association rule learning, neural network, inductive programming logic, support vector machines, Bayesian network, etc. For artificial neural network program implementations, the neural network may be trained using backward propagation to adjust weights and biases at nodes in a hidden layer to produce their output based on the received inputs. In backward propagation used to train a neural network machine learning module, biases at nodes in the hidden layer are adjusted accordingly to produce the locale attraction scores having specified confidence levels based on the input parameters. For instance, the input300to the locale attraction machine learning module120is processed to produce a locale attraction score122with a confidence level. The locale attraction machine learning module120may be trained to produce the locale attraction scores122based on the inputs300from the training sets700i. Backward propagation may comprise an algorithm for supervised learning of artificial neural networks using gradient descent. Given an artificial neural network and an error function, the method may use gradient descent to find the parameters (coefficients) for the nodes in a neural network or function that minimizes a cost function measuring the difference or error between actual and predicted values for different parameters. The parameters are continually adjusted during gradient descent to minimize the error.

In backward propagation used to train a neural network machine learning module, such as the locale attraction machine learning module120, margin of errors are determined based on a difference of the calculated predictions and user rankings of the output. Biases (parameters) at nodes in the hidden layer are adjusted accordingly to minimize the margin of error of the error function.

In an alternative embodiment, the locale attraction machine learning module120may be implemented not as a machine learning module, but implemented using a rules based system to determine the outputs from the inputs. The locale attraction machine learning module120may further be implemented using an unsupervised machine learning module, or machine learning implemented in methods other than neural networks, such as multivariable linear regression models.

Generally, program modules, such as the program components104,108,110,116,120,124,128,130,132, and134may comprise routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. The program components and hardware devices of the computing devices100and102ofFIG.1may be implemented in one or more computer systems, where if they are implemented in multiple computer systems, then the computer systems may communicate over a network.

The program components104,108,110,116,120,124,128,130,132, and134may be accessed by a processor from memory to execute. Alternatively, some or all of the program components104,108,110,116,120,124,128,130,132, and134may be implemented in separate hardware devices, such as Application Specific Integrated Circuit (ASIC) hardware devices.

The functions described as performed by the program104,108,110,116,120,124,128,130,132, and134may be implemented as program code in fewer program modules than shown or implemented as program code throughout a greater number of program modules than shown.

The program components described as implemented in the locale attraction server100may be implemented in the user computer102.

The user computer102may comprise a personal computing device, such as a laptop, desktop computer, tablet, smartphone, etc. The server100may comprise one or more server class computing devices, or other suitable computing devices. Alternatively, the user computer102may be embedded in the augmented reality display112.

FIG.2illustrates an embodiment of session information200, generated when the locale attraction client104submits the initial request for locales of interest to visit, and includes: a unique session identifier (ID)202; a user ID204; a last GPS location206reported by the visiting user computing device102; a user available time208remaining from the initial user availability time period reported in the initial locale request106, which may comprise a timer initially set to the user availability time period in the locale request106that continually decrements as time passes; a transportation mode210being used by the visiting user, such as automobile, walking, taxi/ride sharing, public transport, etc.; locales visited212that the visiting user has visited during the session200; and presented locales not visited214for which augmented reality representations of information were generated in a frame or body of the augmented reality display.

FIG.3illustrates an embodiment of the machine learning input300as including, for a locale for which a locale attraction score is to be calculated: locale feature values (FL)302for a locale from the locale information database400, e.g., type of venue, services offered, goods sold, reviews, metadata tags, likely demographics interested and visiting, entertainment offered; recent change in number of visitors304that have visited; and recent change in user engagement duration by visitors306; and user feature values (FU)308from the visiting user personal information database500, such as user interests, bio-medical and demographic information, expressed likes and dislikes, profession, etc., and any other features or information that have strong prediction of the attraction a user would have toward a locale.

In one embodiment, the recent change information304,306may be generated when generating the machine learning input300and include changes in visitor patterns for visitors having similar user profiles to the user profile of the visiting user initiating the request106.

FIG.4illustrates an embodiment of locale information instance400imaintained for a locale i in the locale information database400, and may include: a locale ID402; a GPS location404of the locale; a time available406of when the locale is open and available; feature values (FL)408of the locale, e.g., type of venue, services offered, goods sold, reviews, metadata tags, likely demographics interested and visiting, entertainment offered; recent change in number of visitors410that have visited; and recent change in user engagement duration by visitors412, etc.

FIG.5illustrates an embodiment of user information500iin the user information database500, and may include a user ID502and user feature values (FU)504such as user interests, bio-medical and demographic information, expressed likes and dislikes, profession, etc., and any other features or information that have strong prediction of the attraction a user would have toward a locale.

FIG.6illustrates a locale group600formed by the locale packager124, and includes: a locale group ID602; one or more qualifying locales604included in the group; and an order of the fastest path608from a last GPS location206to the one or more locales604and optionally to an end node or point the visiting user wants to be after visiting the locales604.

FIG.7illustrates an embodiment of locale visit information700iin the locale visit database700generated by the visit collector130on a locale the visiting user visited, and may include: a locale ID702; a visiting user ID704that visited the locale702; an engagement time/date706the visiting user visited the locale702; and an engagement duration708of time the visiting user was engaged at the locale702.

FIG.8illustrates an embodiment of a training set800igenerated by the training set builder132that may include: a locale ID802; the machine learning input804, comprising information in input300, used by the locale attraction machine learning module120to generate a locale attraction score122; and an adjusted attraction score806that adjusts the generated locale attraction score122upwards or downwards based on whether the visiting user visited the locale802.

FIG.9illustrates an embodiment of operations performed by the locale analyzer116and the locale packager124to generate qualifying groups600of locales to provide to the visiting user computing device102to render augmented reality information on locales604in the group600in the augmented reality display112. Upon receiving (at block900) a locale request106for locales to visit, including the GPS location of the visiting user from the GPS module108and a user availability time period the user has to visit locales, the locale analyzer116generates (at block902) session information200for the request, including session ID202, user ID204, last GPS location of visiting user, user available time208comprising a timer initially set to the user availability time period in the request, and a transportation mode210of the visiting user, e.g., walking, automobile, ride sharing, public transportation, etc. If the user availability time period is in the future, then the user available time208, or timer, would be set to start decrementing at the beginning of the user availability time period in the request106. The locale analyzer116determines (at block904) qualifying locales118accessible during the user available time208.FIG.10provides further detailed operations for determining the qualifying locales118the user has time to visit within the user available time208.

The locale analyzer116determines (at block906), for each qualifying locale118, machine learning input300comprising locale feature values (FL)302for the qualifying locale from the locale information400ifor the qualifying locale; a recent change number of visitors (CNV)304from the field410in the locale information400ifor the locale; recent change in average user engagement duration (CED) at the locale from field412in the in the locale information400ifor the locale; and user feature values (FU)310from field504in the user information500i. This machine learning input300is provided to the locale attraction machine learning module120to produce the attraction score122for the qualifying locale118. The locale analyzer116determines (at block908) a set of qualifying locales118in a top percent or top number of attraction scores122.

The locale packager124forms (at block910) potential groups of one or more qualifying locales118in the set of top locale attraction scores122using a bin packing algorithm. For each potential group, the fastest path algorithm128determines (at block912) a fastest path to travel from the visiting user to the locales in the group and, optionally, to an end point the visiting user wants to be, to determine an order in which to visit the qualifying nodes in the potential group, when there are a plurality of qualifying nodes in the potential group. The locale packager124determines (at block914) group cumulative durations for the potential groups as sum of duration to travel the fastest path in a potential group plus the average engagement durations of users who have visited the locales in the potential group. Potential groups having group cumulative durations less than the current user available time208are indicated (at block916) as qualifying groups600of locales the user has time to visit.

With the embodiment ofFIG.9, machine learning is used to determine locale attraction scores to rank locales and form groups of locales that a user can visit within the remaining user available time to visit locales. Information on the locales of the qualifying groups are then sent to the visiting user computing device102to control an augmented reality generator110to generate augmented reality representations, in the augmented reality display112, of information on the locales in the qualifying group and an order in which the locales should be visited. This information will guide the user toward those locales the user is most likely to want to visit given the features of the visiting user and the locales, and thus increase the chance for positive serendipity moments the user will experience while visiting the locales under the guidance of the augmented reality display112.

In the embodiment ofFIGS.9and11, information on qualifying groups of qualifying locales is determined to send the information on the qualifying groups to the user computing device102. In an alternative embodiment, information on qualifying locales may be sent to the user computing device102to render augmented reality representations on the qualifying locales without determining qualifying groups of qualifying locales.

FIG.10illustrates an embodiment of operations performed by the locale analyzer116to determine qualifying locales at block904inFIG.9. Upon initiating (at block1000) the operations to determine qualifying locales118, the fastest path algorithm128determines (at block1002), based on the current user location206and transportation mode210, locale travel durations comprising times to reach the locale from the user current location206. The locale travel duration may further include a time to travel from the locale to a user designated end point. The locale analyzer116determines (at block1004) reachable locales comprising locales whose local travel duration is less than the user available time208, i.e., how much time remaining for the user to visit locales, which is constantly declining over time. A loop of operations is performed at blocks1006through1022for each reachable locale to determine whether it may be a qualifying locale. For a reachable locale, the locale analyzer116determines (at block1008) similar users that have visited the reachable locale i within a recent time period, e.g., last number of days, last week, last month, etc., that have similar user profiles to the visiting user based on a similarity algorithm, such as a data mining similarity algorithm. The user feature values504of the visiting user and the users that have recently visited the locale may be compared for similarity.

The visits by the similar users are grouped (at block1010) into different time range groups based on the time range that visits occurred, e.g., in last day, last number of days, last week, last month, etc. For each time range group, a determination is made (at block1012) of the average engagement duration of the similar users at the locale during the time range of the time range group. The average engagement duration for each time range group is weighted (at block1016) so more recent groups have a higher weighting factor than groups further in the past. This reflects that more recent visitors provide better prediction of interest and time spent than visitors in the more distant past. A cumulative locale visit duration is determined (at block1018) for the reachable locale as a sum of the average engagement duration plus the locale travel duration, to factor in both the time to travel to the locale and time the user will likely want to spend at the locale for an optimal experience. The reachable locale is indicated (at block1020) as a qualifying locale if the cumulative locale visit duration is less than the user available time208.

With the embodiment ofFIG.10, a qualifying locale is determined if the estimated time spent to travel to the locale and the estimated engagement duration at the locale is within the user available time. Further, to provide a more accurate average user engagement duration, only the engagement durations of users having similar profiles and feature values to the visiting user are considered, under the presumption that similar users to the visiting user are likely to better reflect the user engagement duration at the locale for this particular visiting user than time spent at the locale by users with different features, interests, and profiles. For instance, if similar users to the visiting user spend less time at the locale than users in general, then the locale may comprise a qualifying locale based on the engagement duration by similar users, but not be qualifying based on engagement duration by dissimilar users.

FIG.11illustrates interaction of the locale attraction server100and the visiting user computing device102to provide the visiting user information on locales the user is likely to want to be engaged during available time. Upon the locale analyzer116initiating a session200with a visiting user during user available time period to provide information on locales to visit to render in the augmented reality display112, the locale analyzer116performs the operations inFIG.9to select (at block1102) a qualifying group600of locales to visit. The selected qualifying group is sent (at block1104) to the visiting user computing device102with an order to visit the locales on the fastest travel path and information on the locales in the group600.

Upon the locale attraction client104at the visiting user computing device102receiving (at block1106) the group of locales and information on the locales, the augmented reality generator110is called (at block1108) to render the information on the locales in the group in the augmented reality display112showing the path/order to travel to the locales to draw the visiting user to the locale in the real world. Augmented reality representations of information on locales may provide information on what is currently happening at the locale, number of visitors, services/products offered, etc.

If the visiting user has decided not to visit the locales having augmented reality information rendered in the augmented reality display112, then the visiting user may request (at block1110), during the user available time, another group of locales that indicates locales visited and not visited with the user engagement duration of the visited locales. This request is sent (at block1112) to the locale attraction server100. Upon receiving (at block1114) the request at block1112, which also may include the current GPS location, the locale analyzer116proceeds (at block1116) toFIG.9, block900to calculate a new set of qualifying locales, excluding locales visited and not visited in all past sent groups during the session200, based on last GPS location206and user available time208left in the session200, and proceeds to block1104to send the new group. In this way, over time within the user available time208, the qualifying locales are re-determined because there may not be sufficient time to visit and engage certain locales that previously qualified with more time remaining.

After determining (at block1116) the new qualifying locales, for each locale indicated in the response in the last sent group as not visited, the locale attraction scores122for those locales are adjusted (at block1118) downward to reflect less user interest/attraction for those not visited locales. For each visited locale, the locale analyzer116adjusts (at block1120) the locale attraction score upward to reflect the continued attraction/interest in the locale. For each adjusted attraction score, the training set builder132generates (at block1122) a new training set800iindicating machine learning input804used to generate the pre-adjusted locale attraction score122and the adjusted locale attraction score based on whether the user recently visited or did not visit the locale for which augmented reality information was rendered in the augmented reality display112. The generated training sets800iare stored (at block1124) in the training set database800.

For each locale visited, the locale analyzer116generates (at block1126) locale visit information700iindicating the locale visited702, visiting user704, time/day visited706, and user engagement duration708at the locale and stores the generated locale visit information700iin locale visit database700. The session information200is updated (at block1128) to indicate the locales visited in field212and to indicate the locales, for which augmented reality information was generated, as not visited214.

Upon the locale attraction client104detecting (at block1130) that the user visited a locale during the available time, a message is sent (at block1132) to the local attraction server100indicating locale(s) visited and user engagement duration at the locales visited. The locale analyzer116upon receiving the information sent, may perform the operations at block1120et seq to process the information on a locale that was visited.

With the embodiment ofFIG.11, the user computing device102interacts with the locale attraction server100to provide information on locales visited and locale engagement duration. The locale attraction server100may use this information to generate training sets to train the locale attraction machine learning module120to improve the machine learning predictability about user attraction to a locale, and update information to determine further groups of locales if requested during the user available time.

FIG.12illustrates an embodiment of operations performed by the machine learning trainer134to train the locale attraction machine learning module120with the generated training sets800i. Upon initiating (at block1200) a training operation, the machine learning trainer134selects (at block1202) training set instances800ito generate a training input set comprising the machine learning input804from selected training set instances800i, in a matrix or matrices, and a training output set comprising the corresponding adjusted attraction scores806, which may comprise a vector. The machine learning trainer134trains the locale attraction machine learning module120with the training input set to produce the training output set to update parameter values, biases and weights for the nodes in the locale attraction machine learning module120.

Computing environment1300contains an example of an environment for the execution of at least some of the computer code1301involved in performing the inventive methods, such as the locale analyzer116, locale attraction machine learning module120, locale packager124, fastest path algorithm128, visit collector130, training set builder132, and machine learning trainer134(FIG.1).

In addition to block1301, computing environment1300includes, for example, computer1301, wide area network (WAN)1302, end user device (EUD)1303, remote server1304, public cloud1305, and private cloud1306. In this embodiment, computer1301includes processor set1310(including processing circuitry1320and cache1321), communication fabric1311, volatile memory1312, persistent storage1313(including operating system1322and block1301, as identified above), peripheral device set1314(including user interface (UI) device set1323, storage1324, and Internet of Things (IoT) sensor set1325), and network module1315. Remote server1304includes remote database1330. Public cloud1305includes gateway1340, cloud orchestration module1341, host physical machine set1342, virtual machine set1343, and container set1344.

PROCESSOR SET1310includes one, or more, computer processors of any type now known or to be developed in the future. Processing circuitry1320may be distributed over multiple packages, for example, multiple, coordinated integrated circuit chips. Processing circuitry1320may implement multiple processor threads and/or multiple processor cores. Cache1321is memory that is located in the processor chip package(s) and is typically used for data or code that should be available for rapid access by the threads or cores running on processor set1310. Cache memories are typically organized into multiple levels depending upon relative proximity to the processing circuitry. Alternatively, some, or all, of the cache for the processor set may be located “off chip.” In some computing environments, processor set1310may be designed for working with qubits and performing quantum computing.

Computer readable program instructions are typically loaded onto computer1301to cause a series of operational steps to be performed by processor set1310of computer1301and thereby effect a computer-implemented method, such that the instructions thus executed will instantiate the methods specified in flowcharts and/or narrative descriptions of computer-implemented methods included in this document (collectively referred to as “the inventive methods”). These computer readable program instructions are stored in various types of computer readable storage media, such as cache1321and the other storage media discussed below. The program instructions, and associated data, are accessed by processor set1310to control and direct performance of the inventive methods. In computing environment1300, at least some of the instructions for performing the inventive methods may be stored in persistent storage1313.

VOLATILE MEMORY1312is any type of volatile memory now known or to be developed in the future. Examples include dynamic type random access memory (RAM) or static type RAM. Typically, volatile memory1312is characterized by random access, but this is not required unless affirmatively indicated. In computer1301, the volatile memory1312is located in a single package and is internal to computer1301, but, alternatively or additionally, the volatile memory may be distributed over multiple packages and/or located externally with respect to computer1301.

PERSISTENT STORAGE1313is any form of non-volatile storage for computers that is now known or to be developed in the future. The non-volatility of this storage means that the stored data is maintained regardless of whether power is being supplied to computer1301and/or directly to persistent storage1313. Persistent storage1313may be a read only memory (ROM), but typically at least a portion of the persistent storage allows writing of data, deletion of data and re-writing of data. Some familiar forms of persistent storage include magnetic disks and solid state storage devices. Operating system1322may take several forms, such as various known proprietary operating systems or open source Portable Operating System Interface-type operating systems that employ a kernel. The code included in block1301typically includes at least some of the computer code involved in performing the inventive methods.

PERIPHERAL DEVICE SET1314includes the set of peripheral devices of computer1301. Data communication connections between the peripheral devices and the other components of computer1301may be implemented in various ways, such as Bluetooth connections, Near-Field Communication (NFC) connections, connections made by cables (such as universal serial bus (USB) type cables), insertion-type connections (for example, secure digital (SD) card), connections made through local area communication networks and even connections made through wide area networks such as the internet. In various embodiments, UI device set1323may include components such as a display screen, speaker, microphone, wearable devices (such as goggles and smart watches), keyboard, mouse, printer, touchpad, game controllers, and haptic devices. Storage1324is external storage, such as an external hard drive, or insertable storage, such as an SD card. Storage1324may be persistent and/or volatile. In some embodiments, storage1324may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer1301is required to have a large amount of storage (for example, where computer1301locally stores and manages a large database) then this storage may be provided by peripheral storage devices designed for storing very large amounts of data, such as a storage area network (SAN) that is shared by multiple, geographically distributed computers. IoT sensor set1325is made up of sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer and another sensor may be a motion detector.

NETWORK MODULE1315is the collection of computer software, hardware, and firmware that allows computer1301to communicate with other computers through WAN1302. Network module1315may include hardware, such as modems or Wi-Fi signal transceivers, software for packetizing and/or de-packetizing data for communication network transmission, and/or web browser software for communicating data over the internet. In some embodiments, network control functions and network forwarding functions of network module1315are performed on the same physical hardware device. In other embodiments (for example, embodiments that utilize software-defined networking (SDN)), the control functions and the forwarding functions of network module1315are performed on physically separate devices, such that the control functions manage several different network hardware devices. Computer readable program instructions for performing the inventive methods can typically be downloaded to computer1301from an external computer or external storage device through a network adapter card or network interface included in network module1315.

END USER DEVICE (EUD)1303is any computer system that is used and controlled by an end user (for example, a customer of an enterprise that operates computer1301), and may take any of the forms discussed above in connection with computer1301. EUD1303typically receives helpful and useful data from the operations of computer1301. For example, in a hypothetical case where computer1301is designed to provide a recommendation to an end user, this recommendation would typically be communicated from network module1315of computer1301through WAN1302to EUD1303. In this way, EUD1303can display, or otherwise present, the recommendation to an end user. In some embodiments, EUD1303may be a client device, such as thin client, heavy client, mainframe computer, desktop computer and so on.

REMOTE SERVER1304is any computer system that serves at least some data and/or functionality to computer1301. Remote server1304may be controlled and used by the same entity that operates computer1301. Remote server1304represents the machine(s) that collect and store helpful and useful data for use by other computers, such as computer1301. For example, in a hypothetical case where computer1301is designed and programmed to provide a recommendation based on historical data, then this historical data may be provided to computer1301from remote database1330of remote server1304.

PUBLIC CLOUD1305is any computer system available for use by multiple entities that provides on-demand availability of computer system resources and/or other computer capabilities, especially data storage (cloud storage) and computing power, without direct active management by the user. Cloud computing typically leverages sharing of resources to achieve coherence and economies of scale. The direct and active management of the computing resources of public cloud1305is performed by the computer hardware and/or software of cloud orchestration module1341. The computing resources provided by public cloud1305are typically implemented by virtual computing environments that run on various computers making up the computers of host physical machine set1342, which is the universe of physical computers in and/or available to public cloud1305. The virtual computing environments (VCEs) typically take the form of virtual machines from virtual machine set1343and/or containers from container set1344. It is understood that these VCEs may be stored as images and may be transferred among and between the various physical machine hosts, either as images or after instantiation of the VCE. Cloud orchestration module1341manages the transfer and storage of images, deploys new instantiations of VCEs and manages active instantiations of VCE deployments. Gateway1340is the collection of computer software, hardware, and firmware that allows public cloud1305to communicate through WAN1302.

PRIVATE CLOUD1306is similar to public cloud1305, except that the computing resources are only available for use by a single enterprise. While private cloud1306is depicted as being in communication with WAN1302, in other embodiments a private cloud may be disconnected from the internet entirely and only accessible through a local/private network. A hybrid cloud is a composition of multiple clouds of different types (for example, private, community or public cloud types), often respectively implemented by different vendors. Each of the multiple clouds remains a separate and discrete entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technology that enables orchestration, management, and/or data/application portability between the multiple constituent clouds. In this embodiment, public cloud1305and private cloud1306are both part of a larger hybrid cloud.

The letter designators, such as i, is used to designate a number of instances of an element may indicate a variable number of instances of that element when used with the same or different elements.