Patent Publication Number: US-11645580-B2

Title: Robust reinforcement learning in personalized content prediction

Description:
TECHNICAL FIELD 
     This application relates generally to personalized content prediction and, more particularly, to selection of content elements using trained selection models. 
     BACKGROUND 
     Various network interfaces, such as e-commerce interfaces, are configured to present one or more interface pages including a plurality of containers (or slots). A plurality of content elements may be available for each container. For example, in e-commerce environments, a carousel module may have multiple options for filling the open spots on the carousel. 
     Current interface systems are configured to select elements for filling open containers or slots in an interface based on short-term reward mechanisms. For example, in some embodiments, an element having the highest click-through rate may be selected for presentation to a user. Short-term reward mechanisms and systems are prone to randomness and noise, and fail to take into account long-term or changing user preferences. 
     SUMMARY 
     In various embodiments, a system for content selection and presentation is disclosed. The system includes a computing device configured to receive a plurality of content elements configured for presentation in at least one content container and select one of the plurality of content elements for presentation in the at least one content container. The one of the plurality of content elements is selected by a trained selection model configured to use Thompson sampling. The computing device generates an interface including the selected one of the plurality of content elements. 
     In various embodiments, a non-transitory computer readable medium having instructions stored thereon is disclosed. The instructions, when executed by a processor cause a device to perform operations including receiving a request for an interface. The request includes a user persona. The device further performs the operation of selecting at least one of a plurality of content elements for inclusion in the interface. The at least one of the plurality of content elements is selected using Thompson sampling. The device further performs the operation of generating an interface including the selected at least one of the plurality of content elements. 
     In various embodiments, a computer-implemented method is disclosed. The method includes steps of receiving a plurality of content elements configured for presentation in at least one content container and selecting one of the plurality of content elements for presentation in the at least one content container. The one of the plurality of content elements is selected by a trained selection model configured to use Thompson sampling. An interface including the selected one of the plurality of content elements is generated. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The features and advantages of the present invention will be more fully disclosed in, or rendered obvious by the following detailed description of the preferred embodiments, which are to be considered together with the accompanying drawings wherein like numbers refer to like parts and further wherein: 
         FIG.  1    illustrates a block diagram of a computer system, in accordance with some embodiments. 
         FIG.  2    illustrates a network configured to provide an interface including one or more components selected using a trained content selection model configured to implement Thompson sampling, in accordance with some embodiments. 
         FIG.  3    illustrates a method of generating a trained content selection model, in accordance with some embodiments. 
         FIG.  4    illustrates a process of training a content selection model according to the method of  FIG.  3   , in accordance with some embodiments. 
         FIG.  5    illustrates a content selection process for selecting content for presentation to a user using a trained content selection model, in accordance with some embodiments. 
     
    
    
     DETAILED DESCRIPTION 
     The description of the preferred embodiments is intended to be read in connection with the accompanying drawings, which are to be considered part of the entire written description of this invention. The drawing figures are not necessarily to scale and certain features of the invention may be shown exaggerated in scale or in somewhat schematic form in the interest of clarity and conciseness. In this description, relative terms such as “horizontal,” “vertical,” “up,” “down,” “top,” “bottom,” as well as derivatives thereof (e.g., “horizontally,” “downwardly,” “upwardly,” etc.) should be construed to refer to the orientation as then described or as shown in the drawing figure under discussion. These relative terms are for convenience of description and normally are not intended to require a particular orientation. Terms including “inwardly” versus “outwardly,” “longitudinal” versus “lateral” and the like are to be interpreted relative to one another or relative to an axis of elongation, or an axis or center of rotation, as appropriate. Terms concerning attachments, coupling and the like, such as “connected” and “interconnected,” refer to a relationship wherein structures are secured or attached to one another either directly or indirectly through intervening structures, as well as both moveable or rigid attachments or relationships, unless expressly described otherwise. The term “operatively coupled” is such an attachment, coupling, or connection that allows the pertinent structures to operate as intended by virtue of that relationship. In the claims, means-plus-function clauses, if used, are intended to cover structures described, suggested, or rendered obvious by the written description or drawings for performing the recited function, including not only structure equivalents but also equivalent structures. 
       FIG.  1    illustrates a computer system configured to implement one or more processes, in accordance with some embodiments. The system  2  is a representative device and may comprise a processor subsystem  4 , an input/output subsystem  6 , a memory subsystem  8 , a communications interface  10 , and a system bus  12 . In some embodiments, one or more than one of the system  2  components may be combined or omitted such as, for example, not including an input/output subsystem  6 . In some embodiments, the system  2  may comprise other components not combined or comprised in those shown in  FIG.  1   . For example, the system  2  may also include, for example, a power subsystem. In other embodiments, the system  2  may include several instances of the components shown in  FIG.  1   . For example, the system  2  may include multiple memory subsystems  8 . For the sake of conciseness and clarity, and not limitation, one of each of the components is shown in  FIG.  1   . 
     The processor subsystem  4  may include any processing circuitry operative to control the operations and performance of the system  2 . In various aspects, the processor subsystem  4  may be implemented as a general purpose processor, a chip multiprocessor (CMP), a dedicated processor, an embedded processor, a digital signal processor (DSP), a network processor, an input/output (I/O) processor, a media access control (MAC) processor, a radio baseband processor, a co-processor, a microprocessor such as a complex instruction set computer (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, and/or a very long instruction word (VLIW) microprocessor, or other processing device. The processor subsystem  4  also may be implemented by a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device (PLD), and so forth. 
     In various aspects, the processor subsystem  4  may be arranged to run an operating system (OS) and various applications. Examples of an OS comprise, for example, operating systems generally known under the trade name of Apple OS, Microsoft Windows OS, Android OS, Linux OS, and any other proprietary or open source OS. Examples of applications comprise, for example, network applications, local applications, data input/output applications, user interaction applications, etc. 
     In some embodiments, the system  2  may comprise a system bus  12  that couples various system components including the processing subsystem  4 , the input/output subsystem  6 , and the memory subsystem  8 . The system bus  12  can be any of several types of bus structure(s) including a memory bus or memory controller, a peripheral bus or external bus, and/or a local bus using any variety of available bus architectures including, but not limited to, 9-bit bus, Industrial Standard Architecture (ISA), Micro-Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect Card International Association Bus (PCMCIA), Small Computers Interface (SCSI) or other proprietary bus, or any custom bus suitable for computing device applications. 
     In some embodiments, the input/output subsystem  6  may include any suitable mechanism or component to enable a user to provide input to system  2  and the system  2  to provide output to the user. For example, the input/output subsystem  6  may include any suitable input mechanism, including but not limited to, a button, keypad, keyboard, click wheel, touch screen, motion sensor, microphone, camera, etc. 
     In some embodiments, the input/output subsystem  6  may include a visual peripheral output device for providing a display visible to the user. For example, the visual peripheral output device may include a screen such as, for example, a Liquid Crystal Display (LCD) screen. As another example, the visual peripheral output device may include a movable display or projecting system for providing a display of content on a surface remote from the system  2 . In some embodiments, the visual peripheral output device can include a coder/decoder, also known as Codecs, to convert digital media data into analog signals. For example, the visual peripheral output device may include video Codecs, audio Codecs, or any other suitable type of Codec. 
     The visual peripheral output device may include display drivers, circuitry for driving display drivers, or both. The visual peripheral output device may be operative to display content under the direction of the processor subsystem  6 . For example, the visual peripheral output device may be able to play media playback information, application screens for application implemented on the system  2 , information regarding ongoing communications operations, information regarding incoming communications requests, or device operation screens, to name only a few. 
     In some embodiments, the communications interface  10  may include any suitable hardware, software, or combination of hardware and software that is capable of coupling the system  2  to one or more networks and/or additional devices. The communications interface  10  may be arranged to operate with any suitable technique for controlling information signals using a desired set of communications protocols, services or operating procedures. The communications interface  10  may comprise the appropriate physical connectors to connect with a corresponding communications medium, whether wired or wireless. 
     Vehicles of communication comprise a network. In various aspects, the network may comprise local area networks (LAN) as well as wide area networks (WAN) including without limitation Internet, wired channels, wireless channels, communication devices including telephones, computers, wire, radio, optical or other electromagnetic channels, and combinations thereof, including other devices and/or components capable of/associated with communicating data. For example, the communication environments comprise in-body communications, various devices, and various modes of communications such as wireless communications, wired communications, and combinations of the same. 
     Wireless communication modes comprise any mode of communication between points (e.g., nodes) that utilize, at least in part, wireless technology including various protocols and combinations of protocols associated with wireless transmission, data, and devices. The points comprise, for example, wireless devices such as wireless headsets, audio and multimedia devices and equipment, such as audio players and multimedia players, telephones, including mobile telephones and cordless telephones, and computers and computer-related devices and components, such as printers, network-connected machinery, and/or any other suitable device or third-party device. 
     Wired communication modes comprise any mode of communication between points that utilize wired technology including various protocols and combinations of protocols associated with wired transmission, data, and devices. The points comprise, for example, devices such as audio and multimedia devices and equipment, such as audio players and multimedia players, telephones, including mobile telephones and cordless telephones, and computers and computer-related devices and components, such as printers, network-connected machinery, and/or any other suitable device or third-party device. In various implementations, the wired communication modules may communicate in accordance with a number of wired protocols. Examples of wired protocols may comprise Universal Serial Bus (USB) communication, RS-232, RS-422, RS-423, RS-485 serial protocols, FireWire, Ethernet, Fibre Channel, MIDI, ATA, Serial ATA, PCI Express, T-1 (and variants), Industry Standard Architecture (ISA) parallel communication, Small Computer System Interface (SCSI) communication, or Peripheral Component Interconnect (PCI) communication, to name only a few examples. 
     Accordingly, in various aspects, the communications interface  10  may comprise one or more interfaces such as, for example, a wireless communications interface, a wired communications interface, a network interface, a transmit interface, a receive interface, a media interface, a system interface, a component interface, a switching interface, a chip interface, a controller, and so forth. When implemented by a wireless device or within wireless system, for example, the communications interface  10  may comprise a wireless interface comprising one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, and so forth. 
     In various aspects, the communications interface  10  may provide data communications functionality in accordance with a number of protocols. Examples of protocols may comprise various wireless local area network (WLAN) protocols, including the Institute of Electrical and Electronics Engineers (IEEE) 802.xx series of protocols, such as IEEE 802.11a/b/g/n, IEEE 802.16, IEEE 802.20, and so forth. Other examples of wireless protocols may comprise various wireless wide area network (WWAN) protocols, such as GSM cellular radiotelephone system protocols with GPRS, CDMA cellular radiotelephone communication systems with 1×RTT, EDGE systems, EV-DO systems, EV-DV systems, HSDPA systems, and so forth. Further examples of wireless protocols may comprise wireless personal area network (PAN) protocols, such as an Infrared protocol, a protocol from the Bluetooth Special Interest Group (SIG) series of protocols (e.g., Bluetooth Specification versions 5.0, 6, 7, legacy Bluetooth protocols, etc.) as well as one or more Bluetooth Profiles, and so forth. Yet another example of wireless protocols may comprise near-field communication techniques and protocols, such as electro-magnetic induction (EMI) techniques. An example of EMI techniques may comprise passive or active radio-frequency identification (RFID) protocols and devices. Other suitable protocols may comprise Ultra Wide Band (UWB), Digital Office (DO), Digital Home, Trusted Platform Module (TPM), ZigBee, and so forth. 
     In some embodiments, at least one non-transitory computer-readable storage medium is provided having computer-executable instructions embodied thereon, wherein, when executed by at least one processor, the computer-executable instructions cause the at least one processor to perform embodiments of the methods described herein. This computer-readable storage medium can be embodied in memory subsystem  8 . 
     In some embodiments, the memory subsystem  8  may comprise any machine-readable or computer-readable media capable of storing data, including both volatile/non-volatile memory and removable/non-removable memory. The memory subsystem  8  may comprise at least one non-volatile memory unit. The non-volatile memory unit is capable of storing one or more software programs. The software programs may contain, for example, applications, user data, device data, and/or configuration data, or combinations therefore, to name only a few. The software programs may contain instructions executable by the various components of the system  2 . 
     In various aspects, the memory subsystem  8  may comprise any machine-readable or computer-readable media capable of storing data, including both volatile/non-volatile memory and removable/non-removable memory. For example, memory may comprise read-only memory (ROM), random-access memory (RAM), dynamic RAM (DRAM), Double-Data-Rate DRAM (DDR-RAM), synchronous DRAM (SDRAM), static RAM (SRAM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory (e.g., NOR or NAND flash memory), content addressable memory (CAM), polymer memory (e.g., ferroelectric polymer memory), phase-change memory (e.g., ovonic memory), ferroelectric memory, silicon-oxide-nitride-oxide-silicon (SONOS) memory, disk memory (e.g., floppy disk, hard drive, optical disk, magnetic disk), or card (e.g., magnetic card, optical card), or any other type of media suitable for storing information. 
     In one embodiment, the memory subsystem  8  may contain an instruction set, in the form of a file for executing various methods, such as methods including content selection using a trained content selection model, as described herein. The instruction set may be stored in any acceptable form of machine readable instructions, including source code or various appropriate programming languages. Some examples of programming languages that may be used to store the instruction set comprise, but are not limited to: Java, C, C++, C#, Python, Objective-C, Visual Basic, or .NET programming In some embodiments a compiler or interpreter is comprised to convert the instruction set into machine executable code for execution by the processing subsystem  4 . 
       FIG.  2    illustrates a network  20  configured to provide a network environment including one or more components having content selected using a trained content selection network, in accordance with some embodiments. The network  20  includes one or more user systems  22   a - 22   c , a network interface system  24 , a content selection system  26 , a model training system  28 , and/or any other suitable system. Each of the systems  22   a - 28  can include a computer system, such as the computer system  2  described above in conjunction with  FIG.  1   . It will be appreciated that each of the systems  22   a - 28  can include generic systems and/or special purpose systems, and are within the scope of this disclosure. 
     Each of the systems  22   a - 28  are configured to exchange data over one or more networks, such as, for example, network  30 . In some embodiments, the one or more user devices  22   a - 22   c  are configured to communicate with the network interface system  24 , which is configured to present one or more interfaces, such as an e-commerce interface. The presented interface includes one or more components, such as, for example, one or more carousels, one or more dynamic components, one or more web components, etc. At least one of the components may be filled by a plurality of content elements. In some embodiments, content elements for at least one of the interface components is selected by the content selection system  26 . 
     In some embodiments, the content selection system  26  is configured to select a content element for filling an open slot using a trained content selection network. As discussed in greater detail below, the trained neural network may be configured to balance exploiting content elements having the highest reward (e.g., select from content options having the highest click-rate, purchase rate, etc.), referred to as “exploit” mechanisms, and exploring new content elements to collect information (e.g., selecting new content or content having low information with respect to reward values), referred to as “explore” mechanisms. For example, in some embodiments, a predetermined number of potential content elements may be stored in a database, such as database  32 . A first subset of the potential content elements may have a robust known reward value, such as an actual reward value or estimated reward value, while a second subset of the potential content elements may have limited or no reward values. The trained content selection model is configured to solve the “multi-arm bandit” problem presented by attempting to exploit content elements with the highest reward rate while exploring the potential reward rates of content the having less information. In some embodiments, content elements may be stored in a database  32  and retrieved by the content selection system  26 . For example, in some embodiments, a set of potential content elements may be retrieved from the database  32 . The trained content selection model selects presentation content elements from the set of one or more potential content elements. In some embodiments, the set of potential content elements is selected based on user criteria, such as, for example, a user persona. 
     As discussed in greater detail below, the trained neural network includes a “contextual” element configured to consider the context of a user interacting with the presented interface. For example, in some embodiments, a user may be sorted into one or more predetermined “personas” corresponding to a group of customers having common characteristics. In some embodiments, the “personas” may include, but are not limited to, life-stage personas (e.g., single, married, family with a baby, etc.), department personas (e.g., technology, fashion, sports, etc.), and/or any other suitable personas. In some embodiments, a plurality of trained content selection models are provided to the content selection system  26 . One of the plurality of trained content selection models may be used to perform content selection based on the persona associated with the user interacting with the interface. 
     In some embodiments, the content selection system  26  receives a trained content selection model from a model training system  28 . As discussed below, the model training system  28  is configured to implement a machine learning process using a reinforcement learning mechanism, such as, for example, an explore/exploit mechanism. In some embodiments, the model training system  28  is configured to iteratively modify one or more machine learning (e.g., artificial intelligence, neural network, etc.) models based on additional training data, modified rewards values, and/or other data received from additional systems, such as the network interface system  24  and/or the content selection system  26 . In some embodiments, the model training system  28  implements a state-action-reward-state-action (SARSA) process modified to use Thomspon sampling, as discussed in greater detail below. 
       FIG.  3    is a flowchart illustrating a method  100  of generating a trained content selection model, in accordance with some embodiments.  FIG.  4    is a process flow  150  illustrating various steps of the method  100  illustrated in  FIG.  3   , in accordance with some embodiments. At step  102 , a set of training data  152  is received. The set of training data  152  includes a plurality of user impressions for at least one component in an interface (e.g., a plurality of user interactions with a component and one or more potential content elements for the component). The impression data may include a success or reward indication, such as, for example, a click-through rate, user return rate, historic purchase data, and/or any other suitable reward data. In some embodiments, the set of training data  152  is limited to data obtained from users having a predetermined persona. 
     In some embodiments, the set of training data  152  is used to iteratively train a content selection model  154  to select an action a based on a total reward value, Q. For example, in some embodiments, at least a portion of the set of training data  152  is provided to an untrained content selection model (e.g., neural network). The content selection model selects an action based on a reward value Q. In some embodiments, the value Q is given by the equation:
 
 Q ( S   j   ,a   i )= r ( S   j   ,a   i )+ R ( S   j   ,a   i )
 
where r is the expected reward from a user in a current session (e.g., short-term reward) where S j  is the persona of the user and a i  is the action selected and R is the expected total discounted rewards from the user in future sessions (e.g., long-term reward).
 
     In some embodiments, the short-term reward, r, and the long-term reward, R, may be estimated based on the expected value of a k-th impression:
 
 r ( S   j   ,a   i )= E [ r   k ( S   j   ,a   i )]
 
 R ( S   j   ,a   i )= E [ R   k ( S   j   ,a   i )]
 
The expected value of R may be estimated by the equation:
 
               E   [       R   k     (       S   j     ,     a   i       )     ]     =       ∑     g   =   1       m   +   1           ∑     h   =   1     n         P   ⁡   (       S   g     ,     a   h       )     ×     γ   ⁡   (     t   k     )     ×     Q   ⁡   (       S   g     ,     a   h       )                 
where γ(•) is a long-term reward discount function, t k  is the time until the next session for the user interacting with the k-th impression and S m+1  is the state where a user does not return to the interface. The reward value Q for a user who does not return is: Q(S m+1 ,a i )=0∀i=1, 2, . . . , n, since no rewards are provided for a user that does not interact with an interface. In some embodiments, it may also be assumed that γ(t)=0∀t≥T, where T is a time limit for the specific user to return to and interact with the interface. Based on the foregoing assumptions, the rewards at S m+1  to are irrelevant and can be ignored.
 
     In some embodiments, a posterior distribution technique, such as a state-action-reward-state-action (SARSA) algorithm modified to use Thompson sampling, is applied to the set of training data  152 . At step  104 , the value of Q(•,•) is initialized as a normal distribution. The initial parameters of the normalized distribution may be set arbitrarily, based on empirical estimates, and/or based on prior values of Q for prior trained models. For example, the value of Q(•,•) a posterior (e.g., normalized) distribution by assuming that, for a given action a i , the short-term reward and the state of the user in the next session are independent, the short term rewards of different sessions are independent, and the action taken in a session is independent of short-term rewards of previous sessions. Based on these assumptions,
 
 r ( S   j   ,a   i )   R ( S   j   ,a   i )∀ j= 1,2, . . . ,( m+ 1), i= 1,2, . . . , n  
 
 k= 1,2, . . . , K ( S   j   ,a   i )
 
where K(S j ,a i ) is the total number of impressions with context S j  and action taken a i  in the set of training data  152  (e.g., during a time period covered by the set of training date  152 ). Central limit theory provides:
 
     
       
         
           
             
               
                 
                   
                     ∑ 
                     
                       k 
                       = 
                       1 
                     
                     
                       K 
                       ⁡ 
                       ( 
                       
                         
                           S 
                           j 
                         
                         , 
                         
                           a 
                           i 
                         
                       
                       ) 
                     
                   
                   
                     
                       r 
                       k 
                     
                     ( 
                     
                       
                         S 
                         j 
                       
                       , 
                       
                         a 
                         i 
                       
                     
                     ) 
                   
                 
                 
                   K 
                   ⁡ 
                   ( 
                   
                     
                       S 
                       j 
                     
                     , 
                     
                       a 
                       i 
                     
                   
                   ) 
                 
               
               + 
               
                 
                   
                     ∑ 
                     
                       k 
                       = 
                       1 
                     
                     
                       K 
                       ⁡ 
                       ( 
                       
                         
                           S 
                           j 
                         
                         , 
                         
                           a 
                           i 
                         
                       
                       ) 
                     
                   
                   
                     
                       R 
                       k 
                     
                     ( 
                     
                       
                         S 
                         j 
                       
                       , 
                       
                         a 
                         i 
                       
                     
                     ) 
                   
                 
                 
                   K 
                   ⁡ 
                   ( 
                   
                     
                       S 
                       j 
                     
                     , 
                     
                       a 
                       i 
                     
                   
                   ) 
                 
               
             
             ∼ 
             
               N 
               ⁡ 
               ( 
               
                 
                   Q 
                   ⁡ 
                   ( 
                   
                     
                       S 
                       j 
                     
                     , 
                     
                       a 
                       i 
                     
                   
                   ) 
                 
                 , 
                 
                   
                     
                       σ 
                       2 
                     
                     ( 
                     
                       
                         S 
                         j 
                       
                       , 
                       
                         a 
                         i 
                       
                     
                     ) 
                   
                   
                     K 
                     ⁡ 
                     ( 
                     
                       
                         S 
                         j 
                       
                       , 
                       
                         a 
                         i 
                       
                     
                     ) 
                   
                 
               
               ) 
             
           
         
       
     
     In some embodiments, it can be assumed that the prior distribution is given by:
 
 Q ( S   j   ,a   i )˜ (μ 0 ( S   j   ,a   i ),σ 0   2 ( S   j   ,a   i ))
 
Combining the prior and empirical data, the posterior distribution of Q(S j ,a i ) is:
 
             	         Q   ⁡   (       S   j     ,     a   i       )     ∼     𝒩   ⁡   (         μ   post     (       S   j     ,     a   i       )     ,       σ   post   2     (       S   j     ,     a   i       )       )       ,   where                     μ   post     (       S   j     ,     a   i       )     =           w   1         w   1     +     w   2         ⁢       μ   0     (       S   j     ,     a   i       )       +         w   2         w   1     +     w   2         [           ∑     k   =   1       K   ⁡   (       S   j     ,     a   i       )           r   k     (       S   j     ,     a   i       )         K   ⁡   (       S   j     ,     a   i       )       +         ∑     k   =   1       K   ⁡   (       S   j     ,     a   i       )           R   k     (       S   j     ,     a   i       )         K   ⁡   (       S   j     ,     a   i       )         ]                   	         σ   post   2     (       S   j     ,     a   i       )     =       (       w   1     +     w   2       )       -   1                     	       w   1     =     1       σ   0   2     (       S   j     ,     a   i       )                     	       w   2     =       K   ⁡   (       S   j     ,     a   i       )         σ   2     (       S   j     ,     a   i       )               
In some embodiments, σ 2 (S j ,a i ) is replaced with an empirical estimate, {circumflex over (σ)} 2 (S j ,a i ).
 
     In some embodiments, the long-term reward R k  is not actually observed and estimate may be used instead: 
                 R   k     (       S   j     ,     a   i       )     =       ∑     g   =   1       m   +   1           ∑     h   =   1     n         1     S   g       ×     1     a   h       ×     γ   ⁡   (     t   k     )     ×       Q   ^     (       S   g     ,     a   h       )                 
where S g  is the user context in the next session, a h  is the action taken, and {circumflex over (Q)} is a sample taken from the existing posterior distribution. Based on prior assumptions, the only term remaining to be calculated is {circumflex over (σ)} 2 (S j ,a i ), where:
 
             	           σ   ^     2     (       S   j     ,     a   i       )     =       v   ⁢     a   ^     ⁢   r   ⁢         (       r   k     (       S   j     ,     a   i       )     )       +     v   ⁢     a   ^     ⁢     r   (       R   k     (       S   j     ,     a   i       )     )                       	       v   ⁢     a   ^     ⁢     r   (       r   k     (       S   j     ,     a   i       )     )       =           ∑     k   =   1       K   ⁡   (       S   j     ,     a   i       )           r   k   2     (       S   j     ,     a   i       )         K   ⁡   (       S   j     ,     a   i       )       -       (         ∑     k   =   1       K   ⁡   (       S   j     ,     a   i       )           r   k     (       S   j     ,     a   i       )         K   ⁡   (       S   j     ,     a   i       )       )     2                       v   ⁢     a   ^     ⁢     r   (       R   k     (       S   j     ,     a   i       )     )       =           E   ^         S   g     ,     a   h         [     v   ⁢     a   ^     ⁢     r   (       γ   (     t   k     )     ⁢       Q   ^     (       S   g     ,     a   h       )       )       ]     +     v   ⁢     a   ^     ⁢       r       S   g     ,     a   h         [       E   ^     (       γ   ⁡   (     t   k     )     ⁢       Q   ^     (       S   g     ,     a   h       )       )     ]                     	         E   ^     (       γ   ⁡   (     t   k     )     ⁢       Q   ^     (       S   g     ,     a   h       )       )     =         ∑     l   =   1       L   ⁡   (       S   j     ,       a   i     ⁢     S   g       ,     a   h       )           γ   ⁡   (     t   l     )     ⁢       Q   ^     (       S   g     ,     a   h       )           L   ⁡   (       S   j     ,     a   i     ,     S   g     ,     a   h       )                       v   ⁢     a   ^     ⁢     r   (       γ   (     t   k     )     ⁢       Q   ^     (       S   g     ,     a   h       )       )       =           ∑     l   =   1       L   ⁡   (       S   j     ,       a   i     ⁢     S   g       ,     a   h       )           (       γ   ⁡   (     t   l     )     ⁢       Q   ^     (       S   g     ,     a   h       )       )     2         L   ⁡   (       S   j     ,     a   i     ,     S   g     ,     a   h       )       -       [       E   ^     (       γ   ⁡   (     t   k     )     ⁢       Q   ^     (       S   g     ,     a   h       )       )     ]     2                         E   ^         S   g     ,     a   h         [     v   ⁢     a   ^     ⁢     r   (       γ   (     t   k     )     ⁢       Q   ^     (       S   g     ,     a   h       )       )       ]     =       ∑     g   =   1       m   +   1           ∑     h   =   1     n           L   ⁡   (       S   j     ,     a   i     ,     S   g     ,     a   h       )       K   ⁡   (       S   j     ,     a   i       )       ⁢   v   ⁢     a   ^     ⁢     r   (       γ   (     t   k     )     ⁢       Q   ^     (       S   g     ,     a   h       )       )                         v   ⁢     a   ^     ⁢       r       S   g     ,     a   h         [       E   ^     (       γ   ⁡   (     t   k     )     ⁢       Q   ^     (       S   g     ,     a   h       )       )     ]       =         ∑     g   =   1       m   +   1           ∑     h   =   1     n             L   ⁡   (       S   j     ,     a   i     ,     S   g     ,     a   h       )       K   ⁡   (       S   j     ,     a   i       )       [       E   ^     (       γ   ⁡   (     t   k     )     ⁢       Q   ^     (       S   g     ,     a   h       )       )     ]     2         -       [       ∑     g   =   1       m   +   1           ∑     h   =   1     n           L   ⁡   (       S   j     ,     a   i     ,     S   g     ,     a   h       )       K   ⁡   (       S   j     ,     a   i       )       [       E   ^     (       γ   ⁡   (     t   k     )     ⁢       Q   ^     (       S   g     ,     a   h       )       )     ]         ]     2             
where L(S j ,a i ,S g ,a h ) is the number of impressions where the context S j , action a i  was taken, the context of a subsequent session (before time T) of the same user was S g  and action a h  was taken.
 
     At step  106 , a given context S j , a sample of potential rewards is generated for each action a h  and a subsequent action is selected using Thompson sampling. The prior distribution (or existing posterior distribution if a set of impressions has been previously processed) is used on Q to generate samples  156  needed for estimation of μ post (S j ,a i ) and {circumflex over (σ)} 2 (S j ,a i ) for all j=1, 2, . . . , m and i=1, 2, . . . , n. It is noted that Q(S m+1 ,a i )=0 with a probability of 1 for all i=1, 2, . . . , n such that all samples  156  generated from a distribution of state S m+1  is 0. After the batch update is done on a subset of the set of training data  152 , a new posterior distribution Q is generated and Thompson sampling is applied to choose an action  158  for a given context. 
     At step  108 , the posterior distribution Q is updated using a subset of the impressions contained in the set of training data  152 . For each impression in the subset, a previous session of the user within time T is identified. The state S j  and the action a i  of the user during the previous session (e.g., prior impression) is obtained. For each state S j  and action a i  identified, the posterior distribution Q(S j ,a i ) is determined and the long-term reward information updated based on the updated posterior distribution. The posterior distribution Q is set as a prior distribution and the posterior of Q is updated using the empirical data contained in the training data  152  for the subset of impressions. 
     In some embodiments, steps  104 - 108  are repeated (i.e., iterated) multiple times to generate a trained content selection model  160 . At step  110 , the trained content selection model  160  is output to one or more systems, such as, for example, the content selection system  26 . The trained content selection model  160  is configured to eliminate errors caused by randomness in point estimates and reduces issues due to noise associated with point estimates. In some embodiments, Thompson sampling provides a more robust interface as compared to other explore-exploit techniques. The posterior distribution of expected reward provides a better consideration of possible action as compared to point estimates. 
       FIG.  5    illustrates a content selection process  200  for selecting content for presentation to a user using a trained content selection model  160 , in accordance with some embodiments. A user may interact with a computer environment, such as an e-commerce environment  202 , through one or more systems, such as, for example, a user system  22   a . When a user interacts with the e-commerce interface  202 , the e-commerce environment  202  presents an e-commerce interface  204  having a plurality of content containers  206   a - 206   c  (collectively “content containers  206 ”). At least one of the content containers  206 , such as a first content container  206   a , is configured to receive at least one content element selected from a plurality of content elements  208   a - 208   e . The plurality of content elements  208   a - 208   e  may be stored in and retrieved from any suitable storage, such as, for example, a content database  32 . The potential content elements  208   a - 208   e  may be selected based on any suitable criteria, such as, for example, a persona selected for the user. 
     In some embodiments, each of the potential content elements  208   a - 208   e  are provided to a trained content selection model  160 , which is configured to select one of the plurality of potential content elements  208   a - 208   e  for presentation to a user in the first content container  206   a . In some embodiments, the trained content selection model  160  considers the persona of the user and uses Thompson sampling, as discussed above, to select one of the plurality of potential content elements  208   a - 208   e  while balancing competing explore and exploit requirements. 
     The trained content selection model  160  selects a presentation content element  210  from among the potential content elements  208   a - 208   e  and presents the selected presentation content element  210  to the user in the first content container  206   a  of the e-commerce interface  204 . After receiving the e-commerce interface  204 , a user may perform one or more actions. In some embodiments, a set of presentation content elements  210  are preselected for users having a first persona such that the e-commerce interface  204  with the selected presentation content elements  210  may be cached and provided to a user without delay. The trained content selection model  160  may be configured to select presentation content elements  210  on a predetermined interval, such as, for example, selecting new elements each day, week, month, etc. 
     In some embodiments, the one or more actions performed by the user after presentation of the e-commerce interface  204  including the presentation content element  210  is recorded and used for training of future iterations of the trained content selection model  160 . For example, in various embodiments, the trained content selection model  160  may be replaced with an updated trained content selection model that has been trained using user interaction data from e-commerce interfaces  204  including presentation content elements  210  selected by the prior version of the trained content selection model  160 . In some embodiments, the trained content selection model  160  may be updated and/or replaced on a predetermined interval, such as, for example, hourly, daily, weekly, monthly, bi-monthly, etc. based on compute capacity. 
     Although the subject matter has been described in terms of exemplary embodiments, it is not limited thereto. Rather, the appended claims should be construed broadly, to include other variants and embodiments, which may be made by those skilled in the art.