Patent ID: 11954910
Assignee: MASSACHUSETTS INSTITUTE OF TECHNOLOGY
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 17:
18. The apparatus of claim 17, the operations further comprising generating, at each time step t, a logits z∈L+M−1 from hidden states ht by a fully-connected layer z=FC(ht, θFC) and using a Softmax to generate a categorical distribution πt,, π
   t
  
  =
  
   {
   
    
     
      p
      i
     
     ❘
     
      p
      i
     
    
    =
    
     
      exp
      ⁡
      (
      
       z
       i
      
      )
     
     
      
       
        ∑
         
       
       
        j
        =
        0
       
       
        L
        +
        M
        -
        1
       
      
      ⁢
      
       exp
       ⁡
       (
       
        z
        j
       
       )
      
     
    
   
   }, wherein discrete samples from a categorical distribution are drawn as follows:

{circumflex over (p)}=arg maxi(log pi+Gi), where Gi=−log (−logUi) is a standard Gumbel distribution with Ui sampled from a uniform i.i.d. distribution Unif(0, 1).