System and method for information highways in a hybrid feedforward-recurrent deep network

An apparatus and a method. The apparatus includes a first recurrent network in a first layer; a second recurrent network in a second layer connected to the first recurrent network; a distant input gate connected to the second recurrent network; a first highway gate connected to the distant input gate and the second recurrent network; a first elementwise product projection gate connected to the distant input gate, the highway gate, and the second recurrent network; a second highway gate connected to the first recurrent network and the second recurrent network; and a second elementwise product projection gate connected to the first recurrent network, the second highway gate, and the second recurrent network.

FIELD

The present disclosure relates generally to a system and a method for information highways in a deep neural network, and more particularly, to a system and a method for information highways in a hybrid feedforward-recurrent deep neural network.

BACKGROUND

A deep learning method is a type of machine learning method based on learning representations of data by using a deep graph with multiple processing layers. Such deep learning architectures include deep neural networks. A deep neural network (DNN) provides various functions such as image classification, speech recognition, and natural language processing. For example, Google's ALPHAGO™, a computer program for playing the board game “Go,” based on a deep convolutional neural network (deep CNN or DCNN), recently defeated the human world champion of “Go,” which suggests that complex tasks which were considered to be performed only by human beings can be solved by deep neural networks.

The depth of a neural network represents the number of successive layers in feedforward networks. A deeper neural network can better represent an input feature with lower complexity as compared to a shallow neural network. However, training deep networks is difficult due to vanishing/exploding gradient problems, and existing optimization solvers often fail with an increasing number of layers. Furthermore, the increasing depth of recurrent architectures like a gated recurrent unit (GRU) and long short term memories (LSTM) makes training of recurrent neural network (RNN) architectures more difficult because these architectures already have very deep representations in a temporal domain that may further aggravate vanishing/exploding gradient issues.

SUMMARY

According to one embodiment, an apparatus includes a first recurrent network in a first layer; a second recurrent network in a second layer connected to the first recurrent network; a distant input gate connected to the second recurrent network; a first highway gate connected to the distant input gate and the second recurrent network; a first elementwise product projection gate connected to the distant input gate, the first highway gate, and the second recurrent network; a second highway gate connected to the first recurrent network and the second recurrent network; and a second elementwise product projection gate connected to the first recurrent network, the second highway gate, and the second recurrent network.

According to one embodiment, a method of a hybrid recurrent network with a highway connection includes feeding an output from a first recurrent network in a first layer to a second recurrent network in a second layer via a first highway gate for the highway connection; and receiving a distant input in the second recurrent network via a distant input gate and a second highway gate.

According to one embodiment, an apparatus includes a first recurrent network in a first layer; a second recurrent network in a second layer connected to the first recurrent network; a third recurrent network in a second layer connected to the second recurrent network; a distant input gate in the third recurrent network connected to the first recurrent network; a first highway gate in the third recurrent network connected to the distant input gate; a second highway gate in the third recurrent network connected to the second recurrent network; and a first elementwise product projection gate in the third recurrent network connected to the second recurrent network and the second highway gate.

According to one embodiment, a method of a hybrid recurrent network with multiple highway connections includes feeding an output from a second recurrent network in a second layer to a third recurrent network in a third layer via a first highway gate for the highway connection; and receiving a distant input in the third recurrent network from a first recurrent network in a first layer via a distant input gate and a second highway gate.

According to one embodiment, a method of manufacturing a hybrid recurrent network with a highway connection includes forming the hybrid recurrent network with the highway connection as part of a wafer or package that includes at least one other hybrid recurrent network with a highway connection, wherein the hybrid recurrent network with a highway connection is configured to feed an output from a first recurrent network in a first layer to a second recurrent network in a second layer via a first highway gate for the highway connection, and receive a distant input in the second recurrent network via a distant input gate and a second highway gate; and testing the hybrid recurrent network with the highway connection, wherein testing the hybrid recurrent network with the highway connection comprises testing the hybrid recurrent network with the highway connection and the at least one other hybrid recurrent network with the highway connection using one or more electrical to optical converters, one or more optical splitters that split an optical signal into two or more optical signals, and one or more optical to electrical converters.

According to one embodiment, a method of constructing an integrated circuit includes generating a mask layout for a set of features for a layer of the integrated circuit, wherein the mask layout includes standard cell library macros for one or more circuit features that include a hybrid recurrent network with a highway connection configured to feed an output from a first recurrent network in a first layer to a second recurrent network in a second layer via a first highway gate for the highway connection, and receive a distant input in the second recurrent network via a distant input gate and a second highway gate; disregarding relative positions of the macros for compliance to layout design rules during the generation of the mask layout; checking the relative positions of the macros for compliance to layout design rules after generating the mask layout; upon detection of noncompliance with the layout design rules by any of the macros, modifying the mask layout by modifying each of the noncompliant macros to comply with the layout design rules; generating a mask according to the modified mask layout with the set of features for the layer of the integrated circuit; and manufacturing the integrated circuit layer according to the mask.

Hereinafter, embodiments of the present disclosure are described in detail with reference to the accompanying drawings. It should be noted that the same elements will be designated by the same reference numerals although they are shown in different drawings. In the following description, specific details such as detailed configurations and components are merely provided to assist with the overall understanding of the embodiments of the present disclosure. Therefore, it should be apparent to those skilled in the art that various changes and modifications of the embodiments described herein may be made without departing from the scope and spirit of the present disclosure. In addition, descriptions of well-known functions and constructions are omitted for clarity and conciseness. The terms described below are terms defined in consideration of the functions in the present disclosure, and may be different according to users, intentions of the users, or customs. Therefore, the definitions of the terms should be determined based on the contents throughout this specification.

The present disclosure may have various modifications and various embodiments, among which embodiments are described below in detail with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to the embodiments, but includes all modifications, equivalents, and alternatives within the spirit and the scope of the present disclosure.

The terms used herein are merely used to describe various embodiments of the present disclosure but are not intended to limit the present disclosure. Singular forms are intended to include plural forms unless the context clearly indicates otherwise. In the present disclosure, it should be understood that the terms “include” or “have” indicate existence of a feature, a number, a step, an operation, a structural element, parts, or a combination thereof, and do not exclude the existence or probability of the addition of one or more other features, numerals, steps, operations, structural elements, parts, or combinations thereof.

The present disclosure is related to a system and method for information highways in a hybrid feedforward-recurrent deep neural network. An embodiment of the present disclosure overcomes gradient vanishing/explosion issues on recurrent and hybrid neural networks. The present disclosure provides information flows in very deep neural networks that have both feedforward (fully connected or convolutional) and RNNs. RNNs may be implemented as GRUs and/or LSTMs that control the flow of information into and from a recurrent unit.

According to one embodiment, the present disclosure provides multiple highway connections from multiple inputs coming from different types of layers.

According to one embodiment, the present disclosure provides an LSTM architecture (e.g., H2LSTM) that includes highway gates that may pass information from previous LSTM layers, as well as distant layers such as prior convolutional layers or a distant input layer, where a distant input need not be from another recurrent network.

A highway connection that is relatively farther away may help to avoid gradient vanishing/explosion issues. This is especially useful for a hybrid neural network, because a highway connection from different components may be fed back directly.

The present disclosure provides accurate learning of a desired function as a network may learn a simplified function of an old input by driving weights of extra deep layers to zero.

An inter highway GRU (HGRU) may further provide highway connections from previous, or lower, GRU layers rather than within one GRU layer in combined, or stacked, GRU networks.

In an embodiment of the present disclosure, a GRU architecture (e.g., H2GRU) incorporates a modified HGRU architecture in hybrid networks with a highway gate for controlling the passage of information from distant layers into the HGRU state.

According to one embodiment, an RNN may be described in Equation (1) as follows:
ht=g(Whhht−1Wxhxt)  (1)
where g is a non linear function, xtis an input, and Whhand Wxhare weight matrices.

FIG. 1is a block diagram of an LSTM network architecture100.

Referring toFIG. 1, the LSTM network architecture100in a layer (e.g., layer L+1) includes a first cell activation register ct−1L+1101to hold the cell activation of a previous layer (e.g., layer L), a first output gate ht−1L+1103to hold the output of a previous layer, an input register xtL+1105, a forget gate ftL+1107, an input gate itL+1109, a new input register jtL+1111, a cell output activation register otL+1113, a second cell activation register ctL+1115, and a second output gate htL+1117.

The first cell activation register ct−1L+1101includes an output connected to a first input of a first elementwise product projection gate119, where the first elementwise product projection gate119includes a second input and an output. The first output gate ht−1L+1103includes an output connected to an input of each of the forget gate ftL+1107, the input gate itL+1109, the new input register jtL+1111, and the cell output activation register otL+1113. The input register xtL+1105includes an output connected to the inputs of each of the forget gate ftL+1107, the input gate itL+1109, the new input register jtL+1111, and the cell output activation register otL+1113. The forget gate ftL+1107includes an output connected to the second input of the first elementwise product projection gate119. The input gate itL+1109includes an output connected to a first input of a second elementwise product projection gate121, where the second elementwise product projection gate121includes a second input and an output. The new input register jtL+1111includes an output connected to the second input of the second elementwise product projection gate121. The cell output activation register otL+1113includes an output connected to a first input of a third elementwise product projection gate125, where the third elementwise product projection gate includes a second input and an output. A elementwise adder123includes a first input connected to the output of the first elementwise product projection gate119, a second input connected to the output of the second elementwise product projection gate121, and an output. The second cell activation register ctL+1115includes an input connected to the output of the elementwise adder123and an output connected to the second input of the elementwise product projection gate125. The second output gate htL+1117includes an input connected to the output of the third elementwise product projection gate125and an output.

The forget gate ftL+1107, the input gate itL+1109, the new input register jtL+1111, the cell output activation register otL+1113, the second cell activation ctL+1115, and the second output gate htL+1117, which may each be represented in Equations (2)-(7) as follows, where ⊙ represents an elementwise product:
ftL+1=sigm(WxfL+1xtL+1+WhfL+1ht−1L+1+WcfL+1ct−1L+1+bfL+1)  (2)
itL+1=sigm(WxiL+1xtL+1+WhiL+1ht−1L+1+WciL+1ct−1L+1+biL+1)  (3)
jtL+1=tanh(WxcL+1xtL+1+WhcL+1ht−1L+1+bjL+1)  (4)
otL+1=sigm(WxoL+1xtL+1+WhoL+1ht−1L+1+WcoL+1ctL+1+boL+1)  (5)
ctL+1=ftL+1⊙ct−1L+1+itL+1⊙jtL+1(6)
htL+1=otL+1⊙ tanh(ctL+1).  (7)

The input gate itL+1109controls how much information from the new input register jtL+1111is read. The forget gate ftL+1107controls how much information from the first cell activation register ct−1L+1101is forgotten. The projection layers may be neglected or added.

According to one embodiment, the LSTM network architecture has no peephole connections and gates do not depend on c, and sigm and tanh may be interchanged as in Equations (8)-(13) as follows:
itL+1=tanh(WxiL+1xtL+1+WhiL+1ht−1L+1+biL+1)  (8)
ftL+1=sigm(WxfL+1xtL+1+WhfL+1ht−1L+1+bfL+1)  (9)
jtL+1=sigm(WxcL+1xtL+1+WhcL+1ht−1L+1+bjL+1)  (10)
ctL+1=ftL+1⊙ct−1L+1+itL+1⊙jtL+1(11)
otL+1=tanh(WxoL+1xtL+1+WhoL+1ht−1L+1+boL+1)  (12)
htL+1=otL+1⊙ tanh(ctL+1).  (13)

Stacked LSTM layers may be considered as in Equation (14) as follows:
xtL+1=htL.  (14)

A highway, or depth, gate dtL+1connects a first memory cell in a lower layer L with a second memory cell in an upper layer L+1, and controls information that can flow directly as in Equations (15)-17) as follows:
xtL+1=htL(15)
dtL+1=s(bdL+1+WxdL+1xtL+1+wcdL+1⊙ct−1L+1+wldL+1⊙ctL)  (16)
ctL+1=dtL+1⊙ctL+ftL+1⊙ct−1L+1+itL+1⊙jtL+1.  (17)

According to one embodiment, the present system and method must pass and control information from an extra input. The extra input ptL*may be a much distant htL−1from another LSTM or an intermediate output after another subnetwork before the LSTM layers as CNN layers, or an initial input to the network.

According to one embodiment, a stacked DNN/CNN and an LSTM network may be combined.

FIG. 2is a block diagram of a combined CNN, or DNN, and LSTM network architecture (CNN/DNN-LSTM)200.

Referring toFIG. 2, the CNN/DNN-LSTM200in a layer (e.g., layer L+1) includes a first cell activation register ct−1L+1201to hold the cell activation of a previous layer (e.g., layer L), a first output gate ht−1L+1203to hold the output of a previous layer, an input register xtL+1205, a forget gate ftL+1207, an input gate itL+1209, a new input register jtL+1211, a cell output activation register otL+1213, a second cell activation register ctL+1215, a second output gate htL+1217, and a CNN or DNN227.

The first cell activation register ct−1L+1201includes an output connected to a first input of an elementwise product projection gate219, where the first elementwise product projection gate219includes a second input and an output. The first output gate ht−1L+1203includes an output connected to an input of each of the forget gate ftL+1207, the input gate itL+1209, the new input register jtL+1211, and the cell output activation register otL+1213. The input register xtL+1205includes an output connected to the inputs of each of the forget gate ftL+1207, the input gate itL+1209, the new input register jtL+1211, and the cell output activation register otL+1213. The forget gate ftL+1207includes an output connected to the second input of the first elementwise product projection gate219. The input gate itL+1209includes an output connected to a first input of a second elementwise product projection gate221, where the second elementwise product projection gate221includes a second input and an output. The new input register jtL+1211includes an output connected to the second input of the second elementwise product projection gate221. The cell output activation register otL+1213includes an output connected to a first input of a third elementwise product projection gate225, where the third elementwise product projection gate225includes a second input and an output. A elementwise adder223includes a first input connected to the output of the first elementwise product projection gate219, a second input connected to the output of the second elementwise product projection gate221, and an output. The second cell activation register ctL+1215includes an input connected to the output of the elementwise adder223and an output connected to the second input of the third elementwise product projection gate225. The second output gate htL+1217includes an input connected to the output of the third elementwise product projection gate225and an output. The CNN or DNN227includes an input, and an output connected to an input of the input register xtL+1205.

The LSTM may stacked at layer L+1, with a previous output from a DNN/CNN227at layer L called ptL, then the input register205is as in Equation (18) as follows:
xtL+1=ptL.  (18)

According to one embodiment, a hybrid DNN/CNN with LSTM networks and residual connection may be combined.

FIG. 3is a block diagram of a combined hybrid CNN, or hybrid DNN, and LSTM network architecture (hybrid CNN/DNN-LSTM)300.FIG. 3combines a hybrid DNN/CNN with LSTM networks and a residual connection. A residual network may initially be developed for a feedforward network with no LSTM. A desired distant input ptL*uses xt0as an auxiliary input to the network300, as shown inFIG. 3.

Referring toFIG. 3, the hybrid CNN/DNN-LSTM300in a layer (e.g., layer L+1) includes a first cell activation register ct−1L+1301to hold the cell activation of a previous layer (e.g., layer L), a first output gate ht−1L+1303to hold the output of a previous layer, an input register xtL+1305, a forget gate ftL+1307, an input gate itL+1309, a new input register jtL+1311, a cell output activation register otL+1313, a second cell activation register ctL+1315, a second output gate htL+1317, and CNN or DNN327.

The first cell activation register ct−1L+1301includes an output connected to a first input of an elementwise product projection gate319, where the first elementwise product projection gate319includes a second input and an output. The first output gate ht−1L+1303includes an output connected to an input of each of the forget gate ftL+1307, the input gate itL+1309, the new input register jtL+1311, and the cell output activation register otL+1313. The input register xtL+1305includes an output connected to the inputs of each of the forget gate ftL+1307, the input gate itL+1309, the new input register jtL+1311, and the cell output activation register otL+1313. The forget gate ftL+1307includes an output connected to the second input of the first elementwise product projection gate319. The input gate itL+1309includes an output connected to a first input of a second elementwise product projection gate321, where the second elementwise product projection gate321includes a second input and an output. The new input register jtL+1311includes an output connected to the second input of the second elementwise product projection gate321. The cell output activation register otL+1313includes an output connected to a first input of a third elementwise product projection gate325, where the third elementwise product projection gate325includes a second input and an output. A first elementwise adder323includes a first input connected to the output of the first elementwise product projection gate319, a second input connected to the output of the second elementwise product projection gate321, and an output. The second cell activation register ctL+1315includes an input connected to the output of the first elementwise adder323and an output connected to the second input of the third elementwise product projection gate325. The second output gate htL+1317includes an input connected to the output of the third elementwise product projection gate325and an output. The CNN or DNN327includes an input connected to a first input of a second elementwise adder329, and an output connected to a second input of the second elementwise adder329, where the second elementwise adder329includes an output connected to an input of the input register xtL+1305.

The CNN or DNN327is trained to learn H−ptL*, where H is the desired function. A shortcut connection is simply added is as in Equation (19) as follows such that:
xtL+1=ptL+ptL*,  (19)
where ptL*is the distant input which can be from the initial input, or an output from another network which can be CNN or DNN, or an output of another recurrent layer.

FIG. 4is a block diagram of a combined hybrid CNN, or hybrid DNN, and LSTM network architecture with a highway connection (hybrid CNN/DNN-HLSTM)400. A highway network may be initially developed for a simple feedforward network with no LSTM. A distant input ptL*may be xt0, an initial input to the hybrid CNN/DNN-HLSTM400.

Referring toFIG. 4, the hybrid CNN/DNN-HLSTM400in a layer (e.g., layer L+1) includes a first cell activation register ct−1L+1401to hold the cell activation of a previous layer (e.g., layer L), a first output gate ht−1L+1403to hold the output of a previous layer, an input register xtL+1405, a forget gate ftL+1407, an input gate itL+1409, a new input register jtL+1411, a cell output activation register otL+1413, a second cell activation register ctL+1415, a second output gate htL+1417, a CNN or a DNN427, a highway gate T431, and a highway gate C433.

The first cell activation register ct−1L+1401includes an output connected to a first input of an elementwise product projection gate419, where the first elementwise product projection gate419includes a second input and an output. The first output gate ht−1L+1403includes an output connected to an input of each of the forget gate ftL+1407, the input gate itL+1409, the new input register jtL+1411, and the cell output activation register otL+1413. The input register xtL+1405includes an output connected to the inputs of each of the forget gate ftL+1407, the input gate itL+1409, the new input register jtL+1411, and the cell output activation register otL+1413. The forget gate ftL+1407includes an output connected to the second input of the first elementwise product projection gate419. The input gate itL+1409includes an output connected to a first input of a second elementwise product projection gate421, where the second elementwise product projection gate421includes a second input and an output. The new input register jtL+1411includes an output connected to the second input of the second elementwise product projection gate421. The cell output activation register otL+1413includes an output connected to a first input of a third elementwise product projection gate425, where the third elementwise product projection gate425includes a second input and an output. A first elementwise adder423includes a first input connected to the output of the first elementwise product projection gate419, a second input connected to the output of the second elementwise product projection gate421, and an output. The second cell activation register ctL+1415includes an input connected to the output of the first elementwise adder423and an output connected to the second input of the third elementwise product projection gate425. The second output gate htL+1417includes an input connected to the output of the third elementwise product projection gate425and an output. The CNN or DNN427includes an input connected to an input of the highway gate C433, and an output, where the highway gate C433includes an output connected to a first input of a second elementwise adder429. The highway gate T431includes an input connected to the output of the CNN or DNN427and an output connected to a second input of the second elementwise adder429, where the second elementwise adder429includes an output connected to an input of the input register xtL+1405.

For the hybrid CNN/DNN-HLSTM400, the initial input xt0is passed through a highway gate at the input of the desired LSTM layer as in Equations (20) and (21) as follows:
xtL+1=xt0·C(x,WC)+ptL·T(x,WT)  (20)
T(x,WT)=s(WTx+bT);C(x,WC)=1−T(x,WT),  (21)
where bTis initialized towards a negative value so that the network is biased towards carry over, in highway networks.

FIG. 5is a block diagram of a hybrid LSTM network architecture with highway connections (H2LSTM)500, according to an embodiment of the present disclosure.

Referring toFIG. 5, the H2LSTM500in a first layer (e.g., layer L) includes a first cell activation register ct−1L505to hold the cell activation of a previous layer (e.g., layer L−1), a first output gate ht−1L507to hold the output of a previous layer, an input register xtL509, a forget gate ftL511, an input gate itL513, a new input register jtL515, a cell output activation register otL517, a second cell activation register ctL519, and a second output gate htL521.

The first cell activation register ct−1L505includes an output connected to a first input of a first elementwise product projection gate523, where the first elementwise product projection gate523includes a second input and an output. The first output gate ht−1L507includes an output connected to an input of each of the forget gate ftL511, the input gate itL513, the new input register jtL515, and the cell output activation register otL517. The input register xtL509includes an output connected to the inputs of each of the forget gate ftL511, the input gate itL513, the new input register jtL515, and the cell output activation register otL517. The forget gate ftL511includes an output connected to the second input of the first elementwise product projection gate523. The input gate itL513includes an output connected to a first input of a second elementwise product projection gate525, where the second elementwise product projection gate525includes a second input and an output. The new input register jtL515includes an output connected to the second input of the second elementwise product projection gate525. The cell output activation register otL517includes an output connected to a first input of a third elementwise product projection gate529, where the third elementwise product projection gate529includes a second input and an output. A elementwise adder527includes a first input connected to the output of the first elementwise product projection gate523, a second input connected to the output of the second elementwise product projection gate525, and an output. The second cell activation register ctL519includes an input connected to the output of the elementwise adder527and an output connected to the second input of the third elementwise product projection gate529. The second output gate htL521includes an input connected to the output of the third-elementwise product projection gate529and an output.

The H2LSTM500in a second layer (e.g., layer L+1) includes a first cell activation register ct−1L+1531to hold the cell activation of a previous layer (e.g., layer L), a first output gate ht−1L+1533to hold the output of a previous layer, an input register xtL+1535, a forget gate ftL+1537, an input gate itL+1539, a new input register jtL+1541, a cell output activation register otL+1543, a second cell activation register ctL+1545, a second output gate htL+1547, a distant signal register567, an additional RNN operation block569, a first highway gate571, and a second highway gate575.

The first cell activation register ct−1L+1531of the second layer includes a first output connected to a first input of a first elementwise product projection gate549, a second output connected to a first input of the first highway gate571, and a third output connected to a first input of the second highway gate575, where the first elementwise product projection gate549includes a second input and an output. The first output gate ht−1L+1533includes a first output connected to an input of each of the forget gate ftL+1537, the input gate itL+1539, the new input register jtL+1541, and the cell output activation register otL+1543, and a second output connected to a first input of the additional RNN operation block569. The input register xtL+1535includes an input connected to the output of the second output gate htL521of the first layer, a first output connected to the inputs of each of the forget gate ftL+1537, the input gate itL+1539, the new input register jtL+1541, and the cell output activation register otL+1543, a second output connected to a second input of the first highway gate571, and a third output connected to a second input of the second highway gate575. The forget gate ftL+1537includes an output connected to the second input of the first elementwise product projection gate549. The input gate itL+1539includes an output connected to a first input of a second elementwise product projection gate551, where the second elementwise product projection gate551includes a second input and an output. The new input register jtL+1541includes an output connected to the second input of the second elementwise product projection gate551. The cell output activation register otL+1543includes an output connected to a first input of a third elementwise product projection gate555, where the third elementwise product projection gate555includes a second input and an output. A first elementwise adder553includes a first input connected to the output of the first elementwise product projection gate549, a second input connected to the output of the second elementwise product projection gate551, and an output. A second elementwise adder557includes a first input connected to the output of the first elementwise adder553, a second input connected to an output of a fourth elementwise product projection gate573, a third input, and an output, where the fourth elementwise product projection gate573includes a first input and a second input. The second cell activation register ctL+1545includes an input connected to the output of the second elementwise adder557and an output connected to second input of the third elementwise product projection gate555. The second output gate htL+1547includes an input connected to the output of the third elementwise product projection gate555and an output. The distant signal register567includes an output connected to an input of the additional RNN operation block569. The additional RNN operation block569includes a first output connected to a third input of the first highway gate571, and a second output connected to a first input of a fourth elementwise product projection gate573. The first highway gate571includes an output connected to a second input of the fourth elementwise product projection gate573, where the fourth elementwise product projection gate573includes an output connected to the second input of the second elementwise adder557. The second highway gate575includes a third input connected to a second output of the second cell activation register ctL519of the first layer, and an output connected to a first input of a fifth elementwise product projection gate577, where the fifth elementwise product projection gate577includes a second input connected to a third output of the second cell activation register ctL519of the first layer, and an output connected to the third input of the second elementwise adder557.

The H2LSTM500network architecture includes a new highway gate that may feed directly into an LSTM cell at layer L+k as in Equations (22) and (23) as follows:
xtL+1=htL(22)
dtL+k=s(bdL+k+WxdL+kxtL+k+wcdL+k⊙ct−1L+k+wldL+k⊙ctL+k),  (23)
where dtL+kis a highway, or depth, gate for highway connection from a previous LSTM layer.

There is an additional RNN operation block569as a function of a previous output and an additional input as in Equation (24) as follows:
mtL+k=s(WpmL+kptL*+WhmL+kht−1L+k+bmL+k),  (24)
where mtL+kis an additional input as a function of a highway input and a previous layer's LSTM output. The sigmoid (s) function may be replaced with tanh or rectified linear unit (RELU). WpmL+kserves to project p into the cell dimension, at level L+k.

The highway gate for a distant signal ptL*may be represented as follows:
ytL+k=s(byL+k+WxyL+kxtL+k+wcyL+k⊙ct−1L+k+wmyL+k⊙mtL+k),  (25)
where ytL+kis the first highway gate571for an additional shortcut connection.

In an embodiment of the present system and method, ptL*may be an initial input xt0. Alternatively, the first highway gate571may be calculated based on ptL*directly as in Equation (26) follows:
ytL+k=s(byL+k+WxyL+kxtL+k+wcyL+k⊙ct−1L+k+WpyL+kptL*).  (26)

The second and final cell state ctL+k545is a weighted combination of the first cell state ct−1L+k531from the first layer through the forget state ftL+k537, an RNN with current LSTM input jtL+k541through an input gate itL+k539, a cell state of lower layer LSTM in the second ctL+k−1519of the first layer through the second highway gate, or highway, or depth, gate, dtL+k575, a distant signal register567as lower layer sub-network output from additional RNN operation block mtL+k569or initial input through the first highway gate ytL+k571as in Equation (27) as follows:
ctL+k=ftL+k⊙ct−1L+k+itL+k⊙jtL+k+dtL+k⊙ctL+k−1+ytL+k⊙mtL+k.  (27)

The final cell state ctL+k545takes information regarding ptL*indirectly through the additional RNN operation block569that calculates mtL+kvia the first highway gate571that calculates gate ytL+k.

According to one embodiment of the present system an method, the LSTM layer to which the additional input is passed may be modified as in Equation (28)-(36) as follows:
xtL+1=htL(28)
itL+1=tanh(WxiL+1xtL+1+WhiL+1ht−1L+1+biL+1)  (29)
ftL+1=sigm(WxfL+1xtL+1+WhfL+1ht−1L+1+bfL+1)  (30)
jtL+1=sigm(WxcL+1xtL+1+WhcL+1ht−1L+1+bjL+1)  (31)
otL+1=tanh(WxoL+1xtL+1+WhoL+1ht−1L+1+boL+1)  (32)
dtL+1=sigm(bdL+1+WxdL+1xtL+1+wcdL+1⊙ct−1L+1+wldL+1⊙ctL)  (33)
mtL+1=sigm(WpmL+1ptL+WhmL+1ht−1L+1+bmL+1)  (34)
ytL+1=sigm(byL+1+WxyL+1xtL+1+wcyL+1⊙ct−1L+1+wmyL+1⊙tL+1)  (35)
ctL+1=ftL+1⊙t−1L+1+itL+1⊙jtL+1+dtL+1⊙ctL+ytL+1⊙mtL+1(36)
htL+1=otL+1⊙ tanh(ctL+1).  (37)

According to one embodiment of the present system and method, if there are multiple additional inputs, P of ptL*, then multiple highway gates y(1, . . . , P)tL+1may be added and their outputs update the corresponding m and contribute to the cell value as in Equation (38) as follows:
ctL+1=ftL+1⊙ct−1L+1+itL+1⊙jtL+1+dtL+1⊙ctL+Σ∝=1Py(∝)tL+1⊙m(∝)tL+1.  (38)

Multiple additional inputs based from previous LSTM layers may be added through multiples of the second highway gate, or depth gate, dtL+11575. This may be extended for multiple highway inputs via multiplicities of ptL*and mtL+1. According to one embodiment of the present system and method, different cell outputs from lower LSTM layers may be concatenated, and the concatenated vector may be passed through the highway gate575. The present system and method controls the flow of information from the distant signal register567which provides ptL*directly into the second cell activation register ctL+1545.

Referring toFIG. 6, the GRU network architecture600in a layer (e.g., layer L+1) includes an input gate xtL+1601, an update gate ztL+1603, a reset gate rtL+1605, a candidate gate {tilde over (h)}tL+1607, a first output gate ht−1L+1609to hold the output of a previous layer (e.g., layer L), and a second output gate htL+1621.

The input gate xtL+1601includes an output connected to a first input of each of the update gate ztL+1603, the reset gate rtL+1605, and the candidate gate {tilde over (h)}tL+1607. The first output gate ht−1L+1609includes a first output connected to a second input of the update gate ztL+1603, a second output connected to a second input of the reset gate rtL+1605, a third output connected to a first input of a first elementwise product projection gate617, and a fourth output connected to a first input of a second elementwise product projection gate615. The update gate ztL+1603includes a first output connected to a second input of the first elementwise product projection gate617, and a second output connected to subtract from 1 function block611. The reset gate rtL+1605includes an output connected to a second input of the second elementwise product projection gate615. The candidate gate {tilde over (h)}tL+1607includes a second input connected to an output of the second elementwise product projection gate615, and an output connected to a first input of a third elementwise product projection gate613. The subtract from 1 function block611includes an output connected to a second input of the third elementwise product projection gate613. The first elementwise product projection gate617includes an output connected to a first input of an elementwise adder619. The elementwise added619includes a second input connected to an output of the third elementwise product projection gate613, and an output connected to an input of the second output gate htL+1621.

For a hybrid network, the above architectures for LSTM are also applicable to GRUs, for example, a hybrid CNN-GRU with an input highway. For the description below, a distant input ptL*may be an initial input xt0to a network.

FIG. 7is a block diagram of a GRU network architecture with highway connections (HGRU)700.

Referring toFIG. 7, the HGRU network architecture700in a layer (e.g., layer L+1) includes an input gate xtL+1701, an update gate ztL+1703, a reset gate rtL+1705, a candidate gate {tilde over (h)}tL+1707, a first output gate ht−1L+1709to hold the output of a previous layer (e.g., layer L), a second output gate htL+1721, and a highway gate dtL+1723.

The input gate xtL+1701includes a first output connected to a first input of each of the update gate ztL+1703, the reset gate rtL+1705, and the candidate gate {tilde over (h)}tL+1707, a second output connected to a first input of the highway gate dtL+1723, and a third output connected to a first input of a fourth elementwise product projection gate725. The first output gate ht−1L+1709includes a first output connected to a second input of the update gate ztL+1703, a second output connected to a second input of the reset gate rtL+1705, a third output connected to a first input of a first elementwise product projection gate717, a fourth output connected to a first input of a second elementwise product projection gate715, and a fifth output connected to an input of the highway gate dtL+1723. The update gate ztL+1703includes a first output connected to a second input of the first elementwise product projection gate717, and a second output connected to a first subtract from 1 function block711. The reset gate rtL+1705includes an output connected to a second input of the second elementwise product projection gate715. The candidate gate {tilde over (h)}tL+1707includes a second input connected to an output of the second elementwise product projection gate715, and an output connected to a first input of a third elementwise product projection gate713. The first subtract from 1 function block711includes an output connected to a second input of the third elementwise product projection gate713. The first elementwise product projection gate717includes an output connected to a first input of an elementwise adder719. The highway gate dtL+1723includes a first output connected to a second input of the fourth elementwise product projection gate725, and a second output connected to an input of a second subtract from 1 function block727. A fifth elementwise product projection gate729includes an input connected to an output of the elementwise adder719, a second input connected to an output of the second subtract from 1 function block727, and an output connected to a first input of the second output gate htL+1721. An output of the fourth elementwise product projection gate725is connected to a second input of the second output gate htL+1721.

The highway gate723is within the HGRU network architecture700and controls the information from the current input. A HGRU network architecture700is a modified GRU that includes a connection from the input to the output through the highway gate723as in Equations (43)-(47) as follows:
rt=sigm(Wxrxt+Whrht−1+br)  (43)
zt=sigm(Wxzxt+Whzht−1+bz)  (44)
{tilde over (h)}t=tanh(Wxhxt+Whh(rt⊙ht−1)+bh)  (45)
dt=sigm(Wxdxt+Whdht−1+bd)  (46)
ht=dt⊙xt+(1−dt)⊙(zt⊙ht−1+(1−zt)⊙{tilde over (h)}t).  (47)

FIG. 8is a block diagram of a combined hybrid CNN, or hybrid DNN, and GRU network architecture (hybrid CNN/DNN-GRU)800.

Referring toFIG. 8, the hybrid CNN/DNN-GRU network architecture800in a layer (e.g., layer L+1) includes an input gate xtL+1801, an update gate ztL+1803, a reset gate rtL+1805, a candidate gate {tilde over (h)}tL+1807, a first output gate ht−1L+1809to hold the output of a previous layer (e.g., layer L), a second output gate htL+1821, a CNN or a DNN831, a highway gate T833, and a highway gate C837.

The input gate xtL+1801includes an output connected to a first input of each of the update gate ztL+1803, the reset gate rtL+1805, and the candidate gate {tilde over (h)}tL+1807. The first output gate ht−1L+1809includes a first output connected to a second input of the update gate ztL+1803, a second output connected to a second input of the reset gate rtL+1805, a third output connected to a first input of a first elementwise product projection gate817, and a fourth output connected to a first input of a second elementwise product projection gate815. The update gate ztL+1803includes a first output connected to a second input of the first elementwise product projection gate817, and a second output connected to subtract from 1 function block811. The reset gate rtL+1805includes an output connected to a second input of the second elementwise product projection gate815. The candidate gate {tilde over (h)}tL+1807includes a second input connected to an output of the second elementwise product projection gate815, and an output connected to a first input of a third elementwise product projection gate813. The subtract from 1 function block811includes an output connected to a second input of the third elementwise product projection gate813. The first elementwise product projection gate817includes an output connected to a first input of a first elementwise adder819. The first elementwise adder819includes a second input connected to an output of the third elementwise product projection gate813, and an output connected to an input of the second output gate htL+1821. The CNN or DNN831includes an input connected to an input of the highway gate C837, and an output, where the highway gate C837includes an output connected to a first input of a second elementwise adder835. The highway gate T833includes an input connected to the output of the CNN or DNN831and an output connected to a second input of the second elementwise adder835, where the second elementwise adder835includes an output connected to an input of the input register xtL+1801.

The initial input xt0to the input register xtL+1801may be passed through a highway gate at an input of an LSTM layer as in Equations (48) and (49) as follows:
xtL+1=xt0·C(x,WC)+ptL·T(x,WT)  (48)
T(x,WT)=s(WTx+bT);C(x,WC)=1−T(x,WT),  (49)
where bTis initialized towards a negative value so that the network is biased towards carry over in a highway network.

A stacked DNN/CNN and LSTM network scheme and a hybrid DNN/CNN with LSTM networks and a residual connection scheme as described above for LSTM may also be modified for GRUs, where the highway gate T833and the highway gate C837may each be replaced by a simple addition to the residual network. The highway gate T833may be a transform gate while the highway gate C837may be a carry gate.

FIG. 9is a block diagram of a combined hybrid CNN, or hybrid DNN, and HGRU network architecture (hybrid CNN/DNN-HGRU)900. The hybrid CNN/DNN-HGRU network architecture900has an HGRU as a building block, and a distant input ptL*may be passed to the HGRU input via a carry gate of an input highway.

Referring toFIG. 9, the HGRU network architecture900in a layer (e.g., layer L+1) includes an input gate xtL+1901, an update gate ztL+1903, a reset gate rtL+1905, a candidate gate {tilde over (h)}tL+1907, a first output gate ht−1L+1909to hold the output of a previous layer (e.g., layer L), a second output gate htL+1921, a highway gate dtL+1923, a CNN or a DNN931, a highway gate T933, and a highway gate C937.

The input gate xtL+1901includes a first output connected to a first input of each of the update gate ztL+1903, the reset gate rtL+1905, and the candidate gate {tilde over (h)}tL+1907, a second output connected to a first input of the highway gate dtL+1923, and a third output connected to a first input of a fourth elementwise product projection gate925. The first output gate ht−1L+1909includes a first output connected to a second input of the update gate ztL+1903, a second output connected to a second input of the reset gate rtL+1905, a third output connected to a first input of a first elementwise product projection gate917, a fourth output connected to a first input of a second elementwise product projection gate915, and a fifth output connected to an input of the highway gate dtL+1923. The update gate ztL+1903includes a first output connected to a second input of the first elementwise product projection gate917, and a second output connected to a first subtract from 1 function block911. The reset gate rtL+1905includes an output connected to a second input of the second elementwise product projection gate915. The candidate gate {tilde over (h)}tL+1907includes a second input connected to an output of the second elementwise product projection gate915, and an output connected to a first input of a third elementwise product projection gate913. The first subtract from 1 function block911includes an output connected to a second input of the third elementwise product projection gate913. The first elementwise product projection gate917includes an output connected to a first input of an elementwise adder919. The highway gate dtL+1923includes a first output connected to a second input of the fourth elementwise product projection gate925, and a second output connected to an input of a second subtract from 1 function block927. A fifth elementwise product projection gate929includes an input connected to an output of the elementwise adder919, a second input connected to an output of the second subtract from 1 function block927, and an output connected to a first input of the second output gate htL+1921. An output of the fourth elementwise product projection gate925is connected to a second input of the second output gate htL+1921. The CNN or DNN931includes an input connected to an input of the highway gate C937, and an output, where the highway gate C937includes an output connected to a first input of a second elementwise adder935. The highway gate T933includes an input connected to the output of the CNN or DNN931and an output connected to a second input of the second elementwise adder935, where the second elementwise adder935includes an output connected to an input of the input register xtL+1901.

FIG. 10is a block diagram of a combined GRU with an inter-highway connection and GRU network architecture with highway connections1000. A single layer highway GRU may be modified such that a highway input is from a previous input of a previous GRU layer (and not from the current input of a current GRU layer). However,FIG. 10may also be a block diagram of a combined HGRU with an inter-highway connection (H2GRU) and GRU network architecture with highway connections1000. A single layer highway GRU may be modified such that a highway input is from a previous input of a previous GRU layer (and not from the current input of a current GRU layer). This passes distant information through a highway and allows a network to learn a simpler function of the previous input by driving weights of a stacked layer to zero.

Referring toFIG. 10, the combined GRU with an inter-highway connection and GRU network architecture with highway connections1000in a first layer (e.g., layer L) includes an input gate xtL1001, an update gate ztL1003, a reset gate rtL1005, a candidate gate {tilde over (h)}tL1007, a first output gate ht−1L1009to hold the output of a previous layer (e.g., layer L−1), and a second output gate htL1021.

The input gate xtL1001includes a first output connected to a first input of each of the update gate ztL1003, the reset gate rtL1005, and the candidate gate {tilde over (h)}tL1007, a second output for connecting to a highway gate in another layer, and a third output for connecting to a elementwise product projection gate in another layer. The first output gate ht−1L1009includes a first output connected to a second input of the update gate ztL1003, a second output connected to a second input of the reset gate rtL1005, a third output connected to a first input of a first elementwise product projection gate1017, and a fourth output connected to a first input of a second elementwise product projection gate1015. The update gate ztL1003includes a first output connected to a second input of the first elementwise product projection gate1017, and a second output connected to a subtract from 1 function block1011. The reset gate rtL1005includes an output connected to a second input of the second elementwise product projection gate1015. The candidate gate {tilde over (h)}tL1007includes a second input connected to an output of the second elementwise product projection gate1015, and an output connected to a first input of a third elementwise product projection gate1013. The subtract from 1 function block1011includes an output connected to a second input of the third elementwise product projection gate1013. The first elementwise product projection gate1017includes an output connected to a first input of an elementwise adder1019. The elementwise adder1019includes a second input connected to an output of the third elementwise product projection gate1013, and an output connected to an input of the second output gate htL1021. An output of the second output gate htL1021is connected to an input of an input gate of another layer.

The combined GRU with an inter-highway connection and GRU network architecture with highway connections1000in a second layer (e.g., layer L+1) includes an input register xtL+11031, an update gate ztL+11033, a reset gate rtL+11035, a candidate gate {tilde over (h)}tL+11037, a first output gate ht−1L+11039to hold the output of a previous layer (e.g., layer L), a second output gate htL+11051, and a highway gate dtL+11053.

The input gate xtL+11031of the second layer includes an input connected to the output of the second output gate htL1021of the first layer, a first output connected to a first input of each of the update gate ztL+11033, the reset gate rtL+11035, and the candidate gate {tilde over (h)}tL+11037, and a second output connected to a first input of a fourth elementwise product projection gate1055. The first output gate ht−1L+11039includes a first output connected to a second input of the update gate ztL+11033, a second output connected to a second input of the reset gate rtL+11035, a third output connected to a first input of a first elementwise product projection gate1047, a fourth output connected to a first input of a second elementwise product projection gate1045, and a fifth output connected to a first input of the highway gate dtL+11053. The update gate ztL+11033includes a first output connected to a second input of the first elementwise product projection gate1047, and a second output connected to a first subtract from 1 function block1041. The reset gate rtL+11035includes an output connected to a second input of the second elementwise product projection gate1045. The candidate gate {tilde over (h)}tL+11037includes a second input connected to an output of the second elementwise product projection gate1045, and an output connected to a first input of a third elementwise product projection gate1043. The first subtract from 1 function block1041includes an output connected to a second input of the third elementwise product projection gate1043. The first elementwise product projection gate1047includes an output connected to a first input of a first elementwise adder1049. The highway gate dtL+11053includes a second input connected to the second output of the input gate xtL1001of the first layer, a first output connected to a second input of the fourth elementwise product projection gate1055, and a second output connected to an input of a second subtract from 1 function block1057. A fifth elementwise product projection gate1059includes an input connected to an output of the first elementwise adder1049, a second input connected to an output of the second subtract from 1 function block1057, and an output connected to a first input of a second elementwise adder1061. The fourth elementwise product projection gate1055includes a third input connected to the third output of the input gate xtL1001of the first layer, and an output connected to a second input of the second elementwise adder1061. The second elementwise adder1061, includes an output connected to an input of the output gate htL+11051.

FIG. 11is a block diagram of a combined hybrid CNN/DNN with a GRU and a GRU with highway connections1100. However,FIG. 11may also be a block diagram of a combined hybrid CNN/DNN with an H2GRU and a GRU with highway connections1100. The hybrid CNN/DNN with an H2GRU and a GRU with highway connections1100allows an input to a layer L to be from another highway connection over a feedforward CNN or DNN network.

Referring toFIG. 11, the combined hybrid CNN/DNN with a GRU and a GRU with highway connections1100in a first layer (e.g., layer L) includes an input gate xtL1101, an update gate ztL1103, a reset gate rtL1105, a candidate gate {tilde over (h)}tL1107, a first output gate ht−1L1109to hold the output of a previous layer (e.g., layer L−1), a second output gate htL1121, a CNN or a DNN1171, a highway gate T1173, and a highway gate C1177.

The input gate xtL1101includes an input, a first output connected to a first input of each of the update gate ztL1103, the reset gate rtL1105, and the candidate gate {tilde over (h)}tL1107, a second output for connecting to a highway gate in another layer, and a third output for connecting to a elementwise product projection gate in another layer. The first output gate ht−1L1109includes a first output connected to a second input of the update gate ztL1103, a second output connected to a second input of the reset gate rtL1105, a third output connected to a first input of a first elementwise product projection gate1117, and a fourth output connected to a first input of a second elementwise product projection gate1115. The update gate ztL1103includes a first output connected to a second input of the first elementwise product projection gate1117, and a second output connected to a subtract from 1 function block1111. The reset gate rtL1105includes an output connected to a second input of the second elementwise product projection gate1115. The candidate gate {tilde over (h)}tL1107includes a second input connected to an output of the second elementwise product projection gate1115, and an output connected to a first input of a third elementwise product projection gate1113. The subtract from 1 function block1111includes an output connected to a second input of the third elementwise product projection gate1113. The first elementwise product projection gate1117includes an output connected to a first input of an elementwise adder1119. The elementwise adder1119includes a second input connected to an output of the third elementwise product projection gate1113, and an output connected to an input of the second output gate htL1121. An output of the second output gate htL1121is connected to an input of an input gate of another layer. The CNN or DNN1171includes an input connected to an input of the highway gate C1177, and an output, where the highway gate C1177includes an output connected to a first input of a second elementwise adder1175. The highway gate T1173includes an input connected to the output of the CNN or DNN1171and an output connected to a second input of the second elementwise adder1175, where the second elementwise adder1175includes an output connected to the input of the input register xtL1101.

A combined hybrid CNN/DNN with an H2GRU and a GRU with highway connections1100in a second layer (e.g., layer L+1) includes an input gate xtL1131, an update gate ztL+11133, a reset gate rtL+11135, a candidate gate {tilde over (h)}tL+11137, a first output gate ht−1L+11139to hold the output of a previous layer (e.g., layer L), a second output gate htL+11151, and a highway gate dtL+11153.

The input gate xtL+11131of the second layer includes an input connected to the output of the second output gate htL1121of the first layer, a first output connected to a first input of each of the update gate ztL+11133, the reset gate rtL+11135, and the candidate gate {tilde over (h)}tL+11137, and a second output connected to a first input of a fourth elementwise product projection gate1155. The first output gate ht−1L+11139includes a first output connected to a second input of the update gate ztL+11133, a second output connected to a second input of the reset gate rtL+11135, a third output connected to a first input of a first elementwise product projection gate1147, a fourth output connected to a first input of a second elementwise product projection gate1145, and a fifth output connected to a first input of the highway gate dtL+11153. The update gate ztL+11133includes a first output connected to a second input of the first elementwise product projection gate1147, and a second output connected to a first subtract from 1 function block1141. The reset gate rtL+11135includes an output connected to a second input of the second elementwise product projection gate1145. The candidate gate {tilde over (h)}tL+11137includes a second input connected to an output of the second elementwise product projection gate1145, and an output connected to a first input of a third elementwise product projection gate1143. The first subtract from 1 function block1141includes an output connected to a second input of the third elementwise product projection gate1143. The first elementwise product projection gate1147includes an output connected to a first input of a first elementwise adder1149. The highway gate dtL+11153includes a second input connected to the second output of the input gate xtL1001of the first layer, a first output connected to a second input of the fourth elementwise product projection gate1155, and a second output connected to an input of a second subtract from 1 function block1157. A fifth elementwise product projection gate1159includes an input connected to an output of the first elementwise adder1149, a second input connected to an output of the second subtract from 1 function block1157, and an output connected to a first input of a second elementwise adder1161. The fourth elementwise product projection gate1155includes a third input connected to the third output of the input gate xtL1001of the first layer, and an output connected to a second input of the second elementwise adder1161. The second elementwise adder1161, includes an output connected to an input of the output gate htL+11151.

FIG. 12is a block diagram of a combined GRU with an inter GRU and a GRU with multiple highway connections1200, according to an embodiment of the present disclosure. However,FIG. 12may also be a block diagram of a combined GRU with an H2GRU and a GRU with multiple highway connections1200, according to an embodiment of the present disclosure. A HGRU may be modified to take two highway connections at a given layer, one from a lower GRU layer, and one from a further distant signal, ptL*.

Referring toFIG. 12, the combined GRU with a GRU and a GRU with multiple highway connections1200in a first layer (e.g., layer L−1) includes an input gate xtL−11201, an update gate ztL−11203, a reset gate rtL−11205, a candidate gate {tilde over (h)}tL−11207, a first output gate ht−1L−11209to hold the output of a previous layer (e.g., layer L−2), and a second output gate htL−11221.

The input gate xtL−11201in the first layer includes an output connected to a first input of each of the update gate ztL−11203, the reset gate rtL−11205, and the candidate gate {tilde over (h)}tL−11207. The first output gate ht−1L−11209includes a first output connected to a first input of a first elementwise product projection gate1217, and a second output connected to a first input of a second elementwise product projection gate1215. The update gate ztL−11203includes a first output connected to a second input of the first elementwise product projection gate1217, and a second output connected to a subtract from 1 function block1211. The reset gate rtL−11205includes an output connected to a second input of the second elementwise product projection gate1215. The candidate gate {tilde over (h)}tL−11207includes a second input connected to an output of the second elementwise product projection gate1215, and an output connected to a first input of a third elementwise product projection gate1213. The subtract from 1 function block1211includes an output connected to a second input of the third elementwise product projection gate1213. The first elementwise product projection gate1217includes an output connected to a first input of an elementwise adder1219. The elementwise adder1219includes a second input connected to an output of the third elementwise product projection gate1213, and an output connected to an input of the second output gate htL−11221. An output of the second output gate htL−11221is connected to an input of an input gate of another layer.

The combined GRU with a GRU and a GRU with multiple highway connections1200in a second layer (e.g., layer L) includes an input gate xtL1231, an update gate ztL1233, a reset gate rtL1235, a candidate gate {tilde over (h)}tL1237, a first output gate ht−1L1239to hold the output of a previous layer (e.g., layer L−1), and a second output gate htL1251.

The input gate xtL1231in the second layer includes an input connected to the output of the the second output gate htL1221of layer L−1, a first output connected to a first input of each of the update gate ztL1233, the reset gate rtL1235, and the candidate gate {tilde over (h)}tL1237, a second output for connecting to a highway gate in another layer, and a third output for connecting to a elementwise product projection gate in another layer. The first output gate ht−1L1239includes a first output connected to a second input of the update gate ztL1233, a second output connected to a second input of the reset gate rtL1235, a third output connected to a first input of a first elementwise product projection gate1247, and a fourth output connected to a first input of a second elementwise product projection gate1245. The update gate ztL1233includes a first output connected to a second input of the first elementwise product projection gate1247, and a second output connected to a subtract from 1 function block1241. The reset gate rtL1235includes an output connected to a second input of the second elementwise product projection gate1245. The candidate gate {tilde over (h)}tL1237includes a second input connected to an output of the second elementwise product projection gate1245, and an output connected to a first input of a third elementwise product projection gate1243. The subtract from 1 function block1241includes an output connected to a second input of the third elementwise product projection gate1243. The first elementwise product projection gate1247includes an output connected to a first input of an elementwise adder1249. The elementwise adder1249includes a second input connected to an output of the third elementwise product projection gate1243, and an output connected to an input of the second output gate htL1251. An output of the second output gate htL1251is connected to an input of an input gate of another layer.

The combined combined GRU with a GRU and a GRU with multiple highway connections1200in a third layer (e.g., layer L+1) includes an input gate xtL+11261, an update gate ztL+11263, a reset gate rtL+11265, a candidate gate {tilde over (h)}tL+11267, a first output gate ht−1L+11269to hold the output of a previous layer (e.g., layer L), a second output gate htL+11281, a highway gate dtL+11283, a distant input gate1291including a first input for receiving a distant input1293, and a highway gate for the distant input1295.

The input gate xtL+11261in the third layer includes an input connected to the output of the second output gate htL1251of the second layer, a first output connected to a first input of each of the update gate ztL+11263, the reset gate rtL+11265, and the candidate gate {tilde over (h)}tL+11267, and a second output connected to a first input of the highway gate of the distant input1295. The first output gate ht−1L+11269includes a first output connected to a second input of the update gate ztL+11263, a second output connected to a second input of the reset gate rtL+11265, a third output connected to a first input of a first elementwise product projection gate1277, a fourth output connected to a first input of a second elementwise product projection gate1275, a fifth output connected to a first input of the highway gate dtL+11283, a sixth output connected to a second input of the distant input gate1291, and a seventh output connected to a second input of the highway gate for the distant input1295. The update gate ztL+11263includes a first output connected to a second input of the first elementwise product projection gate1277, and a second output connected to a first subtract from 1 function block1271. The reset gate rtL+11265includes an output connected to a second input of the second elementwise product projection gate1275. The candidate gate {tilde over (h)}tL+11267includes a second input connected to an output of the second elementwise product projection gate1275, and an output connected to a first input of a third elementwise product projection gate1273. The first subtract from 1 function block1271includes an output connected to a second input of the third elementwise product projection gate1273. The first elementwise product projection gate1277includes an output connected to a first input of a first elementwise adder1279. The highway gate dtL+11283includes a second input connected to the second output of the input gate xtL1231of the second layer, a first output connected to a second input of the fourth elementwise product projection gate1285, and a second output connected to a first input of a second elementwise adder1299. The distant input gate1291includes a first output connected to a third input of the highway gate for the distant input1295and a second output connected to a first input of a sixth elementwise product projection gate1297. The highway gate for the distant input1295includes a first output connected to a second input of the second elementwise adder1299and a second output connected to a second input of the sixth elementwise product projection gate1297. The second elementwise adder1299includes an output connected to an input to the second subtract from 1 function block1298. A fifth elementwise product projection gate1289includes a first input connected to an output of the first elementwise adder1279, a second input connected to an output of the second subtract from 1 function block1298, and an output connected to a first input of a third elementwise adder1296. The fourth elementwise product projection gate1285includes a second input connected to the third output of the input gate xtL1231of the second layer, and an output connected to a second input of the third elementwise adder1296. The third elementwise adder1296, includes an output connected to an input of the second output gate htL+11281.

A recurrent operation with a distant input ptL*may be described as in Equation (56) as follows:
mtL+k=tanh(WpmL+1ptL*+WhmL+1ht−1L+1+bmL+1).  (56)

The highway gate for the distant input1295, as a function of the distant input ptL*, a current input, and a current input state, may be described as in Equation (57) as follows:
ytL+1=sigm(WxyL+1xtL+1+WhyL+1ht−1L+1+WpyL+1ptL*+byL+1).  (57)

A gated recurrent information from a distant output may then be passed through a highway gate, together with that from a lower GRU layer. Various gates may be represented as in Table 1 as follows:

mtL+11291may be a recurrent operation added as a function of an additional distant input. ytL+11295may be a highway gate for a distant input as a function of the distant input ptL*, current input, and current input state. According to one embodiment, multiple y gates may be initiated in case of multiple connections from previous layers, which may be combined with multiple inter-GRU layer highway connections as well.

According to one embodiment, a recurrent network may be implemented as GRUs or LSTMs that control the flow of information into and from the recurrent network. An H2LSTM network architecture may provide multiple highway connections from multiple inputs coming from different types of layers. An H2LSTM network may further learn a simplified function of previous inputs by driving weights of extra deep layers to zero. An H2GRU may provide highway connections from previous GRU layers as well as other layers such as feedforward layers, rather than within only one GRU layer. A recurrent network allows highway connections in hybrid networks with both feedforward (fully connected or convolutional) and recurrent layers. Inner states of LSTM or GRU recurrent units still depend on current input and previous states only, and additional highway gates allow control of information from a distant input to the cell, after transformation of the distant input with an additional recurrent cell. This supplements the respective LSTM or GRU unit output with additional information about distant input which helps the learning process.

FIG. 13is a flowchart of a method of a hybrid recurrent network with a highway connection, according to an embodiment of the present disclosure. For example, a method of a hybrid LSTM network architecture with highway connections (H2LSTM).

Referring toFIG. 13, the method includes feeding an output from a first recurrent network in a first layer to a second recurrent network in a second layer via a highway, or depth, gate for the highway connection at1301.

At1303, the method includes receiving a distant input in the second recurrent network via a distant input gate and a highway gate.

FIG. 14is a flowchart of a method of a hybrid recurrent network with multiple highway connections, according to an embodiment of the present disclosure.

Referring toFIG. 14, the method includes feeding an output from a second recurrent network in a second layer to a third recurrent network in a third layer via a highway, or depth, gate for the highway connection, at1401.

At1403, the method includes receiving a distant input in the third recurrent network from a first recurrent network in a first layer via a distant input gate and a highway gate.

FIG. 15is a flowchart of a method of manufacturing a hybrid recurrent network with a highway connection, according to an embodiment of the present disclosure.

Referring toFIG. 15, the method, at1501, the method includes forming the hybrid recurrent network with the highway connection as part of a wafer or package that includes at least one other hybrid recurrent network with a highway connection, wherein the hybrid recurrent network with a highway connection is configured to feed an output from a first recurrent network in a first layer to a second recurrent network in a second layer via a highway, or depth, gate for the highway connection, and receive a distant input in the second recurrent network via a distant input gate and a highway gate.

At1503, the method includes testing the hybrid recurrent network with the highway connection, wherein testing the hybrid recurrent network with the highway connection comprises testing the hybrid recurrent network with the highway connection and the at least one other hybrid recurrent network with the highway connection using one or more electrical to optical converters, one or more optical splitters that split an optical signal into two or more optical signals, and one or more optical to electrical converters.

FIG. 16is a flowchart of constructing an integrated circuit, a method of manufacturing a hybrid recurrent network with a highway connection.

Referring toFIG. 16, the method, at1601, constructs initial layout data. For example, the method may generate a mask layout for a set of features for a layer of the integrated circuit, wherein the mask layout includes standard cell library macros for one or more circuit features that include a hybrid recurrent network with a highway connection configured to feed an output from a first recurrent network in a first layer to a second recurrent network in a second layer via a highway, or depth gate for the highway connection, and receive a distant input in the second recurrent network via a distant input gate and a highway gate, and disregarding relative positions of the macros for compliance to layout design rules during the generation of the mask layout.

At1603, the method includes performing a design rule check. For example, the method may check the relative positions of the macros for compliance to layout design rules after generating the mask layout.

At1605, the method includes adjusting the layout. For example, the method, upon detection of noncompliance with the layout design rules by any of the macros, may modify the mask layout by modifying each of the noncompliant macros to comply with the layout design rules.

At1607, the method includes generating a new layout design. For example, the method may generate a mask according to the modified mask layout with the set of features for the layer of the integrated circuit. Then, the integrated circuit layer according to the mask may be manufactured.

Although certain embodiments of the present disclosure have been described in the detailed description of the present disclosure, the present disclosure may be modified in various forms without departing from the scope of the present disclosure. Thus, the scope of the present disclosure shall not be determined merely based on the described embodiments, but rather determined based on the accompanying claims and equivalents thereto.