Patent ID: 11881038
Assignee: TSINGHUA UNIVERSITY
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 6:
7. The method of claim 5, wherein:
the calculation process of the hidden representation H is H=LN(Z+FFD(Z)),
wherein LN means a layer normalization operation, and Z+FFD(Z) means a residual connection,
the calculation process of a feed forward network module is FFD(Z)=conv2(φ(conv1(Z))),
conv1 and conv2 denote multi-channel 1-dimensional convolutions with the kernel size of 1×1, and φ represents an activation function of a Gaussian error linear unit (GELU), and
the calculation process of Z is

Z=conv3(Y1)∥conv4(Y2)∥conv5(Y3)

Y1=LN(X+MEALocal(X))

Y2=LN(X+MEANeighbor(X))

Y3=LN(X+MEAGlobal(X)), and

wherein conv3, conv4, and conv5 are 1×1 convolution operations in which the number of channels is D/3, where D is the channel number of the original feature maps, the symbol “∥” means a channel-dimension concatenation operation, MEALocal, MEANeighbor, and MEAGlobal mean a local attention module, a neighboring attention module, and a global attention module, respectively, and X+MEALocal(X), X+MEANeighbor(X), and MEAGlobal(X)are residual connections.