COMMUNICATION SYSTEMS

Examples relate to machine readable storage storing instructions arranged, when processed, to realise feedback code encoding and decoding of a source bitstream using attention neural networks.

BACKGROUND

The present application claims priority from UK patent application no. 2207152.6, filed May 16, 2022; the content of which is incorporated herein by reference for all purposes.

Feedback codes are a class of error correction codes that protect a message transmitted from a terminal A to another terminal B over a noisy communication channel. In contrast to classical forward error correction codes, feedback codes leverage the feedback signal from terminal B to terminal A to aid encoding at terminal A, and the encoding proceeds iteratively such that each transmitted symbol depends not only the intended message, but also all the feedback signals received so far.

DETAILED DESCRIPTION

The following notation is used:bold lower case and capital bold letters denote vectors and matrices respectively, i.e., v and V,capital calligraphic letters, e.g., V to denote a set corresponding to V,subscript indexing is used to identify the particular indices in a vector or a matrix, i.e., viis the ithelement of vector v and vi:jis a vector containing the elements of the vector v from the ithindex to the jthindex,v(S)refers to a sub-vector of v that contains the elements in the given set of indices S, for a given matrix V,for a matrix subscript refers to a particular row, e.g., Viis the ithrow of matrix V,a superscript for a vector or matrix represents an index for time/iteration, i.e., v(t), particularly when it is changing over time/iterations,stands for the set of real numbers andstands for the real Gaussian distribution.

FIG.1illustrates a schematic view of a point to point communication system100comprising two nodes A102and B104. The two nodes102and104are a transmitter and a receiver, respectively.

The communication goal is to deliver a vector of K bits b∈{0,1}K103reliably from node A102to node B104. Therefore, nodes A102and B104are arranged to communicate over T interactions. In the τthinteraction, τ=1, 2, . . . , T, node A102transmits a packet of q(τ)symbols c(τ)to node B104over the forward channel. In turn, node B104feeds back a packet of {tilde over (q)}(τ)symbols {tilde over (c)}(τ)to node A102over the feedback channel. In the examples described herein, it will be assumed that q(τ)={tilde over (q)}(τ)=q, ∀τ.

Node A102transmits, via the forward channel106, a vector of symbols c∈q110. The received signal or symbols at Node B104is given by y=c+n112, where n∈q114is a vector of Additive White Gaussian Noise (AWGN), the elements of which have a Gaussian distribution(0,σf2) in an independent and identically distributed (i.i.d.) manner.

Node B104feeds back a vector of symbols {tilde over (c)}∈q116to node A102. The received signal or received symbols, at node A102, are given by {tilde over (y)}={tilde over (c)}+ñ118, where the elements of the AWGN ñ∈q120have a Gaussian distribution(0,σb2) in an independent and identically distributed (i.i.d.) manner.

Nodes A102and B104are arranged to operate at a code rate

where N=Tq such that

In the examples described, nodes A102and B104can be subject to average power constraints P and {tilde over (P)} respectively:

where E is the expectancy. Therefore, the signal-to-noise ratio (SNR) of the feedforward106and feedback108channels are respectively given by

Node A102comprises an encoder122, and an accumulator124. The encoder122is arranged to produce the packet of q(τ)symbols c(τ)110. The encoder122produces the q(τ)symbols c(τ)110from a number of inputs. Examples can be realised in which the number of inputs comprise an information vector Q(τ)126. The information vector126is also known as a knowledge vector. The information vector Q(τ)126is derived from the accumulator124and the bitstream b103to be modulated. The accumulator124receives the feedback symbols {tilde over (y)}={tilde over (c)}+ñ118and constructs the information vector Q(τ)126for processing by the encoder122as follows:

Node B104comprises an accumulator128. The accumulator128is arranged to produce the packet of {tilde over (q)}(τ)symbols {tilde over (c)}(τ)116. The accumulator122produces the packet of {tilde over (q)}(τ)symbols {tilde over (c)}(τ)116from at least one input. Examples can be realised in which the at least one input comprises an information vector {tilde over (Q)}(τ)130. The information vector130is also known as a knowledge vector. The information vector {tilde over (Q)}(τ)130is derived by accumulator128from the received signals y(1), . . . , y(τ)112. Examples can be realised, using active feedback, in which the information vector {tilde over (Q)}(τ)130is derived from previous feedback symbols {tilde over (c)}(1), . . . , {tilde over (c)}(τ-1)as well as the received signals y(1), . . . , y(τ)112. Examples that provide active feedback can be provided in which node B comprises an encoder132. The encoder132is arranged to derive the feedback symbols {tilde over (c)}(1), . . . , {tilde over (c)}(τ)116from the information vector {tilde over (Q)}(τ)130. Therefore, examples, in the case of active feedback, can be realised in which the information vector {tilde over (Q)}(τ)130is given by

Examples, in the case of passive feedback, can be realised in which the information vector {tilde over (Q)}(τ)130is given by

The encoders128and132are arranged to encode the forward channel and feedback channels via respective encoding mechanisms M(τ)and {tilde over (M)}(τ), which, for each communication block τ, are realised via, respectively,

Node B104also comprises a decoder134. The decoder134is arranged to predict the original bitstream b103from the information vector {tilde over (Q)}(τ)130following a decoding mechanism D given by

As indicated above, the encoder132at node B104actively processes the information vector {tilde over (Q)}(τ)130via {tilde over (M)}(τ)to generate the vector of symbols {tilde over (c)}(τ)116transmitted to node A102over the feedback channel108. In the case of examples that use passive feedback, the encoder132at node B104implements a relay mechanism in which the information vector {tilde over (Q)}(τ)130, via {tilde over (M)}(τ), generates the vector of symbols {tilde over (c)}(τ)116transmitted to node A102over the feedback channel108using

where α is a scalar that can be used to scale the received vector y(τ)to the above average power constraint, if imposed. In the cases of passive feedback, {tilde over (q)}(τ)=q(τ).

Referring toFIG.2, there is shown a view200of the structure of the encoder122and decoder134. The encoder122has a structure that is the same as the structure of the encoding layer in “Attention is all you need”, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume30. Curran Associates, Inc., 2017. Therefore, the encoder122comprises an encoding block202. The encoding block202comprises multiple encoding layers204. In the example depicted, the encoding block202comprises ds2sencoding layers204, where ds2s≥1. Each encoding layer204comprises a layer normalisation layer206, a self-attention neural network208, a further layer normalisation layer210, and a linear-ReLu-Linear layer212.

A further layer normalisation layer214is provided to receive the bitstream b103. The output of layer214is coupled to a position encoder216. The output of the Linear-ReLu-Linear layer212is coupled to a further layer normalisation layer218.

The encoder layers202ofFIG.2are arranged to map respective sequences of l vectors of size dininto respective sequences of vectors of size dout. The encoder layers are also known as sequence to sequence neural networks that implement attention-based neural encoding. Therefore, the encoding layer realises the following mapping: Hencoder:V=(V1, . . . , Vl)→{tilde over (V)}=({tilde over (V)}1, . . . , {tilde over (V)}l) such that Vi∈din, {tilde over (V)}i∈dout. It can be appreciated that relative to the encoder layer of “Attention is all you need”, the encoder122comprises an additional two layers; namely, a first feature extractor layer220, which comprises the normalisation layer214and the position encoding layer216, and a symbol mapping layer222. The feature extractor layer220extracts a feature vector of size dinfrom the feature matrix Q(τ). The symbol mapping layer222maps the output of the encoding block202into a vector c(τ)of length q symbols for transmission to node B104.

An example implementation of systematic, passive, feedback encoding and decoding will be described with reference toFIG.3, which is known as Block Attention Feedback (BAF) coding. During an initial phase, a predetermined modulation is used for transmitting the bitstream b103to node B104. More particularly, BPSK modulation is employed to transmit the original bitstream b103to node B104. A second phase generates the symbols using a deep neural network. The overall flow of such block attention feedback coding is described below in Algorithm 1, BAF code, as depicted inFIG.4.

Algorithm 1 BAF code:1. Transmitter side/Node A:2. Phase 0:3. Send BPSK modulated bit streamM(1):Q(1)=b⟶BPSKc(1)=b_=2*b-1⁢(15)4. Phase 1:5. Receive the feedback symbols and execute the IPSE algorithm, ofAlgorithm 2 or FIG. 156. Receive side/Node B:7. Relay the received block of noisy symbols to Node A,M~(τ):Q~(τ)⟶relayc~(τ)=α⁢y(τ)+α⁢n(τ).(14)

Although the above uses BPSK modulation to send the bitstream, examples are not limited thereto. Examples can be realised in which some other form of higher order modulation is used such as, for example, QPSK, 8-PSK, QAM etc. Since the encoder block comprises ds2slayers, the parity symbols can be generated in parallel by dividing the bitstream b103into multiple blocks with each layer of the encoder block being arranged to process a respective block of the bitstream b130. The process of encoding the bitstream b130is shown inFIG.3.

Referring toFIG.3, there is shown a view300of parallel encoding of the bitstream b130. The bitstream b130is divided into l blocks302to306each of m bits such that l·m=K. Feedback symbols corresponding to each block are appended to respective blocks to form input vectors308to312to respective feature extractor layers314to318. The feature extractor layers each correspond to the above-described feature extractor layer220. The feature extractor layers outputs are fed to respective encoding blocks of the depicted encoding blocks layers320to322. The encoding blocks layers320to322feed multiple possible symbols into respective fully connected layers324to328. The fully connected layers324to328reduce the multiple possible symbols to the symbol vector c(τ)for transmission to node B104. The symbol vector c(τ)comprises symbols330to334from each of the fully connected layers324to328. The fully connected layers324to328correspond to the above-described layer normalisation layer218or the mapping layer222that implements Hmapas will be described below.

It will be appreciated that a total of l=K/m symbols are transmitted at each iteration corresponding to the l blocks of information bits of the bitstream b130. The foregoing is repeated until n parity symbols have been transmitted for each block. It will be appreciated that the above gives a coding rate of R=m/(m+n) and requires

communication blocks. Therefore, the rate of a code can be adjusted by changing the block size m and the number of parity symbols per block n. Repeating the process results in iterative party symbol encoding, which is presented in Algorithm 2 below and also illustrated in, and described with reference to,FIG.15.

At line 1, a for loop is established so that a single parity symbol is generated per block at each pass. The knowledge vector Q(τ)is updated using the bitstream103, the parity symbols transmitted thus far and the received feedback symbols or signals received thus far. The knowledge vector Q(τ)is pre-processed at steps 4 to 13, as will be described below. Feature extraction occurs at lines 14 to 16, attention-based neural-encoding is implemented at line 17 and symbol mapping is determined at lines 19 and 20.

Referring to line 5, Se(⋅) defines how the knowledge vector is pre-processed and fed to the deep neural network (DNN) architecture. Firstly, Se(⋅) generates l equal-sized knowledge vectors, i.e., Se(Q(τ))={Qi(τ), . . . , Ql(τ)}, each of which corresponds to respective different blocks.

The above Algorithm 2 presents four ways to pre-process the knowledge vector, which are expressed in lines 7, 9, 11, and 13.

A first way to pre-process the knowledge vector is given in line 7, in which the knowledge vector Q(τ)is arranged to comprise a current or respective block, b((i-1)*m+1:i*m), of the bitstream b together with the thus far, or current, received feedback signals {tilde over (y)}i(1), . . . , {tilde over (y)}i(τ-1).

A second way to pre-process the knowledge vector Q(τ)is given in line 9. It should be noted that node A102, by subtracting the vector of the symbols c(τ)110from the received noisy version of the feedback symbols {tilde over (y)}(τ)gives a cumulative noise vectorn(τ), that is,n(τ)={tilde over (y)}(τ)−c(τ)=n(τ)+ñ(τ). Including the noise,n(τ), associated with feedback symbols as part of the knowledge vector Q(τ)is known as the disentanglement of the feedback network. An example can be realised, as will be appreciated from line 9 of Algorithm 2, in which only the cumulative noise vector ñ(τ)is added to the knowledge vector Q(τ). An advantage of disentanglement is that it allows flexibility in expressing accumulated estimated noise realisations, which supports improved noise suppression that, in turn, supports improved feature extraction.

A third way to pre-process the knowledge vector Q(τ)is given in line 11. In the third example, the knowledge vector is constructed to comprise a current or respective block, b((i-1)*m+1:i*m), of the bitstream b together with the thus far, or current, transmitted symbols ci(1), . . . , ci(τ-1)and the above-described cumulative noise vector {tilde over (y)}(τ)−c(τ)=n(τ)=n(τ)+ñ(τ)={tilde over (y)}i(1)−ci(1), . . . , {tilde over (y)}i(τ)−ci(τ-1).

A fourth way to construct the knowledge vector Q(τ)is given in line 13. In the fourth example, the knowledge vector is constructed to comprise a current, or respective, block, b((i-1)*m+1:i*m), of the bitstream b together with the thus far, or current, transmitted symbols ci(1), . . . , ci(τ-1)and together with the thus far, or current, received feedback signals {tilde over (y)}i(1), . . . , {tilde over (y)}i(τ-1).

The encoder architecture depicted inFIG.2comprises a sequence of multiple encoder layers of the transformer architecture described in “Attention is all you need”. The structure of a single encoder layer is shown inFIG.2, which comprises three aspects that are a path224that has no transformations, a multi-head attention module208, and the layer normalisation layer or module210. An example of the feedforward layer212is shown inFIG.4B. The feedforward layer212comprises two fully connected layers (FC)402B and404B with a ReLu activation layer406B between the two fully connected layers402B and404B.

Examples of the encoder can be realised in which the layer normalisation module210can be implemented with different orders, in particular, examples can provide either post-normalisation or pre-normalisation. Pre- and post-layer normalisation are described in detail in

Examples can be realised in which the encoder uses pre-layer normalisation to stabilise gradient flow. Locating the layer normalisation inside any residual blocks allows training to be performed without a warm-up stage and supports faster training convergence.

It will be appreciated also that the mask that is used in transformers as described in “Attention is all you need” is not used in the example encoders and decoders described herein. The ability to remove the use of masks follows from not using sequential processing in the input. Furthermore, the examples described herein feature extractor220and symbol mapper222are realised as fully connected layers.

Examples of the decoder134have an identical architecture to examples of the encoder122with the exception that decoding is not performed in an iterative manner.

Although examples can be realised that use a two-phase communication protocol, such as described above in Algorithm 1, examples are not limited to such an arrangement. Examples can be realised in which the first phase in which the modulated symbols corresponding to the bitstream b130are not transmitted in advance of the generating symbols using feedback. In such examples, node A102directly communicates parity symbols and receives corresponding feedback symbols. Examples of the Block Attention Feedback codes that do not use an initial phase will be known as Generalised Block Attention Feedback (GBAF) codes.

For GBAF codes, since the initial phase is removed, examples use T=m+n communication blocks to obtain the same transmission or coding rate of R=m/(m+n). In examples of GBAF, all iterations use the IPSE algorithm, that is, Algorithm 2, including when τ=1, such that line 1 of Algorithm 2 becomes

for τ=1, . . . , T do # Generate 1 parity symbol per block at each pass.

Referring again toFIG.4, for examples that use a single phase, that is, for examples of GBAF, feature extraction by the feature extractor220can be improved, especially in a low SNR regime of the forward channel, i.e., for an SNR regime of −1 dB. Feature extraction is performed on the knowledge vector. Accordingly, examples can be realised that use a number of fully connected layers together with an activation layer. In the example depicted inFIG.4, two fully connected layers402and404are used together with an activation layer to reduce or prevent large noise realisations from dominating, or otherwise interfering with extracted features. Examples can be realised in which the activation layer is a Rectified Linear Unit (ReLu) layer or a Gaussian Error Linear Unit (GeLu) layer406. Noise suppression can, therefore, be improved by having a number of layers to suppress noise. Although the example depicted inFIG.4uses two fully connected layers402and404with an activation layer disposed in between, examples are not limited to such an arrangement. Examples can be realised in which a number of activation layers are disposed between respective pairs of fully connected layers.

Referring toFIG.5, there is shown a view500of an example of the accumulator124generating the knowledge vector or matrix Q(τ)126. In the examples depicted inFIGS.5to11, the following assumptions prevail: K=51, N=153, m=3, l=┌K/m┐, q=l=17, T=┌N/q┐=9 and that Disentanglement applies. It will be appreciated that examples are not limited to the above values. Examples can equally well be realised that use other values.

The accumulator124outputs the feature matrix Q(τ)126of dimension

to the encoder122. The feature matrix Q(τ)126comprises:

the bitstream b103is divided into l=17 blocks each having a length of m=3, which gives a vector Fb=[s1, s2, . . . sl]502,

a symbols vector of previously transmitted symbols

504, and

a vector of cumulative noise vectors

The feature matrix Q(τ)126is fed to the encoder122.

Referring toFIG.6, there is shown a view600of an example of the encoder122generating a plurality of coded symbols110from the knowledge matrix Q(τ)126output by the accumulator124.

The feature matrix Q(τ)126is input to the feature extractor220. The feature extractor neural network220produces an extracted features matrix V(τ)∈Rbs×17×32602. Further detail on the structure of the feature extractor neural network is given inFIG.7. The extracted features matrix602is input to a sequence to sequence neural network604, Hs2s, as described above with reference toFIG.2. The structure of the sequence to sequence neural network604is given inFIG.7. The sequence to sequence neural604network is arranged to process the extracted features matrix V(τ)602to produce a possible symbol matrix W(τ)∈Rbs×17×32606. The possible symbol matrix WM606comprises multiple sets of possible symbols for transmission to node B104. Each set comprises multiple possible symbols for each symbol to be transmitted to node B104. Examples can be realised in which bs=8192. However, examples are not limited to bs=8192. Examples can be realised in which bs takes other values depending on the available computational resources.

The possible symbols matrix WM606is output to a symbol mapping neural network222Hmapper. The structure of the symbol mapping neural network222Hmapperis described in greater detail with reference toFIG.7. The symbols mapping neural network222maps the multiple sets of possible symbols in the possible symbol matrix W(τ)606into the coded symbols110for transmission to node B104. The symbols mapping neural network222maps the multiple sets of possible symbols to respective coded symbols110using softmax encoding.

The code symbols110can be transmitted to node B104without further processing. However, preferred implementations also provide at least one, or both, of power normalisation and power reallocation as described above.

Referring toFIG.7, there is shown a view700of an example of the functional elements of each of the feature extractor neural network220, the sequence to sequence neural network604and the symbol mapping neural network222.

In the example depicted inFIG.7, the features extractor neural network comprises a number of linear layers702to706and a number of activation layers708to710. Examples are realised in which there are bs or q instances of the feature extractor neural network; each instance is arranged to process a respective feature matrix. Therefore, in training, bs instances of training are performed in parallel, that is, there are bs instances of Q(τ)in one training.

In the example shown, the linear layers702to706are fully connected linear layers and the activation layers708to710use GeLu activation functions. Although the example illustrated inFIG.7uses GeLu activation functions, examples can be realised in which other activation functions are used such as, for example, the above-described ReLu activation functions. Furthermore, although the example shown inFIG.7uses three linear layers702to706and two activation layers708to710disposed in between the linear layers702to706, examples are not limited to such an arrangement of layers, or to such a number of linear layers or such a number of activation layers.

In the example shown inFIG.7, a first linear layer702has dimensions 19×64. Although the first layer702has 64 output nodes, examples are not limited to such an arrangement. Examples can be realised in which some other number of output nodes is used to achieve a desired balance between accurate modulation and demodulation of data and processing power required for at least one of training and implementation. It will be appreciated that the notionbsis an indication of the number of instances of Hextractorthat are arranged in parallel.

There are bs instances of the feature extractor neural network220. The output712of the, or of each instance of the, first linear layer702is a matrix or tensor having dimensionsbs×17×64. Each of the matrices of dimensionbs×17×64is fed into respective instances of the first activation layer708. Each of the inputs to the activation layer is passed through a respective activation function. In the example depicted, the activation function is a GeLu activation function. The first activation layer708comprises l, where l=17 in the present example, GeLu activation functions. The output714of the first activation layer708is a matrix or tensor having dimensionsbs×17×64that is, bs instances of 2D matrices of dimension17×64.

The output714matrix is input into a second linear layer704. The second linear layer comprises a 64×64 neural network that produces an output matrix716having dimensionsbs×17×64. The output matrix716is fed into the second activation layer710, where each input is subjected to a GeLu activation function. The output718of the second activation layer710is also a matrix of dimensionsbs×17×64.

The output718of the second activation layer710is input into the third linear layer706. The third linear layer comprises a 64×32 neural network that produces an output matrix720having dimensionsbs×17×32. The output matrix720corresponds to the above-described extracted features matrix V(τ)∈bs×17×32.

Still referring toFIG.7, there is shown an instance or example of the sequence to sequence neural network Hs2s604. The sequence to sequence neural network604comprises multiple encoding blocks722to724. In the example depicted inFIG.7, two encoding blocks722and724are illustrated. The encoding blocks are realised as neural networks and are examples of the above-described encoding blocks202. A first encoding block722takes the extracted features matrix V(τ)602as an input and produces an output matrix726having dimensionsbs×17×32. The output matrix726is fed as an input into the second encoding block724. The second encoding block produces an output matrix728having dimensionsbs×17×32. The output matrix728corresponds to the above-described possible symbols matrix WM606.

The possible symbols matrix W(τ)606is fed to the symbols mapper neural network222. The symbols mapper neural network222comprises a linear layer neural network730. The linear layer neural network730has dimensions 32×2 and produces an output matrix732of coded symbols. The matrix732corresponds to the above-described matrix of coded symbols110.

Referring toFIG.8, there is shown a view800of the processing and output performed by the accumulator128at node B104. The accumulator128is arranged to generate the above-described knowledge matrix Q(τ)130. Again, the above-described assumptions prevail, that is, K=51, N=153, m=3, l=┌K/m┐, q=l=17, T=┌N/q┐=9 and that Disentanglement applies. Therefore, it can be appreciated that the knowledge matrix {tilde over (Q)}(τ)130has dimensions {tilde over (Q)}(τ)∈Rbs×l×T. The knowledge matrix is given by

The knowledge matrix {tilde over (Q)}(τ)130is output to the decoder neural network134.

Referring toFIG.9, there is shown a view900of the decoder neural network134. The decoder neural network134is arranged to process the received knowledge matrix {tilde over (Q)}(τ)130having dimensions l=17802and T=9804. The decoder neural network134comprises a feature extraction neural network902, a sequence to sequence neural network904and a symbol mapping neural network906. The structure or architecture of the decoder neural network134is identical to the above-described encoder neural network122. The dimensions of the neural networks are given inFIG.10.

The feature extraction neural network902processed the knowledge matrix {tilde over (Q)}(τ)130to generate an extracted features matrix {tilde over (V)}∈bs×17×19908. The extracted features matrix {tilde over (V)}908is processed by the sequence to sequence neural network904to produce a candidate symbol matrix {tilde over (W)}∈bs×17×32910. The candidate symbol matrix {tilde over (W)}910comprises a plurality of candidate symbols. The candidate symbol matrix {tilde over (W)}910is processed by the symbol mapping neural network906to produce a decoded bitstream vector {circumflex over (b)}∈bs×51×2912containing estimates of the initially transmitted bitstream b103. It will be appreciated that the actual dimension of {circumflex over (b)} is {circumflex over (b)}∈bs×51×1. However, since for each bit, a binary distribution (p,p−1) is generated, the output of the neural network906is {circumflex over (b)}∈bs×51×2from which {circumflex over (b)}∈bs×51×1is decoded.

Returning toFIG.8, it can be seen that the output of the decoder134can be reshaped to give the estimated bitstream {circumflex over (b)}912.

Referring toFIG.10, there is shown a view1000of an example of the functional elements of each of the feature extractor neural network902, the sequence to sequence neural network904and the symbol mapping neural network906.

In the example depicted inFIG.10, the features extractor neural network902comprises a number of linear layers1002to1006and a number of activation layers1008to1010. Examples are realised in which there are bs instances of the feature extractor neural network; each instances is arranged to process a respective feature matrix.

In the example shown, the linear layers1002to1006are fully connected linear layers and the activation layers1008to1010use GeLu activation functions. Although the example illustrated inFIG.10uses GeLu activation functions, examples can be realised in which other activation functions are used such as, for example, the above-described ReLu activation functions. Furthermore, although the example shown inFIG.10uses three linear layers1002to1006and two activation layers1008to1010disposed in between the linear layers1002to1006, examples are not limited to such an arrangement of layers, or to such a number of linear layers or such a number of activation layers.

In the example shown inFIG.10, a first linear layer1002is a neural network that has dimensions 9×64. There are bs instances of the feature extractor neural network902. The output1012of the, or of each instance of the, first linear layer1002is a matrix having dimensionsbs×17×64. That matrix having dimensionsbs×17×64is fed into the first activation layer1008. The first activation layer1008comprises l, where l=17 in the present example, GeLu activation functions. The output1014of the first activation layer1008is a matrix having dimensionsbs×17×64.

The output1014matrix having dimensionsbs×17×64is input into a second neural network linear layer1004. The second linear layer comprises a 64×64 neural network that produces an output matrix1016having dimensionsbs×17×64. The output matrix1016is fed into the second activation layer1010, where each input is subjected to a GeLu activation function. The output1018of the second activation layer1010is also a matrix of dimensionsbs×17×64.

The output1018of the second activation layer1010is input into the third linear layer1006. The third linear layer comprises a 64×32 neural network that produces an output matrix1020having dimensionsbs×17×32. The output matrix1020corresponds to the above-described extracted features matrix {tilde over (V)}(τ)∈bs×17×32908.

Still referring toFIG.10, there is shown an instance or example of the sequence to sequence neural network {tilde over (H)}s2s904. The sequence to sequence neural network904comprises multiple encoding blocks1022to1025. In the example depicted inFIG.10, three encoding blocks are illustrated. The encoding blocks are realised as neural networks and are examples of the above-described encoding blocks202. A first encoding block1022takes the extracted features matrix {tilde over (V)}(τ)908as an input and produces an output matrix1026having dimensionsbs×17×32. The output matrix1026is fed as an input into the second encoding block1024. The second encoding block produces an output matrix1028having dimensionsbs×17×32. The output matrix1028is fed as an input into the third encoding block neural network1025and produces an output matrix1029having dimensionsbs×17×32. The output matrix1029corresponds to the above-described possible symbols matrix {tilde over (W)}(τ)910.

The possible symbols matrix W(τ)910is fed to the symbols mapper neural network906. The symbols mapper neural network906comprises a linear layer neural network1030, a reshape function1031and a softmax function1032. The linear layer neural network1030has dimensions 32×6 and produces an output matrix1034of coded symbols having dimensionsbs×17×6. The reshaping function1031processes the output matrix1034to produce a reshaped matrix. The reshaped matrix comprises a rearrangement of the values of the output matrix1034to produce an output matrix1036of candidate decoded symbols. The output matrix1026has dimensionsbs×51×2. Each instance of the output matrix1034of the possible decoded symbols two values is processed by the softmax function1032to generate a matrix1038of decoded symbols. The matrix1038corresponds to the above-described matrix of decoded symbols912.

It can be appreciated from the above that the neural-encoder at node A102performs simultaneously two tasks; namely, keeping track of a current belief regarding the original bits at the receiver and generating symbols accordingly in order to refine the belief. It can be appreciated that the above-described GBAF uses a single network for both tasks.

Referring again toFIG.1, it can be seen that there is disclosed a belief network. The structure and function of the belief network will be described with reference toFIG.11. Providing a belief network increases the information processing capacity of the network layer by adding additional self-attention layers with the specific task of keeping track of the current belief at the receiver. An architecture that uses belief feedback is known as a GBAF with belief feedback.

It will be appreciated from the above GBAF that the feedback information (parity symbols and combined noise values) and the original bitstream (modulated or unmodulated) are processed simultaneously. However, examples using belief feedback support learning the residual error between the original bits of the bitstream and the prediction at node B104based on symbols received so far. Therefore, examples can be realised that add another deep neural network for generating a belief vector on the original bitstream based on the feedback information comprising the parity symbols and the combined noise values. The belief vector can be concatenated with the vector of the original unmodulated bitstream103and conveyed to the encoder as part of the information vector or knowledge vector Q(τ)126

Referring toFIG.11, there is shown a view1100of a belief network1102according to examples. The deep neural network architecture used for generating beliefs is identical to the one used for generating parity symbols. Any of the examples described herein can be realised with or without a belief network1102. The belief deep neural network1102is a neural network for generating a matrix of beliefs B(τ)∈bs×17×321104. The belief neural network1102comprises three main parts; namely, a feature extractor neural network1106, a belief sequence to sequence neural network1108and a belief mapping neural network1110.

The feature extractor neural network1106has an input matrix {tilde over (Y)}(τ)∈bs×17×81112, that is, {tilde over (Y)}(τ)∈i.e. the totality of all feedback from node B104, that is, all received feedback signals comprising feedback symbols and noise, where

The feature extractor neural network1106processed the input matrix {tilde over (Y)}(τ)1112to produce a feature matrix {tilde over (V)}′∈b×17×321114. The feature matrix {tilde over (V)}′1114is input into the belief sequence to sequence neural network1108. The belief sequence to sequence neural network1108processes the feature matrix {tilde over (V)}′1114to produce a matrix of candidate beliefs {tilde over (W)}′∈bs×17×321116. The matrix of candidate beliefs {tilde over (W)}′1116forms an input to the belief mapping neural network1110. The belief mapping neural network1110processes the matrix of candidate beliefs {tilde over (W)}′1116to form the matrix of beliefs B(τ)∈bs×17×321104. The architecture for the belief neural network1102is almost identical to the architecture for the encoder122described above with the exception that the belief mapping neural network1110uses an extra softmax layer to generate the beliefs in the form of output probabilities.

The matrix of beliefs B(τ)∈bs×17×321104is fed to the accumulator124of node A102to form part of, or be used with, the information vector or knowledge vector Q(τ)126, in particular, pre-processing of the knowledge vector is given by

Se(Q(τ),B(τ))={Qi(r), . . . , Ql(r)}. The overall architecture for feedback encoding and decoding incorporating beliefs is known asUnified Iterative Parity Symbol Encoding(UIPSE) and is shown below in detail in Algorithm 3.

Again, it can be appreciated that a for loop for τ=1, . . . , T is established, that is, a parity symbol is generated per block at each pass. The knowledge vector Q(τ)126is updated at line 3 as Q(τ)=[b, c(1). . . , c(τ-1), {tilde over (y)}(1), . . . , y(τ-1)]. A determination is made at line 4 whether or not belief feedback is enabled. If belief processing is not enabled, processing continues at line 29, where Algorithm 2 is implemented. If belief feedback is enabled, processing continues with lines 5 to 27 as follows. At line 6, the knowledge vector Q(τ)126vector is established for the belief network as Sb(Q(τ))={{tilde over (Q)}i(τ), . . . , {tilde over (Q)}l(τ)} such that {tilde over (Q)}i(τ)=[{tilde over (y)}i(τ), . . . , {tilde over (y)}i(τ-1)]. The features of the input vector1112are extracted at line 8 by Vi(τ)=Hextractbelief({tilde over (Q)}i(τ)). Attention-based neural encoding, that is, sequence to sequence encoding, is realised at line 9 using

{tilde over (V)}belief(τ)=Hencoderbelief(Vbelief(τ)). The belief feedback is generated at line 10 using Bi(τ)=Hmapbelief({tilde over (V)}i(τ)). The information vector or knowledge vector Q(τ)126is pre-processed at line 12 such that Se(Q(τ), B(τ))={Qi(τ), . . . , Ql(τ)}, where Qi(τ)is established according to one of the following conditions: feedback only, noise only, disentanglement or beliefs, symbols and feedback. If feedback only is selected, Qi(τ)is given by

Referring again toFIG.1, a unified architecture for feedback encoding is presented including all features identified using dashed lines, which are options in the most basic examples of feedback encoding.

The unified architecture ofFIG.1comprises a plurality of feedback mechanisms; namely inner feedback, outer feedback, and belief feedback.

Inner feedback refers to the process of using generated parity symbols as inputs to the encoder122at consecutive iterations as will be described with reference toFIG.12hereafter. The objective of inner feedback is to enable the encoder to recall or use previously transmitter symbols.

Outer feedback refers to the feedback channel information received at node A102from node B104, which enables the encoder to track noise realisations.

Belief feedback is the output of the additional deep neural network employed at node A102that is used to track node B's belief about the bitstream after each transmission block.

It will be appreciated that enabling and disabling the belief feedback supports switching between variations of GBAF and GBAF-BF. Still further, examples can be realised that disable or enable the inner or outer feedback mechanisms as well, which supports realising different variations of the unified GBAF. Therefore, examples can be realised in which the encoder comprises a selectable plurality of feedback mechanisms to support feedback encoding of a bitstream, the selectable plurality of feedback mechanisms comprising at least one of the following, taken jointly and severally in any and all permutations: inner feedback comprising processing generated parity symbols as inputs to the encoder, outer feedback comprising processing feedback channel to determine noise associated with at least one, or both, of a feedforward channel and a feedback channel, and belief feedback comprising data associated with the bitstream at a receiver after each transmission block.

Referring toFIG.12, there is shown a view1200of an iterative decoder1202for feedback encoding and decoding. The decoder1202comprises an initial decoding module1204and an iterative decoding module1206. The initial decoding module1204is invoked once to map received parity symbols1208to1212for each block to a latent representation1214to1218.

The iterative decoding module1206is invoked multiple times and is arranged to use previous decoding outputs1220to1224as inputs to the iterative decoding process by concatenating the latent representations1214to1218and the previous decoding outputs1220to1224. The iterative decoding process forms a belief propagation mechanism through a multi-layer attention encoder1226. The iterative decoding module1206comprises multiple fully connected layers. In the example depicted inFIG.12, the multiple fully connected layers comprise a number of input fully connected layers1228to1232, and a number of fully connected output layers1234to1338.

It will be appreciated that the iterative decoding module1206uses the output of the output fully connected layers1234to1238as beliefs to refine predictions in a manner similar to recurrent neural network architectures. The fully connected layers1228to1238are arranged to align the sizes of the latent representations1214to1218.

In the example described, there are two layers, that is, there are two encoders. However, examples are not limited to two such layers. Examples can be used in which two or more than two such layers are used. Furthermore, the iterative decoding module1206can be invoked multiple times. Examples can be realised in which the iterative decoding module1206can be invoked three times. However, examples are not limited to the iterative decoding layer1206being involved three times. Examples can be realised in which the iterative decoding layer1206is involved two or more times. Accordingly, examples can be realised in which the iterative decoding module1206comprises a plurality of fully connected layers and a multi-layer attention encoder1226that are invoked two or more times.

It will be appreciated fromFIGS.7and10that the encoder and decoders use different numbers of encoding blocks or encoding layers. In the examples described, the encoder comprises two encoding blocks or layers and the decoder comprises three encoding blocks or layers. Accordingly, examples can be realised in which the encoder comprises a respective number encoder encoding blocks or layers and the decoder comprises a respective number of decoder encoding blocks or layers. The respective number encoder encoding blocks or layers and the respective number of decoder encoding blocks or layers can be the same or different. Examples can be realised in which the number of encoder encoding blocks is greater than the number of decoder encoding blocks or layers. Alternatively, examples can be realised in which the number of encoder encoding blocks is less than the number of decoder encoding blocks or layers.

In training the neural networks of the examples, an AdamW optimizer was utilised, which is a variation of the Adam optimizer but with decoupled weight decay regularization. Also, a batch size of B=8192 was used, with an initial learning rate of 0.001 and a weight decay parameter of 0.01. Furthermore, gradient clipping was applied with a threshold of 0.5 and the neural networks were trained for 600,000 batches together with applying polynomial decay to the learning rate.

Referring toFIG.13, there is shown a view1300of BLER versus Feedforward SNR (dB) performance graphs of the examples described here compared to existing coding strategies for the same coding rate, where K=51, N=153, m=3.

The view1300shows:

a BLER performance curve1308for DRFC coding, and

a BLER performance curve1310according to GBAF coding as described herein.

It can be appreciated that the performance of GBAF is significantly better than the above prior art feedback coding techniques.

Referring toFIG.14, there is shown a view1400of a flowchart for generating training data for training the neural networks of the examples described herein.

At1402, the parameters K, N, m are selected, where K represents the number of bits in the bitstream103, N represents the total number of bits to be transmitted and m represents the number of bits per block.

At1404, a bitstream b103of K bits is generated; the K bits comprise randomly generated bits. The bitstream is reshaped, at1406, to produce a bit matrix Fb∈, where=┌K/m┐.

At1408,real symbols s=Fbζ, where ζ=[2m-1, 2m-2, . . . , 21, 20]Tare constructed for “transmission” to node B. The word “transmission” is in quotes since the channel and transmission are simulated such that actual transmission does not take place when generating the training data and using that data to train neural networks. Therefore, it will be appreciated that “transmission” within this training context means subjecting the bit matrix to a transfer function representing the channel conditions of a given channel to produce transmitted/received symbols.

Generating the bitstream103at1406and constructing thereal symbols at1408is repeated a predetermined number of times, that is, a predetermined number of interactions take place. Examples can be realised in which the predetermined number of times is governed by T=N/interactions. Accordingly, at1412, a determination is made regarding the number of interactions that have taken place thus far. If the determination at1412is that T=N/or fewer interactions have taken place, processing resumes at1406where a new bitstream is generated. If more than T=N/interactions have taken place, processing proceeds to1414, where the feature matrix {tilde over (Q)}(τ)is constructed and, at1416,real symbols ŝ∈are decoded.

Once thereal symbols ŝ∈have been decoded, the above-described encoders and decoders, that is, the above described neural networks, are trained, at1418, to minimise the error between s and ŝ.

Alternatively, during actual encoding and decoding, that is, during actual feedback encoding and decoding once the neural networks have been trained, following decoding of thereal symbols ŝ∈at1416, the estimated or decoded symbols ŝ are transformed into a corresponding bit matrix {circumflex over (F)}b∈at1420, and, at1422, the bit matrix {circumflex over (F)}bis reshaped to give a decoded or demodulated bitstream to {circumflex over (b)}∈{0,1}K×1.

Referring toFIG.15, there is shown a view1500of a flowchart for Algorithm 2. At1502, a for loop is established to step through all blocks of data to be transmitted. The for loop is for τ=2, . . . , T. At1504, the information vector or knowledge vector Q(τ)126is established or updated to give Q(τ)=[b, c(1), . . . , c(τ-1), {tilde over (y)}(1), . . . , {tilde over (y)}(τ-1)]. At1506, the knowledge vector Q(τ)126is pre-processed to generate Se(Q(τ))={Q1(τ), . . . , Ql(τ)} where Qi(τ)is selected according to which prevailing mode of operation has been selected. Examples can provide a number of modes of operations including the following: Feedback only mode, noise only mode, disentanglement mode or hybrid mode. A determination is made at1508if the current mode is the feedback only mode. If the determination is positive, Qi(τ)is established as Qi(τ)=[b((i-1*m1:i*m), {tilde over (y)}i(1), . . . , {tilde over (y)}i(τ-1)] at1510. If the determination is negative, a determination is made1512if the current or selected mode is the noise only mode. If the determination at1512is positive, Qi(τ)is established as Qi(τ)=[b((i-1)*m1:i*m), {tilde over (y)}i(1)−ci(1), . . . , {tilde over (y)}i(τ-1)−ci(τ-1)] at1514. If the determination at1512is negative, a determination is made at1516if the current or selected mode is disentanglement mode. If the determination at1516is positive, Qi(τ)is established as Qi(τ)=[b((i-1)*m1:i*m), ci(1), . . . , ci(τ-1), {tilde over (y)}i(1)−ci(1), . . . , {tilde over (y)}i(τ-1)−ci(τ-1)at1518. If the determination at1516is negative, Qi(τ)is established as Qi(τ)=[b((i-1)*m1:i*m), ci(1), . . . , ci(τ-1), {tilde over (y)}i(1)−ci(1), . . . , {tilde over (y)}i(τ-1)−ci(τ-1)at1520. Having established Qi(τ)and, therefore, Se(Q(τ))={Q1(τ), . . . , Ql(τ)}, processing continues at1522, where feature extraction is performed by processing all Qi(τ)for all i∈[l] to establish Vi(τ)=Hextract(Qi(τ)). At1524, attention-based neural processing is performed to establish V(τ)=Hencoder({tilde over (V)}(τ)). Having established V(τ), the symbols to be transmitted are established at1526for all i∈[l] as follows: ci(τ)=Hmap({tilde over (V)}i(τ)).

Referring toFIG.16, there is shown a view1600of a flowchart for Algorithm 3 relating toUnified Iterative Parity Symbol Encoding(UIPSE). At1602, a count is established to step through all blocks to be transmitted as f or τ=1, . . . , T. At1604, the information vector or knowledge vector Q(τ)126is established or updated to give Q(τ)=[b, c(1), . . . , c(τ-1), {tilde over (y)}(1), . . . , {tilde over (y)}(τ-1)]. A determination is made, at1606, regarding whether or not beliefs will be taken into account in encoding the bitstream b103. If the determination at1606is negative, processing continues at1608, where algorithm2is executed. If the determination at1606is positive, pre-processing of the knowledge vector is commenced at1610to establish Se(Q(τ))={Q1(τ), . . . , Ql(τ)} such that, at1612, {tilde over (Q)}i(τ)=[{tilde over (y)}i(1), . . . , {tilde over (y)}i(τ-1)] is established. At1614, belief features are extracted from the knowledge vector via Vi(τ)=Hextractbelief({tilde over (Q)}i(τ)). The extracted beliefs features are subjected to attention-based neural-encoding at1616to establish the candidate associated beliefs via {tilde over (V)}belief(τ=Hencoderbelief(Vbelief(τ)). The candidate associated beliefs are processed at1618to establish feedback beliefs via Bi(τ)=Hmapbelief({tilde over (V)}i(τ)). Next, at1620, the knowledge vector is pre-processed Se(Q(τ), B(τ))={Qi(τ), . . . , Ql(τ)}, where Ql(τ)is established according to one of the following conditions: feedback only, Noise only, Disentanglement or beliefs, symbols and feedback. A determination, at1622, is made regarding whether or not feedback only mode is selected. If the determination at1622is positive, Qi(τ)is established, at1624, as Qi(τ)=[b((i-1)*m+1:i*m), Bi(τ), {tilde over (y)}i(1), . . . , {tilde over (y)}i(τ-1)]. If the determination at1622is negative, a determination is made, at1626, regarding whether or not the currently selected mode is noise only. If the determination at1626is positive, Qi(τ)is established, at1628, by Qi(τ)=[b((i-1)*m+1:i*m), Bi(τ), {tilde over (y)}i(1)−ci(1), . . . , {tilde over (y)}i(τ-1)−ci(τ-1)]. If the determination at1626is negative, a determination is made, at1630regarding whether or not Disentanglement mode is selected. If the determination at1630is positive, that is, Disentanglement is enabled, Qi(τ)is given by Qi(τ)=[b((i-1)*m+1:i*m), Bi(τ), {tilde over (y)}i(1)−ci(1), . . . , {tilde over (y)}i(τ-1)−ci(τ-1)] at1632. If the determination at1630is negative, Qi(τ)is established, at1634, as Qi(τ)=[b((i-1)*m+1:i*m), Bi(τ), {tilde over (y)}i(1)−ci(1), . . . , {tilde over (y)}i(τ-1)−ci(τ-1)].

Having established the selected mode of operating and having established the knowledge vector in response, feature extraction is performed at1636for all/blocks by establishing a for loop as for i∈[l]do,Vi(τ)=Hextract(Qi(τ)), followed by Attention-based Neural-encoding at1638given by V(τ)=Hencoder({tilde over (V)}(τ)). Finally, symbol mapping is performed, at1640, via for i∈[l]do,ci(τ)=Hmap({tilde over (V)}i(τ)).

Examples can be realised in which active feedback is used to generate modulated data. Referring again toFIG.1, it can be appreciated that node B104comprises an encoder132. The encoder132is arranged to generate the feedback symbols {tilde over (c)}(1), . . . , {tilde over (c)}(τ)116from the information vector {tilde over (Q)}(τ)130. Such active feedback is described in Algorithm 4 below and implemented at node B104.

Referring to Algorithm 4, a for loop is established at line 1 so that a feedback symbol per block is generated at each pass.

A determination is made at line 2 regarding whether or not active feedback is enabled. If active feedback is not enabled, processing proceeds from line 19 where the τthfeedback symbol cτis determined from the received signal. Examples can be realised in which the τthfeedback symbol cτis determined as a scaled version of the (τ−1)threceived signal αy(τ-1). If active feedback is enabled, the information vector or knowledge vector {tilde over (Q)}(τ)130is updated at line 4 as {tilde over (Q)}(τ)=[{tilde over (c)}(1), . . . , {tilde over (c)}(τ-1), y(1), . . . , y(τ)].

At lines 6 to 10, the knowledge vector {tilde over (Q)}(τ)130is pre-processed, that is, Se(⋅) generates l equal-sized knowledge vectors, i.e., Se({tilde over (Q)}(τ))={{tilde over (Q)}i(τ), . . . , {tilde over (Q)}l(τ)}, each of which corresponds to respective different blocks, such that each knowledge vector is determined according to whether or not Parity only should be taken into account or if previously transmitted feedback symbols should be taken into account as well as parity. It can be appreciated that from the perspective of the encoder132of node B104, the received signals correspond to or represent parity symbols and the feedback symbols correspond to or represent transmitted signals.

If parity only is active, then each of the l knowledge vectors is determined from {tilde over (Q)}i(τ)=[yi(1), . . . , yi(τ)] as can be appreciated from line 8. If parity only is not active, then each of the l knowledge vectors also takes into account the previous feedback symbols such that each of the l knowledge vectors is determined from {tilde over (Q)}i(τ)=[{tilde over (c)}i(1), . . . , {tilde over (c)}i(τ-1), yi(1), . . . , yi(τ)] as can be appreciated at line 10.

The knowledge vectors {tilde over (Q)}i(τ), . . . , {tilde over (Q)}l(τ)are processed at lines 12 and 13 to extract the features {tilde over (V)}i(τ)={tilde over (H)}extractfeedback({tilde over (Q)}i(τ)). Sequence to Sequence processing, via an attention-based neural network, as described above in respect of the encoder122of node A102, is performed at line 14 via {tilde over (W)}(τ)={tilde over (H)}s2sfeedback({tilde over (V)}(τ)).

Finally, feedback symbol mapping is performed at lines 16 and 17 to establish the feedback symbols116via {tilde over (c)}i(τ)={tilde over (H)}mapfeedback({tilde over (W)}i(τ)), which are outputs for transmission to node A102.

Referring toFIG.17, there is shown a flowchart1700for implementing Algorithm 4. At1702, a for loop is established at line 1 so that a feedback symbol per block is generated at each pass. A determination is made, at1704, regarding whether or not active feedback is enabled. If active feedback is not enabled, the τthfeedback symbol cτis determined, at1706, from the received signal. Examples can be realised in which the τthfeedback symbol cτis determined as a scaled version of the (τ−1)threceived signal αy(τ-1). If active feedback is enabled, the information vector or knowledge vector {tilde over (Q)}(τ)130is updated, at1708, as {tilde over (Q)}(τ)=[{tilde over (c)}(1), . . . , {tilde over (c)}(τ-1), y(1), . . . , y(τ)].

At1710, the knowledge vector {tilde over (Q)}(τ)130is pre-processed, that is, Se(⋅) generates l equal-sized knowledge vectors, i.e., Se({tilde over (Q)}(τ))={{tilde over (Q)}i(τ), . . . , {tilde over (Q)}l(τ)}, each of which corresponds to respective different blocks, such that each knowledge vector is determined according to whether or not Parity only should be taken into account or if previously transmitted feedback symbols should be taken into account as well as parity. It can be appreciated that from the perspective of the encoder132of node B104, the received signals correspond to or represent parity symbols and the feedback symbols correspond to or represent transmitted signals.

Therefore, a determination is, at1712, if parity only is active. If the determination at1712is that parity only is active, then each of the/knowledge vectors is determined, at1714, from {tilde over (Q)}i(τ)=[yi(1), . . . , yi(τ)]. However, if parity only in not active, then each of the l knowledge vectors also takes into account the previous feedback symbols such that each of the l knowledge vectors is determined, at1716, from {tilde over (Q)}i(τ)=[{tilde over (c)}i(1), . . . , {tilde over (c)}i(τ-1), yi(1), . . . , yi(τ)].

The knowledge vectors {tilde over (Q)}i(τ), . . . , {tilde over (Q)}l(τ)are processed, at1718, to extract the features {tilde over (V)}i(τ)={tilde over (H)}extractfeedback({tilde over (Q)}i(τ)). Sequence to Sequence processing, via an attention-based neural network, as described above in respect of the encoder122of node A102, is performed, at1720, via {tilde over (W)}(τ)={tilde over (H)}s2sfeedback({tilde over (V)}(τ)).

Finally, feedback symbol mapping is performed, at1722, to establish the feedback symbols116via {tilde over (c)}i(τ)={tilde over (H)}mapfeedback({tilde over (W)}i(τ)), which are outputs for transmitting node A102.

The functionality of the system100and any parts thereof can be realised in the form of machine instructions that can be processed by a machine comprising or having access to the instructions. The machine can comprise a computer, processor, processor core, DSP, a special purpose processor implementing the instructions such as, for example, an FPGA or an ASIC, circuitry or other logic, compiler, translator, interpreter or any other instruction processor. Processing the instructions can comprise interpreting, executing, converting, translating or otherwise giving effect to the instructions. The instructions can be stored on a machine readable medium, which is an example of machine-readable storage. The machine-readable medium can store the instructions in a non-volatile, non-transient or non-transitory, manner or in a volatile, transient, manner, where the term ‘non-transitory’ does not encompass transitory propagating signals. The instructions can be arranged to give effect to any and all operations described herein taken jointly and severally in any and all permutations. The instructions can be arranged to give effect to any and all of the operations, devices, systems, flowcharts, protocols or methods described herein taken jointly and severally in any and all permutations. In particular, the machine instructions can give effect to, or otherwise implement, the operations of the algorithms and/or flowcharts depicted in, or described with reference to,FIGS.4,14,15,16and17, taken jointly and severally in any and all permutations.

Therefore,FIG.18shows a view1800of machine instructions1802stored using machine readable storage1804for implementing the examples described herein. The machine instructions1802can be processed by, for example, a processor1806or other processing entity, such as, for example, an interpreter, as indicated above.

The machine instructions1802comprise at least one or more than one of:

Instructions1810to implement an accumulator at node A102,

Instructions1812to implement a belief neural network,

Instructions1814to realise an accumulator at node B104,

Instructions1818to realise an encoder at node B104

Instructions1824to implement Algorithm 3, and

the foregoing instructions1808to1826being taken jointly and severally in any and all permutations.

Advantageously, one or more than one of the examples described herein address or otherwise solve the following limitations of existing deep neural networks:Communication overhead: In practice, each distinct use of the forward and feedback channels introduces a certain level of overhead and an additional delay independent of the number of bits transmitted. The corresponding communication overhead is defined as the number of ‘switches’ at the transmitter, between transmitting parity symbols and receiving feedback symbols. In existing schemes, each time Hd is used, only two symbols are generated and transmitted. Hence, the communication overhead scales with the length of the bit-stream K.Limited range of feasible rates: Existing schemes are limited to rates

k∈Z+.DeepCode is a systematic feedback scheme or architecture, which is limiting.

Examples can be realised in accordance with the following clauses:

Clause 1: An encoding method for a modulator of a transmitter to encode a source bitstream b∈{0,1}K×1comprising K source bits using feedback encoding; the method comprising:

constructing a feature matrix,

where Fbcomprises the source bits, the feature matrix also comprising at least selectable ones of: pairs of previously transmitted coded symbols, Fc, and estimated noise realisations, Fn, optionally, selectable ones of tuples of source bits, previously transmitted coded symbols, Fc, and estimated noise realisations, Fn, received via a-feedback signal transmitted by a receiver;

encoding the feature matrix, using attention-based neural sequence to sequence

(s2s) mapping, to generate a vector of l coded symbols, and

outputting the l coded symbols for transmitting to the receiver.

Clause 2: The method of clause 1, in which encoding the feature matrix to generate a vector of l coded symbols comprises:

preprocessing the feature matrix to extract a set of features that will influence encoding the feature matrix,

transforming (s2s), using an attention encoder, the feature matrix, Q(τ), into a sequence to establish new correlations between portions of the feature matrix using existing correlations between portions of the feature matrix, and

mapping the sequence into l coded symbols.

Clause 3: The method of clause 2, in which the transforming comprises transforming (s2s), using the attention encoder, the feature matrix, Q(τ), into the sequence to establish new column-wise correlations between columns of the feature matrix using existing column-wise correlations between columns of the feature matrix.

Clause 4: The method of any preceding clause, in which constructing the feature matrix,

comprising the source bits, previously transmitted coded symbols and estimated noise realisations received at the transmitter comprises:

generating a vector Fb∈{0,1}m×1, comprising the l groups of m source bits, Fb=[s1, s2, . . . sl].

Clause 5: The method of any preceding clause, in which constructing the feature matrix,

comprising the source bits, previously transmitted coded symbols and estimated noise realisations received at the transmitter comprises:

generating a vector Fc∈R(τ-1)×lcomprising the previously transmitted coded symbols; each row of Fccomprising c(i), for i=1, . . . , τ−1, and zero-padded for i=τ, . . . , T−1, where τ is a temporal index of order of the previously transmitted symbols;

Clause 6: The method of any preceding clause, in which constructing the feature matrix,

comprising the source bits, previously transmitted coded symbols and estimated noise realisations received at the transmitter comprises:

generating a vector of estimated noise realisations Fn∈R(τ-1)×lobserved at the feedback channel of the transmitter, such that

from the received feedback signal.

Clause 7: The method of any preceding clause in which outputting the1coded symbols for transmitting to the receiver comprises at least one, or both, of power normalisation and power reallocation to generate the l coded symbols c(τ)∈R1×l.

Clause 8: A decoding method for a demodulator of a receiver to decode a coded symbol stream comprising T symbols C(τ), τ=1, 2, . . . , T, iteratively derived from a bitstream b∈{0,1}K×1comprising K source bits arranged into l=┌K/m┐ groups of size m, such that b=[s1T, s2T, . . . , slT] using feedback provided by the receiver; the decoding method comprising:

receiving a current signal of a plurality of signals

y(τ)=c(τ)+n(τ); c(τ), n(τ)∈R1×l, comprising the T symbols; and

transmitting received symbols, c(τ), or the currently/most recently received signal, y(τ), comprising a currently/most recently received symbol, c(τ), to a transmitter associated with generating the symbols, c(τ);

constructing a feature matrix,

generating a decoded bitstream vector {circumflex over (b)}∈{0,1}K, comprising the l groups of m source bits, {circumflex over (b)}=[s1, s2, . . . sl], from the feature matrix, {tilde over (Q)}(τ), using a sequence to sequence neural network/attention neural network.

Clause 9: The method of clause 8, in which generating the decoded bitstream vector {circumflex over (b)}, comprising the l groups of m source bits, {circumflex over (b)}=[s1, s2, . . . sl], from the feature matrix, {tilde over (Q)}(τ), comprises:

preprocessing the feature matrix, {tilde over (Q)}(τ), to extract a set of features, {tilde over (V)}∈Rbsxlx, for influencing generating the decoded bitstream;

transforming (s2s), using an attention encoder, the feature matrix, {tilde over (Q)}(τ), into a sequence to using correlations between portions of the feature matrix, and

mapping the sequence into the l decoded symbols.

Clause 10: The method of clause 9, in which transforming (s2s), using an attention encoder, the feature matrix, {tilde over (Q)}(τ), into a sequence to using correlations between portions of the feature matrix comprises

transforming (s2s), using an attention encoder, the feature matrix, {tilde over (Q)}(τ), into a sequence using column-wise correlations between columns of the feature matrix.

Clause 11: The method of any preceding clause in which generating a decoded bitstream vector, {circumflex over (b)}, comprises reshaping the output from sequence to sequence neural network/attention neural network.

Clause 12: Machine readable instructions arranged, when processed, to implement a method of any preceding clause.

Clause 13: Machine readable storage storing machine readable instructions of clause 12.

Clause 14: An encoder comprising circuitry arranged to implement a method of any of clauses 1 to 7.

Clause 15: A decoder comprising circuitry arranged to implement a method of any of clauses 8 to 11.