Patent Description:
A simple communications system includes a transmitter, a transmission channel, and a receiver. The design of such communications systems may involve the separate design and optimisation of each part of the system. An alternative approach is to consider the entire communication system as a single system and to seek to optimise the entire system. Although some attempts have been made in the prior art, there remains scope for further developments in this area.

"<NPL> discloses a novel physical layer scheme for single user Multiple-Input Multiple-Output (MIMO) communications based on unsupervised deep learning using an autoencoder.

Example embodiments will now be described, by way of non-limiting examples, with reference to the following schematic drawings, in which:.

<FIG> is a block diagram of an example end-to-end communication system implemented as a neural network, indicated generally by the reference numeral <NUM>, in which example embodiments may be implemented. The system <NUM> includes a transmitter <NUM>, a channel <NUM> and a receiver <NUM>. Viewed at a system level, the system <NUM> converts an input symbol (s) (also called a message) received at the input to the transmitter <NUM> into an output symbol (ŝ) at the output of the receiver <NUM>.

The transmitter <NUM> implements a transmitter algorithm. Similarly, the receiver <NUM> implements a receiver algorithm. As described in detail below, the algorithms of the transmitter <NUM> and the receiver <NUM> are trained in order to optimise the performance of the system <NUM> as a whole.

The system <NUM> therefore provides an autoencoder implementing an end-to-end communication system. The autoencoder can be trained with respect to an arbitrary loss function that is relevant for a certain performance metric, such as block error rate (BLER). (The terms 'autoencoder' and 'communication system' are both used below to described the system <NUM>.

As discussed further below, one obstacle to practical hardware implementation of the communication system (or autoencoder) <NUM> is the high memory requirement and computational complexity of the involved neural networks. Hardware acceleration may be needed to achieve reasonable interference time. However, graphical processing units (GPUs) that can be used to accelerate neural network evaluations come at a high monetary and energy cost that may not be viable in many communication systems.

The transmitter algorithm implemented by the transmitter <NUM> may be implemented as a differentiable parametric function and may include at least some trainable weights (which may be trainable through stochastic gradient descent). Similarly, the receiver algorithm implemented by the receiver <NUM> may be implemented as a differentiable parametric function and may include at least some trainable weights (which may be trainable through stochastic gradient descent).

The transmitter <NUM> seeks to communicate one out of M possible messages s ∈ <IMG> = {<NUM>,<NUM>,. , M} to the receiver <NUM>. To this end, the transmitter <NUM> sends a complex-valued vector representation <MAT> of the message through the channel <NUM>. Generally, the transmitter hardware imposes constraints on x, e.g. an energy constraint <MAT>, n, an amplitude constraint |xi| ≤ <NUM>∀i, or an average power constraint <MAT>. The channel is described by the conditional probability density function (pdf) p(ylx), where y ∈ <MAT> denotes the received signal. Upon reception of y, the receiver produces the estimate ŝ of the transmitted message s.

As shown in <FIG>, the transmitter <NUM> includes a dense layer of one or more units <NUM>, <NUM> (e.g. including one or more neural networks) and a normalization module <NUM>. The dense layers <NUM>, <NUM> may include an embedding module. The modules within the transmitter <NUM> are provided by way of example and modifications are possible.

The message index s may be fed into an embedding module, embedding: <MAT>, that transforms s into an nemb-dimensional real-valued vector.

The embedding module may be followed by several dense neural network (NN) layers <NUM>, <NUM> with possible different activation functions (such as ReLU, tanh, signmoid, linear etc.). The final layer of the neural network may have has 2n output dimensions and a linear activation function. If no dense layer is used, nemb = 2n.

The output of the dense layers <NUM>, <NUM> may be converted to a complex-valued vector through the mapping <MAT>, which could be implemented as <MAT> <MAT>. (This is not shown and is a purely optional step.

A normalization is applied by the normalization module <NUM> that ensures that power, amplitude or other constraints are met. The result of the normalization process is the transmit vector x of the transmitter <NUM> (where <MAT>). As noted above, modifications may be made to the transmitter <NUM>, for example the order of the complex vector generation and the normalization could be reversed.

The transmitter <NUM> defines the following mapping: <MAT>.

In other words, TX maps an integer from the set <IMG> to an 2n-dimensional real-valued vector. One example mapping is described above. Other neural network architectures are possible and the illustration above services just as an example.

As shown in <FIG>, the receiver <NUM> includes a dense layer of one or more units <NUM>, <NUM> (e.g. including one or more neural networks), a softmax module <NUM> and an arg max module <NUM>. As described further below, the output of the softmax module is a probability vector that is provided to the input of an arg max module <NUM>. The modules within the receiver <NUM> are provided by way of example and modifications are possible.

If the channel output vector <MAT> is complex-valued, it may be transformed by the receiver <NUM> into a real-valued vector of 2n dimensions through the mapping <MAT> <MAT>, which could be implemented as <MAT>. This step is not necessary for real-valued channel outputs <MAT>.

The result is fed into the one or more layers <NUM>, <NUM>, which layers may have different activation functions such as ReLU, tanh, sigmoid, linear, etc. The last layer may have M output dimensions to which a softmax activation is applied (by softmax module <NUM>). This generates the probability vector <MAT>, whose ith element [p]i can be interpreted as Pr(s = i|y). A hard decision for the message index is obtained as ŝ = arg max(p) by arg max module <NUM>.

The transmitter <NUM> and the receiver <NUM> may be implemented as neural networks having parameter vectors θT and θR respectively. If a differential channel model is available, then the channel model can be used as an intermediate non-trainable layer, such that the entire communication system <NUM> can be seen as a single neural network with parameters vector θ = (θT + θR), which defines the mapping: <MAT>.

Neural networks are often trained and exploited on computationally powerful platforms with general processing units (GPU) acceleration, supporting high precision floating point arithmetic (e.g. <NUM>-bit or <NUM>-bit floating point arithmetic). Such hardware may not be available for some communication systems. Accordingly, the neural networks.

implementing the transmitter <NUM> and the receiver <NUM> of the system <NUM> described above may be compressed to meet practical constraints. This may be achieved by using a more compact representation of neural network parameters, at the cost of reduced precision. For example, as described further below, compression of weights and/or biases of the neural networks may be achieved through quantization, such that the weights are forced to take values within a codebook with a finite number of entries (e.g. at a lower precision that that provided by a training module).

<FIG> is a flow chart showing an algorithm, indicated generally by the reference numeral <NUM>, in accordance with a non-claimed embodiment.

The algorithm <NUM> starts at operation <NUM>, where an autoencoder (such as the autoencoder/communication system <NUM>) is trained.

For example, the autoencoder <NUM> may be trained in a supervised manner using the following stochastic gradient descent (SGD) algorithm:.

At operation <NUM>, the parameters generated by the operation <NUM> are quantized using the following function: θq = Π(θ), where:.

The algorithm <NUM> then terminates at operation <NUM>. The system <NUM> can then be implemented using the quantized parameters θq. This approach significantly reduces memory requirements within the neural networks of the system <NUM> and reduces the complexity of arithmetic operations, but results in reduced precision.

<FIG> is a flow chart showing an algorithm, indicated generally by the reference numeral <NUM>, in accordance with an example embodiment.

The algorithm <NUM> starts at operation <NUM>, where the transmitter <NUM> and the receiver <NUM> of the transmission system <NUM> are initialised, thereby providing a means for initialising parameters of a transmission system.

Operations <NUM> to <NUM> implement a learning operation <NUM>, a compression operation <NUM> and a parameter updating operation <NUM> that collectively implement a learning-compression algorithm, as discussed further below. At operation <NUM>, it is determined whether or not the learning-compression algorithm is complete. If not, the operations <NUM> to <NUM> are repeated. When the operation <NUM> determines that the learning-compression algorithm is complete (e.g. if a sufficient number of iterations of the operations <NUM> to <NUM> have been completed or if a given performance threshold has been met), the algorithm <NUM> terminates at operation <NUM>.

The learning operation <NUM> adjusts the weights of the neural networks of the transmitter <NUM> and the receiver <NUM> in the non-quantized space. The operation <NUM> is similar to the training operation <NUM> described above, but includes an additional penalty term, related to the allowed codebook values in the quantized space. Thus, the learning operation <NUM> implements a means for updating trainable parameters of the transmission system based on a loss function.

More specifically, we denote by <IMG> the cross-entropy, which is the loss function minimized during training defined by: <MAT>.

The learning operation <NUM> solves the following optimization problem: <MAT>.

At compression operation <NUM>, a compression process is carried out in which the adjusted weights are quantized. Thus, the compression operation <NUM> implements a means for quantizing trainable parameters of the transmission system.

More specifically, the compression operation <NUM> may perform the element-wise quantization of <MAT>: <MAT>.

Where θ was computed during the learning operation <NUM>. We denote the codebook by <IMG>. The quantizing the scalar <MAT> can be expressed as:
<MAT>.

Which simply means to find the closest element within the quantization codebook C.

At operation <NUM>, various parameters of the algorithm <NUM> are updated, as discussed further below.

Thus, the algorithm <NUM> iterates between the learning operation <NUM> and the compression operation <NUM>. At the end of each operation, the Lagrange multiplier elements are updated. The algorithm <NUM> can be expressed mathematically as set out below.

The initialisation operation <NUM> may comprise:.

The learning operation <NUM> may comprise:.

The compression operation <NUM> may comprise:.

The update parameters operation <NUM> may comprise:.

The complete operation <NUM> may comprise:.

However, the criterion for the complete operation <NUM> could be implemented in other ways.

The algorithm <NUM> converges to a local solution as µ(i) → ∞. This could be achieved, for example, by following a multiplicative schedule µ(i) ← µ(i-<NUM>)a, where a > <NUM> and µ(<NUM>) are parameters of the algorithm.

It should be noted that the sequence µ(i) can be generated in many ways. Using a multiplicative schedule (as discussed above), the initial value of µ(<NUM>), as well as a are optimisation parameters (and may be optimised as part of the training operation <NUM> described above).

It should also be noted that the batch size N as well as the learning rate (and possibly other parameters of the chosen SGD variant, e,g, ADAM, RMSProp, Momentum) could be optimization parameters of the training operations <NUM> and <NUM> described above.

As discussed above, quantization of the weights of the transmitter and receiver algorithms can lead to precision loss, which can in turn lead to poor performance if the received signal (e.g. the signal y in <FIG>) is out of the range of values approximated by the codebook range. To keep the received signal in the range of values approximated by the codebook <IMG>, the receiver input can be scaled by a value η, which may or may not depend on the received signal.

<FIG> is a block diagram of a receiver <NUM> of an example end-to-end communication system in accordance with an example embodiment. The receiver <NUM> includes the softmax module <NUM> and the arg max module <NUM> described above with reference to <FIG>. Further, the receiver includes a dense layer of one or more units <NUM>, <NUM> and <NUM> (similar to the layers <NUM> and <NUM> described above). The input stage includes a multiplier <NUM> by which inputs are scaled by the scaling factor η. The scaling factor η may be a directly trainable factor (that may be trained as part of the learning operation <NUM>), a fixed value, or the output of another neural network whose weights are optimized in the training process described above. Scaling can therefore be used to ensure that the dynamic range of the signal y is matched to the quantization codebook.

In some embodiments, once the autoencoder (e.g. the autoencoder <NUM>) is trained, the transmitter of the autoencoder can be implemented by a simple look-up table, which maps a message s ∈ <IMG> to a quantized real vector <MAT> or a complex vector <MAT> (depending on the implementation of the transmitters), where elements <IMG>(xi) and <IMG>(xi) are quantized to the codebook C. Therefore, compression of the transmitter weights θT in such an arrangement is not necessary, as they will not be required when exploiting the autoencoder. For example, if the message set to be transmitted is relatively small, then the set of quantized neural network weights for each message can be stored.

<FIG> is a block diagram of a transmitter <NUM> of an example end-to-end communication system in accordance with an example embodiment. The transmitter <NUM> includes the dense layer of units <NUM>, <NUM> and the normalization layer <NUM> of the transmitter <NUM> described above. The system <NUM> also comprises a quantization layer <NUM> in which the signal x is transformed into a quantized output xq. The quantization layer <NUM> allows the training of the autoencoder to be done considering that the output of the transmitter is quantized. Once training is complete, the vector xq generated by the transmitter <NUM> for each message s ∈ <IMG> is stored in a look-up table implementing the transmitter.

It should be noted that the quantization operation Π is not generally differentiable. For back propagation, a simple work-around is to assign a custom derivative for Π, such as <MAT>. That is, during the forward pass of the neural network, the quantization is performed by the quantization layer <NUM>, but during the backward-pass (to compare the gradients), the quantization is treated as the identity function.

As described above, one method for quantizing the weights is to use K-bit fixed point arithmetic, instead of using a floating point arithmetic (such as <NUM>-bits or <NUM>-bits floating point arithmetic). Combining the principles of quantization and the use of fixed point arithmetic not only results in using fewer bits to represent the weights of the neural.

networks described herein, but also greatly reduces the complexity of arithmetic operators.

Using K-bits fixed point arithmetic, with KE bits used for the integer part and KF bits used for the fractional part (such that KE + KF = K), a weight w is represented by: <MAT>.

Where we,i and wf,j take values in {<NUM>,<NUM>}. The number of bits K, as well as the sizes of the integer and fractional parts KE and KF are fixed. The scalar w is represented by a K+<NUM> bit word (ws, we,<NUM>,. , weKE-<NUM>, wf,<NUM>,. , wf,KF), where ws is a sign bit (i.e. a bit indicating the sign of the weight).

For completeness, <FIG> is a schematic diagram of components of one or more of the modules described previously (e.g. the transmitter or receiver neural networks), which hereafter are referred to generically as processing systems <NUM>. A processing system <NUM> may have a processor <NUM>, a memory <NUM> closely coupled to the processor and comprised of a RAM <NUM> and ROM <NUM>, and, optionally, hardware keys <NUM> and a display <NUM>. The processing system <NUM> may comprise one or more network interfaces <NUM> for connection to a network, e.g. a modem which may be wired or wireless.

The memory <NUM> may comprise a non-volatile memory, a hard disk drive (HDD) or a solid state drive (SSD). The ROM <NUM> of the memory <NUM> stores, amongst other things, an operating system <NUM> and may store software applications <NUM>. The RAM <NUM> of the memory <NUM> is used by the processor <NUM> for the temporary storage of data. The operating system <NUM> may contain code which, when executed by the processor, implements aspects of the algorithms <NUM> and <NUM>.

The processor <NUM> may take any suitable form. For instance, it may be a microcontroller, plural microcontrollers, a processor, or plural processors.

The processing system <NUM> may be a standalone computer, a server, a console, or a network thereof.

In some embodiments, the processing system <NUM> may also be associated with external software applications. These may be applications stored on a remote server device and may run partly or exclusively on the remote server device. These applications may be termed cloud-hosted applications. The processing system <NUM> may be in communication with the remote server device in order to utilize the software application stored there.

<FIG> show tangible media, respectively a removable memory unit <NUM> and a compact disc (CD) <NUM>, storing computer-readable code which when run by a computer may perform methods according to embodiments described above. The removable memory unit <NUM> may be a memory stick, e.g. a USB memory stick, having internal memory <NUM> storing the computer-readable code. The memory <NUM> may be accessed by a computer system via a connector <NUM>. The CD <NUM> may be a CD-ROM or a DVD or similar. Other forms of tangible storage media may be used.

Reference to, where relevant, "computer-readable storage medium", "computer program product", "tangibly embodied computer program" etc., or a "processor" or "processing circuitry" etc. should be understood to encompass not only computers having differing architectures such as single/multi-processor architectures and sequencers/parallel architectures, but also specialised circuits such as field programmable gate arrays FPGA, application specify circuits ASIC, signal processing devices and other devices. References to computer program, instructions, code etc. should be understood to express software for a programmable processor firmware such as the programmable content of a hardware device as instructions for a processor or configured or configuration settings for a fixed function device, gate array, programmable logic device, etc..

As used in this application, the term "circuitry" refers to all of the following: (a) hardware-only circuit implementations (such as implementations in only analogue and/or digital circuitry) and (b) to combinations of circuits and software (and/or firmware), such as (as applicable): (i) to a combination of processor(s) or (ii) to portions of processor(s)/software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a server, to perform various functions) and (c) to circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even if the software or firmware is not physically present.

It will be appreciated that the above described example embodiments are purely illustrative and are not limiting on the scope of the invention. Other variations and modifications will be apparent to persons skilled in the art upon reading the present specification.

Claim 1:
An apparatus comprising:
means for initialising parameters of a transmission system (<NUM>), wherein the transmission system comprises a transmitter (<NUM>, <NUM>), a channel (<NUM>) and a receiver (<NUM>, <NUM>), wherein the transmitter includes a transmitter algorithm having at least some trainable weights and the receiver includes a receiver algorithm having at least some trainable weights;
means for updating (<NUM>) trainable parameters of the transmission system based on a loss function, wherein the trainable parameters include the trainable weights of the transmitter and the trainable weights of the receiver and wherein the loss function includes a penalty term;
means for quantizing (<NUM>) said trainable parameters, such that said weights can only take values within a codebook having a finite number of entries that is a subset of the possible values available during updating;
means for repeating the updating and quantizing until a first condition is reached; and
wherein said penalty term includes a variable that is adjusted on each repetition of the updating (<NUM>) and quantizing (<NUM>) such that, on each repetition, more weight is given in the loss function to a difference between the trainable parameters and the quantized trainable parameters.