## 2 Background
### Notation  
We choose to index elements in vectors, matrices, and tensors starting from 0 (not from 1), as in C or Python languages. This choice is slightly more convenient than natural indexing for some operations on tensors like reshaping or taking subranges. Additionally, for convolutional filters, we apply a convention that allows for negative indexes of weights. This convention treats the point (pixel) at which the filter is currently anchored as a temporary center, i.e., the \((0,0)\) point. More specifically, assuming an odd-sized \(F\times F\) filter (\(1\times 1\), \(3\times 3\), \(5\times 5\), etc. ), one can write its indexed weights as \(w_{a,b}\), where both \(a,b\) take values in the set \(\{-\lfloor F/2\rfloor,\ldots,-1,0,1,\ldots,\lfloor F/2\rfloor\}\). Let us set an example; suppose that \(X\) denotes an input image, and \(Y\) denotes the resulting convolution image (for simplicity, both with 1 channel). Additionally, let \(F\)=7. Then, elements of \(Y\) are computed as follows:  
\[y_{j,k}=\sum_{-3\leqslant a,b\leqslant 3}w_{a,b}\cdot x_{j+a,k+b}. \tag{1}\]  
Obviously, \(x_{j+a,k+b}\) must be well defined for regions outside the proper borders of \(X\) (padding with zeros is a standard practice that we address later on). We remark that, in general, not pairs but quadruples shall be used as indexes of filter weights to suitably handle input and output channels. In many places where indicator functions are needed, we apply the so-called Iverson convention. It represents an indicator with a square bracket, \([\cdot]\), yielding 1 if its argument (any logical statement) is true, and 0 otherwise. As regards multiplications, we use the symbol \"\(\cdot\)\" for standard algebraic products of numbers or matrices, whereas we use \"\(\circ\)\" for element-wise products of tensors with the same size (also known as the Hadamard product). In some places, even if the multiplication symbol is mathematically redundant, we still write it explicitly in order to emphasize the particular terms involved in the backpropagation chain. Identity matrices are denoted as \(I\) with a suitable subscript to define the size, whereas bold symbols, **1**, stand for tensors consisting of ones, e.g., \(1_{5\times 1}=(1,1,1,1,1)^{T}\). We apply the \"\(:=\)\" signs to indicate assignments, algorithmic or computational, and we use them distinctly from purely mathematical equality signs, \"\(=\)\". In places related to tensor-form formulas, it shall be convenient for us to introduce two operators: R--performing a _reshape_ of a tensor, and S--generating a _stack_ of tensors as a function of certain iterators. For example, given a vector, \(x=(x_{0},\ldots,x_{23})\), theexpression \(\underset{3\times 2\times 4}{\mathbb{R}}\) yields a three-dimensional tensor of shape \(3\times 2\times 4\) with the following successive slices:  
\[ \begin{pmatrix}x_{0}&x_{1}&x_{2}&x_{3}\\ x_{4}&x_{5}&x_{6}&x_{7}\end{pmatrix} , \begin{pmatrix}x_{8}&x_{9}&x_{10}&x_{11}\\ x_{12}&x_{13}&x_{14}&x_{15}\end{pmatrix} , \begin{pmatrix}x_{16}&x_{17}&x_{18}& x_{19}\\ x_{20}&x_{21}&x_{22}&x_{23}\end{pmatrix} , \tag{2}\]  
where the last dimension changes the fastest. Obviously, the superposition \(\underset{24}{\mathbb{R}}\underset{3\times 2\times 4}{\mathbb{R}}\) yields back the original \(x\). Shaping \(x\) explicitly to become a row or a column vector is also possible by writing \(\underset{1\times 24}{\mathbb{R}}\) or \(\underset{24\times 1}{\mathbb{R}}\), respectively. As regards the second operator--S, it can be understood as a parameterized generator of larger tensors produced from smaller ones. It works by iterating over a certain variable (or variables) and stacks the successive operands together, thereby generating a tensor extended to an extra dimension with respect to the operands. When the argument under S is a vector, then the resulting tensor will be a matrix consisting of stacked vectors. When the argument is a matrix, the result will be a three-dimensional tensor consisting of stacked matrices, and so forth.