Patent Description:
Machine learning and machine recognition finds several applications, such as, for example, automated passport control at airports, where a digital image of a person's face may be compared to biometric information, stored in a passport, characterizing the person's face.

Another example of machine recognition is in handwriting or printed document text recognition, to render contents of books searchable, for example. A yet further example is pedestrian recognition, wherein, ultimately, a self-driving car is thereby seen as being enabled to become aware a pedestrian is ahead and the car can avoid running over the pedestrian.

In addition to visual recognition, spoken language may be the subject of machine recognition. When spoken language is recognized, it may be subsequently input to a parser to provide commands to a digital personal assistant, or it may be provided to a machine translation program to thereby obtain a text in another language, corresponding in meaning to the spoken language.

Machine recognition technologies employ algorithms engineered for this purpose. For example, artificial neural networks may be used to implement machine vision applications. Artificial neural networks may be referred to herein simply as neural networks. Machine recognition algorithms may comprise processing functions, in recognition of images such processing functions may include, for example, filtering, such as morphological filtering, thresholding, edge detection, pattern recognition and object dimension measurement.

A neural network may comprise, for example, fully connected layers and convolutional layers. A fully connected layer may comprise a layer wherein all neurons have connections to all neurons on an adjacent layer, such as, for example, a preceding layer. A convolutional layer may comprise a layer wherein neurons receive input from a part of a preceding layer, such part being referred to as a receptive field, for example.

A convolution result may be obtained from a linear operation with a receptive field of a preceding layer and a filter, for populating neurons of a succeeding layer with values.

Document <NPL>, discloses use of a Max-Feature-Map, MFM, operation in activation in a CNN, the MFM taking as inputs values from more than one feature map. Document <NPL>, discloses using two cascades of SVM classifiers in pedestrian detection, a LBP cascade and a HOG cascade which has a high accuracy in detecting pedestrians. Document <NPL>, discloses using visual and thermal cameras in person recognition in a bid to reduce the effects of noise. A CNN is employed for feature extraction.

According to a first aspect of the present invention, there is provided an apparatus comprising a memory and at least one processing core configured to perform storing a convolutional artificial neural network, comprising at least one convolutional layer and an activation layer and convolutional artificial neural network information comprising at least one filter definition, wherein the convolutional artificial neural network is configured to receive an input which comprises an image and output information that indicates that one or more vertically aligned objects have been detected in the image generating, from the output of a preceding layer of the convolutional artificial neural network, a convolutional result of a succeeding layer of the artificial neural network in accordance with the at least one filter definition, and generating, from the convolutional result, an activation result of the succeeding layer by using an activation function, the activation function taking three arguments derived from the convolutional result, wherein one of the arguments is a first convolutional result value, wherein the at least one processing core is configured to derive the other two of the arguments from a neighbourhood of the first convolutional result value, and wherein the other two of the arguments comprise a value indicating a horizontal variation and a value indicating vertical variation.

According to a second aspect of the present invention, there is provided a method comprising storing a convolutional artificial neural network, comprising at least one convolutional layer and an activation layer and convolutional artificial neural network information comprising at least one filter definition, wherein the convolutional artificial neural network is configured to receive an input which comprises an image and output information that indicates that one or more vertically aligned objects have been detected in the image, generating, from the output of a preceding layer of the convolutional artificial neural network, a convolutional result of a succeeding layer of the artificial neural network in accordance with the at least one filter definition, and generating, from the convolutional result, an activation result of the succeeding layer by using an activation function, the activation function taking three arguments derived from the convolutional result, wherein one of the arguments is a first convolutional value, and deriving the other two of the arguments from a neighbourhood of the first convolutional result value, and wherein the other two of the arguments comprise a value indicating a horizontal variation and a value indicating vertical variation.

According to a third aspect of the present invention, there is provided a computer program comprising instructions for causing an apparatus to perform at least the following: store a convolutional artificial neural network, comprising at least one convolutional layer and an activation layer and convolutional artificial neural network information comprising at least one filter definition, wherein the convolutional artificial neural network is configured to receive an input which comprises an image and output information that indicates that one or more vertically aligned objects have been detected in the image, generate, from an output of a preceding layer of the convolutional artificial neural network, a convolutional result of a succeeding layer of the artificial neural network in accordance with the at least one filter definition, and generate, from the convolutional result, an activation result of the succeeding layer by using an activation function, the activation function taking three arguments derived from the convolutional result wherein one of the arguments is a first convolutional result value, and derive the other two of the arguments from a neighbourhood of the first convolutional result value, and wherein the other two of the arguments comprise a value indicating a horizontal variation and a value indicating vertical variation.

An activation function may be used in populating a layer of an artificial neural network, such that an activation function takes an input of a convolutional result from a preceding layer, obtained using filter coefficients, and produces an activation result as an output, the activation result being then entered, or populated, into neurons of the layer. A three-dimensional activation function is herein described, such that the activation result reflects a more nuanced rendering of features of the convolutional result. For example, the three-dimensional activation function may take as inputs a base element from the convolutional result, and values characterizing gradients in two directions in the neighbourhood of the base element. This may enable a higher selectivity for, for example, vertically oriented objects such as pedestrians.

<FIG> illustrates an example system capable of supporting at least some embodiments of the present invention. <FIG> has a view <NUM> of a road <NUM>, on which a pedestrian <NUM> is walking. While described herein in connection with <FIG> in terms of detecting pedestrians, the invention is not restricted thereto, but as the skilled person will understand, the invention is applicable also more generally to machine recognition in visual, data,. For example, bicyclist recognition, handwriting recognition, facial recognition and traffic sign recognition, may benefit from the present invention, depending on the embodiment in question.

In <FIG>, road <NUM> is imaged by a camera. The camera may be configured to capture a view <NUM> that covers the road, at least in part. The camera may be configured to pre-process image data obtained from an image capture device, such as a charge-coupled device, CCD, comprised in the camera. Examples of pre-processing include reduction to black and white, contrast adjustment and/or brightness balancing to increase a dynamic range present in the captured image. In some embodiments, the image data is also scaled to a bit depth suitable for feeding into an image recognition algorithm, such as AdaBoost, for example. Pre-processing may include selection of an area of interest, such as area <NUM>, for example, for feeding into the image recognition algorithm. Pre-processing may be absent or limited in nature, depending on the embodiment. The camera may be installed, for example, in a car that is configured to drive itself, or collect training data. Alternatively, the camera may be installed in a car designed to be driven by a human driver, but to provide a warning and/or automatic braking if the car appears to be about to hit a pedestrian or an animal.

An image feed from the camera may be used to generate a test dataset for use in training a neural network. Such a dataset may comprise training samples. A training sample may comprise a still image, such as a video image frame, or a short video clip, for example. Where the incoming data to be recognized is not visual data, the incoming data may comprise, for example, a vector of digital samples obtained from an analogue-to-digital converter. The analogue-to-digital converter may obtain an analogue feed from a microphone, for example, and generate the samples from the analogue feed. As a neural network may be trained to recognize objects in view <NUM>, a training phase may precede a use phase, or test phase, of the neural network.

The neural network is illustrated schematically in <FIG> as first layer <NUM>, second layer <NUM> and third layer <NUM>. An actual network may comprise more than three layers. When populating neurons of third layer <NUM>, for example, a convolutional filter may be applied to data in second layer <NUM>. For example, a convolutional filter may have sixteen coefficients, corresponding to a 4X4 grid. Applying the convolutional filter to the preceding layer, that is, second layer <NUM>, yields a convolutional result. In the convolutional result, each element corresponds to a result of a linear operation between the filter coefficients and a receptive field in second layer <NUM>. In general, third layer <NUM> succeeds second layer <NUM>, and second layer <NUM> precedes third layer <NUM>.

The convolutional result may have dimensions matching those of third layer <NUM>, or, more generally, the succeeding layer. Where the dimensions to not match, they may be made to match, for example by cropping. Zero-padding of the preceding layer may also be used, in some embodiments, to facilitate the convolutional linear operations. In principle, the convolutional result may be stored directly in the succeeding layer. However, using an activation function may provide improved results in operation. A simple activation function is a linear rectifier f, which produces an activation result f(x) from an element in the convolutional result x as f(x) = max(<NUM>, x). This simply limits the values to be populated into the succeeding layer to zero and above, such that any negative values in the convolutional result are mapped to zeroes.

In at least some embodiments of the invention, the used activation function takes three arguments. The arguments may be derived from the convolutional result, one of the arguments is a result value of the convolutional result. Further, at least one of the arguments may be derived using a mathematical process from elements of the convolutional result.

By taking three arguments it is herein meant the activation function is three-dimensional in the sense that a domain of the activation function is a three dimensional space. Equivalently, this may be expressed as the activation function being a real multivariate function. The activation function may take the three arguments each invocation. The activation function may take the type r(x,y,z) = a, with a denoting the activation result. The activation result may be one-dimensional or two-dimensional, for example. An example of a two-dimensional activation result is r(x,y,z) = [gh(x,y), gv(x,z)] where the subscript h denotes horizontal and the subscript v denotes vertical. Where the activation result is two-dimensional, the activation function has a range that is a two-dimensional space.

For example, in an application geared at detecting pedestrians, enabling in the activation function a sensitivity to vertical aspects of the convolutional result may enhance detection of pedestrians, since pedestrians have a vertical character owing to the way humans walk in an upright position, as illustrated in <FIG>. An example of sensitivity to a vertical aspect in this regard is that one of the arguments of the activation function is a local vertically oriented gradient in the convolutional result. Likewise, a horizontal sensitivity may be obtained by determining a local horizontally oriented gradient in the convolutional result. One example of three arguments to be provided to the activation function is thus an element comprised in the convolutional result, and local vertical and horizontal gradients in the convolutional result.

Another example of an activation function taking three arguments is one where a first one of the three arguments is an element comprised in the convolutional result, and the other two arguments characterize frequency aspects in the convolutional result in the neighbourhood of the element that is the first one of the arguments. Such localized frequency aspects may be obtained using wavelets, for example. Such localized frequency aspects may comprise indications as to the presence or absence of certain frequencies, for example. Such localized frequency aspects may be oriented in the horizontal and vertical direction, respectively, for example.

In general, where the activation function takes three arguments, one of the arguments may be an element comprised in the convolutional result, a second one of the arguments may characterize a horizontal property of the convolutional result in the neighbourhood of the first argument, and a third one of the arguments may characterize a vertical property of the convolutional result in the neighbourhood of the first argument. Gradients and frequency characteristics are examples of properties the second and third arguments may characterize.

<FIG> illustrates aspects relating to activation functions. In part a) of the FIGURE, a simple one-argument activation function is illustrated, which is a rectifier of the type f(x) = max(<NUM>, x), as discussed above. Another example of an activation function is a sigmoid function:
<MAT>.

In part b) of <FIG>, a part of a convolutional result is illustrated, in the neighbourhood of element x. Thus each element here corresponds to a result of a linear operation between a filter and a respective receptive field in a preceding layer. Part c) of <FIG> illustrates a horizontal gradient field of the convolutional result neighbourhood of part b). Each element of the gradient field of part c) of the FIGURE is calculated as a difference between an element in part b) and a horizontal neighbour thereof. For example, the top left element of part c) is a difference between elements x<NUM> and x<NUM> of part b).

A horizontal variation of in the neighbourhood of x may be expressed as:
<MAT>
in other words, the variation may be expressed as a sum of squares of horizontally aligned differences. Likewise, a vertical variation may be expressed as:
<MAT>.

Using these in an activation function which takes three arguments may take the form
<MAT>
where f may be a rectifier of the type illustrated in <FIG>, or f may alternatively be a sigmoid function as defined above, or indeed another kind of rectifier. Expressed in other terms, deriving the arguments for the activation function may comprise initially determining the neighbours of element x in the convolutional result, deriving gradients and obtaining variations as sums-of-squares of the gradients. In another embodiment, absolute values may be used instead of squares in obtaining the variation from the gradients. Deriving one of the arguments may simply comprise taking an element from the convolutional result, such as x in <FIG>.

The proposed three-dimensional activation function r(x,y,z) can be used in various deep learning algorithm architectures. For example, a traditional activation function may be replaced with the proposed activation function taking three arguments, and then the network may be trained with a standard back-propagation algorithm, for example.

ADAS, or Advanced Driver Assistance Systems/Autonomous Vehicle is taken as an example to illustrate how the described method may be used in real applications. A vision system may be equipped with the described method in an ADAS system or in an autonomous vehicle. The described method may be integrated into the vision system of such a system. In the vision system, an image may be captured by a camera and important objects such as pedestrians and bicycles may be detected from the image by a trained deep convolutional neural network, CNN, where the proposed collaborative activation function is employed. In ADAS, some indication, such as a warning voice, may be generated if important objects, such as pedestrians, are detected so that the attention of the driver of the vehicle is directed to the objects and he may take steps to avoid a traffic accident. In an autonomous vehicle, the detected objects may be used as inputs of a control module, the control module being configured to take proper action as a response to the objects. Both kinds of system may be based on detecting objects in an image.

To accomplish object detection, the described method may comprise a training stage and a testing stage. The aim of the training stage is to design the architecture of a deep CNN and learn the parameters of the deep CNN. With the designed structure and learned parameters, the testing stage may be used to classify a batch of unknown images, which are also known as testing images, as either object images or background images.

First, the training stage is described. In a data preparation step a set of training images and their labels are prepared. A label indicates that an image is either an object image or a background image.

In an architecture design step a design for the CNN is established. For example, let the deep CNN consist of S convolutional layers. The number of feature maps in layer i is Ni and the number of feature maps in layer i-<NUM> is Ni-<NUM>.

In a convolution step a convolutional filter Wi of size wi x hi x Ni is used to obtain the convolutional result of layer i. Here w denotes a width and h a height of a feature map.

In an activation step neighbours for convolutional result x of convolutional layer i are determined. Then, horizontal gradients of the neighbours are obtained and horizontal variation of the neighbours of x is calculated by summing the squared horizontal gradients, for example. Vertical gradients of the neighbours are obtained and vertical variation of the neighbours of x is calculated by summing the squared vertical gradients, as described above. Finally, the activation result is calculated based on the three-dimensional activation function
<MAT>.

In an optional pooling step a pooling operation may be applied on one or more convolutional layers.

In an optimization step the parameters of the convolutional filters are obtained by minimizing a mean squared error of the training set, for example. The standard back-propagation algorithm may be used for solving the minimization problem. In the back-propagation algorithm, the gradients of the mean squared error with respect to the parameters of the filters are computed and back-propagated. The back-propagation may be conducted in several epochs until convergence. In general, there are many ways to optimize the filter parameters, as the skilled person knows, and the described method is not restricted to any specific one. The optimization may be an iterative procedure.

Now, the testing stage is described. With the architecture and filter parameters obtained in the training stage, the CNN can be used in classifying images, or parts thereof. First, from the first layer to the last layer, the convolutional result is computed and the activation result obtained by using the described three-dimensional activation function. If a pooling operation is needed to be applied on a convolutional layer, then a pooling operation such as max-pooling may be applied to the layer. Then, the result of the last layer is taken as a detection result.

Because a three-dimensional activation function can simultaneously take into account the convolutional result in question, x, the variation in vertical direction, and the variation in horizontal direction, the described method is more powerful in feature detection in deep learning. Consequently, the deep learning adopting the proposed three-dimensional activation function yields a better recognition rate.

To demonstrate the effectiveness of the described method, experimental results on the CIFAR10 dataset and the ImageNet dataset are reported. Comparison is done with the classical NIN [<NUM>] method, the VGG method [<NUM>], and their versions with different activation functions. To compare with traditional NIN, two schemes are adopted. In the first scheme, the described method adopts the same architecture as NIN and the number of parameters is the same as NIN. Because the described activation function outputs two values, in the second scheme the number of parameters is two times larger than that of NIN. Similarly, two schemes are employed for comparing with VGG. In the NIN method and VGG method, the classical rectified linear unit, ReLU, activation function or existing two-dimensional activation function [<NUM>] is employed. In the described methods, the ReLU activation function is replaced with the described three-dimensional activation function as described herein above. Table <NUM> gives the recognition error rates of different methods on different datasets. From Table <NUM>, one can see that adopting the described three-dimensional activation function yields the lowest recognition error rate whenever the number of parameters is either equal to or larger than that of traditional methods.

<FIG> illustrates an example apparatus capable of supporting at least some embodiments of the present invention. Illustrated is device <NUM>, which may comprise, for example, computing device such a server, node or cloud computing device. Device <NUM> may be configured to run a neural network, such as is described herein. Comprised in device <NUM> is processor <NUM>, which may comprise, for example, a single- or multi-core processor wherein a single-core processor comprises one processing core and a multi-core processor comprises more than one processing core. Processor <NUM> may comprise more than one processor. A processing core may comprise, for example, a Cortex-A8 processing core by ARM Holdings or a Steamroller processing core produced by Advanced Micro Devices Corporation. Processor <NUM> may comprise at least one Qualcomm Snapdragon and/or Intel Core processor, for example. Processor <NUM> may comprise at least one application-specific integrated circuit, ASIC. Processor <NUM> may comprise at least one field-programmable gate array, FPGA. Processor <NUM> may be means for performing method steps in device <NUM>. Processor <NUM> may be configured, at least in part by computer instructions, to perform actions.

Computer instructions in memory <NUM> may comprise a plurality of applications or processes. For example, machine learning algorithms, such as an AdaBoost algorithm with its classifiers, may run in one application or process, a camera functionality may run in another application or process, and an output of a machine learning procedure may be provided to a further application or process, which may comprise an automobile driving process, for example, to cause a braking action to be triggered responsive to recognition of a pedestrian in a camera view.

Device <NUM> may comprise a transmitter <NUM>. Device <NUM> may comprise a receiver <NUM>. Transmitter <NUM> and receiver <NUM> may be configured to transmit and receive, respectively, information in accordance with at least one communication standard. Transmitter <NUM> may comprise more than one transmitter. Receiver <NUM> may comprise more than one receiver. Transmitter <NUM> and/or receiver <NUM> may be configured to operate in accordance with wireless local area network, WLAN, Ethernet, universal serial bus, USB, and/or worldwide interoperability for microwave access, WiMAX, standards, for example. Alternatively or additionally, a proprietary communication framework may be utilized.

Device <NUM> may comprise user interface, UI, <NUM>. UI <NUM> may comprise at least one of a display, a keyboard, a touchscreen, a vibrator arranged to signal to a user by causing device <NUM> to vibrate, a speaker and a microphone. A user may be able to operate device <NUM> via UI <NUM>, for example to configure machine learning parameters and/or to switch device <NUM> on and/or off.

Device <NUM> may comprise further devices not illustrated in <FIG>. For example, where device <NUM> comprises a smartphone, it may comprise at least one digital camera. Some devices <NUM> may comprise a back-facing camera and a front-facing camera, wherein the back-facing camera may be intended for digital photography and the front-facing camera for video telephony. Device <NUM> may comprise a fingerprint sensor arranged to authenticate, at least in part, a user of device <NUM>. In some embodiments, device <NUM> lacks at least one device described above.

Processor <NUM>, memory <NUM>, transmitter <NUM>, receiver <NUM>, and/or UI <NUM> may be interconnected by electrical leads internal to device <NUM> in a multitude of different ways. For example, each of the aforementioned devices may be separately connected to a master bus internal to device <NUM>, to allow for the devices to exchange information. However, as the skilled person will appreciate, this is only one example and depending on the embodiment various ways of interconnecting at least two of the aforementioned devices may be selected without departing from the scope of the present invention.

<FIG> is a flow graph of a method in accordance with at least some embodiments of the present invention. As described above, a training phase is comprised of phases <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and <NUM>. A testing stage comprises phases <NUM> and <NUM>.

Initially, to enable training, data is prepared for input to the neural network. This is illustrated as phase <NUM>. An architecture of the convolutional neural network to be trained is set in phase <NUM>. The architecture may comprise a number of layers and the number of feature maps per layer, for example. The number of feature maps need not be the same on each layer.

Using the training data, convolution, phase <NUM> and activation, phase <NUM>, are performed as described above and in accordance with the selected architecture of the neural network. The pooling phase <NUM> is optional. Optimization, phase <NUM>, is continued until the training data has been run through the network, using a suitable optimization algorithm to define the convolutional filter coefficients that are to be used in the testing stage. Optimization of phase <NUM> may comprise an iterative process, where phases <NUM>, <NUM> and <NUM> are run repeatedly, until filter coefficients of the neural network have been optimized.

In the testing stage, data captured, for example, using a camera or video camera may be provided as input, run through the neural network using the convolutional filter coefficients obtained in the training stage, illustrated in <FIG> as computation phase <NUM>, to arrive at the detection phase <NUM>.

<FIG> is a flow graph of a method in accordance with at least some embodiments of the present invention. The phases of the illustrated method may be performed in a device arranged to run the neural network, for example, by a control device of such a device.

Phase <NUM> comprises storing convolutional artificial neural network information comprising at least one filter definition. Phase <NUM> comprises generating, from a preceding layer, a convolutional result of a succeeding layer of the artificial neural network in accordance with the at least one filter definition. Finally, phase <NUM> comprises generating, from the convolutional result, an activation result of the succeeding layer by using an activation function, the activation function taking three arguments, the three arguments being derived from the convolutional result. Further, the succeeding layer may be populated with the activation result, or with data derived from the activation result.

At least some embodiments of the present invention find industrial application in optimizing machine recognition, to, for example, reduce traffic accidents in self-driving vehicles.

Claim 1:
An apparatus (<NUM>) comprising a memory (<NUM>) and at least one processing core (<NUM>) configured to perform:
- storing (<NUM>) a convolutional artificial neural network, comprising at least one convolutional layer and an activation layer and convolutional artificial neural network information comprising at least one filter definition, wherein the convolutional artificial neural network is configured to receive an input which comprises an image and output information that indicates that one or more vertically aligned objects have been detected in the image;
- generating (<NUM>), from the output of a preceding layer of the convolutional artificial neural network, a convolutional result of a succeeding layer of the artificial neural network in accordance with the at least one filter definition; and
- generating (<NUM>), from the convolutional result, an activation result of the succeeding layer by using an activation function, the activation function taking three arguments derived from the convolutional result, wherein one of the arguments is a first convolutional result value, wherein
- the at least one processing core (<NUM>) is configured to derive the other two of the arguments from a neighbourhood of the first convolutional result value, and wherein the other two of the arguments comprise a value indicating a horizontal variation and a value indicating vertical variation.