Patent Description:
Machine learning and machine recognition finds several applications, such as, for example, automated passport control at airports, where a digital image of a person's face may be compared to biometric information, stored in a passport, characterizing the person's face.

Another example of machine recognition is in handwriting or printed document text recognition, to render contents of books searchable, for example. A yet further example is pedestrian recognition, wherein, ultimately, a self-driving car is thereby seen as being enabled to become aware a pedestrian is ahead and the car can avoid running over the pedestrian.

In addition to visual recognition, spoken language may be the subject of machine recognition. When spoken language is recognized, it may be subsequently input to a parser to provide commands to a digital personal assistant, or it may be provided to a machine translation program to thereby obtain a text in another language, corresponding in meaning to the spoken language.

Machine recognition technologies employ algorithms engineered for this purpose. For example, artificial neural networks may be used to implement machine vision applications. Artificial neural network may be referred to herein simply as neural networks. Machine recognition algorithms may comprise processing functions, in recognition of images such processing functions may include, for example, filtering, such as morphological filtering, thresholding, edge detection, pattern recognition and object dimension measurement.

Neural network may be comprise, for example, fully connected layers and convolutional layers. A fully connected layer may comprise a layer wherein all neurons have connections to all neurons on an adjacent layer, such as, for example, a previous layer. A convolutional layer may comprise a layer wherein neurons receive input from a part of a previous layer, such part being referred to as a receptive field, for example. The article by<NPL>, discloses an apparatus and a method to train an artificial pattern recognition neural network by applying a test dataset with at least one stochastic rectified linear unit.

According to a first aspect of the present invention, there is provided an apparatus as defined by claim <NUM>.

According to a second aspect of the present invention, there is provided method as defined by claim <NUM>.

Using an activation function with randomization, effects of overtraining a neural network to a specific training dataset may be alleviated. In detail, a stochastic rectified linear unit that varies a gradient of a linear function defining an output has been found to outperform classical rectified linear units. The stochastic rectified linear unit has also been found to outperform noisy rectified linear units, which use additive Gaussian noise to randomize the output.

<FIG> illustrates an example system. <FIG> has a view <NUM> of a road <NUM>, on which a pedestrian <NUM> is walking.

In <FIG>, road <NUM> is imaged by a camera. The camera may be configured to capture a view <NUM> that covers the road, at least in part. The camera may be configured to pre-process image data obtained from an image capture device, such as a charge-coupled device, CCD, comprised in the camera. Examples of pre-processing include reduction to black and white, contrast adjustment and/or brightness balancing to increase a dynamic range present in the captured image. In some embodiments, the image data is also scaled to a bit depth suitable for feeding into an image recognition algorithm, such as AdaBoost, for example. Pre-processing may include selection of an area of interest, such as area <NUM>, for example, for feeding into the image recognition algorithm. Pre-processing may be absent or limited in nature, depending on the embodiment. The camera may be installed, for example, in a car that is configured to drive itself, or collect training data. Alternatively, the camera may be installed in a car designed to be driven by a human driver, but to provide a warning and/or automatic braking if the car appears to be about to hit a pedestrian or an animal.

An image feed from the camera is used to generate a test dataset for use in training a neural network. Such a dataset comprises training samples. A training sample may comprise a still image, such as a video image frame, or a short video clip, for example. Where the incoming data to be recognized is not visual data, the incoming data may comprise, for example, a vector of digital samples obtained from an analogue-to-digital converter. The analogue-to-digital converter may obtain an analogue feed from a microphone, for example, and generate the samples from the analogue feed. Overall, as discussed above, data of non-visual forms may also be the subject of machine recognition. For example, accelerometer or rotation sensor data may be used to detect whether a person is walking, running or falling. As a neural network may be trained to recognize objects in view <NUM>, a training phase precedes a use phase, or test phase, of the neural network.

A challenge with training neural networks with test datasets is over-fitting of the neural network to the test dataset. As a neural network may comprise a large number of parameters, even millions of parameters, the network may become specialized in recognizing characteristics of the test dataset, rather than becoming specialized in performing the recognition task in a generic setting. To control the over-fitting problem, an element of randomization may be introduced between layers of the neural network.

One way to introduce an element of randomization between layers of the neural network is so-called dropout, where, during training, half of activations are randomly, or stochastically, selected and set to zero. The selecting may be re-done for each training sample, for example. Dropout may be applied to fully connected layers, for example, where it produces more of a benefit than in convolutional layers. Dropout may be seen as providing a way of approximatively combining exponentially many different neural network architectures in an efficient manner. Dropout is typically applied to fully connected layers, where it may provide a benefit. Dropout does not seem to be similarly beneficial in convolutional layers.

Another way to introduce an element of randomization between layers of the neural network is stochastic pooling, wherein deterministic pooling operations, such as average and maximum pooling, are replaced with a stochastic procedure for regularizing convolutional neural networks. This procedure randomly picks the activation within each pooling region according to a multinomial distribution given by activities within the pooling region. In deep convolutional neural networks, pooling does not necessarily follow each layer. Consequently, stochastic pooling may be applied a few times. Stochastic pooling needs to compute probabilities for each region at both training time and test time, resulting in an increased computational load in a device running the neural network.

The neural network is illustrated schematically in <FIG> as first layer <NUM>, rectifier <NUM> and second layer <NUM>. An actual network may comprise more than two layers. Rectifier <NUM> may be comprised functionally in first layer <NUM> or second layer <NUM>. Rectifier <NUM> may perform an activation function, and/or rectifier may comprise a rectified linear unit, ReLU. First and second layers may comprise convolutional layers. Alternatively, at least one, and optionally both, of first layer <NUM> and second layer <NUM> may comprise a fully connected layer.

Rectifier <NUM> may be configured to process an output of first layer <NUM>, for input into second layer <NUM>. For example, rectifier <NUM> may be configured to produce an output of zero from inputs that have negative values, effectively preventing negative values from being fed from first layer <NUM> to second layer <NUM>. A traditional rectifier produces an output according to function f, such that f(x) = max(<NUM>, x). Values x may be comprised in real numbers, represented in a digital system by floating-point values or an integer representation, for example.

A so-called noisy rectifier, NReLU produces an output according to f, such that f(x) = max (<NUM>, x + N(σ(x)) ), where N is Gaussian noise with variance σ(x), the Gaussian noise being employed to randomize the output of the rectifier. The variance may be obtained using all the units of one layer, for example.

A stochastic rectifier, SReLU, in accordance with the present invention, operates by obtaining an output as f(x) = max(<NUM>, bx), such that multiplier b is randomly or pseudorandomly selected from the range (<NUM> - a, <NUM> + a). The parameter a may take the value of <NUM>, <NUM>, <NUM>, <NUM> or <NUM>, for example. Multiplier b may be randomly or pseudorandomly re-obtained for each training sample, for example. Multiplier b is randomly or pseudorandomly re-obtained several times during training of the neural network. For example, multiplier b may be so re-obtained every ten, or every hundred, training samples. In other words, to obtain the output, the stochastic rectifier multiplies an input with a multiplier that is randomly selected. Put in another way, a positive output is produced from a positive input such that the positive output is a linear function of the positive input, a gradient of the linear function having a variability. The variability is random or pseudorandom. In at least some embodiments of the SReLU, no noise is separately generated and additively added in to obtain the output. For negative inputs, the stochastic rectifier may be arranged to return a zero output.

The stochastic rectifier, SReLU, is used at training time, while at test time, also referred to as simply during use, a traditional rectifier is used, wherein in the traditional rectifier, the output f(x) produced by input x is f(x) = max(<NUM>, x).

The stochastic rectifier, as defined above, yields improved recognition results with compared to both the traditional rectifier and the noisy rectifier. In a study conducted by the inventor, the following results were obtained, dropout being optionally used in a fully connected layer:.

Introducing randomization into the neural network improves performance, since two similar training samples will produce similar, but not the same, responses with randomization. Thus the test dataset effectively becomes larger, leading to improved performance. Over-fitting is also prevented, since the neural network cannot fit exactly to the training samples, the training samples producing the randomized, and no longer identical, or fully deterministic, output.

<FIG> illustrates rectifiers. In the upper part of the figure, denoted as a), is illustrated a traditional rectifier, ReLU, wherein f(x)=x for positive x and f(x)=<NUM> for negative or zero x. The response for positive x is linear with a gradient of unity.

In the middle part of the figure, denoted by b), is a noisy rectifier, NReLU, wherein f(x)=max(<NUM>, x+N), N being Gaussian noise. The output of the rectifier for positive x lies between the two lines, denoted as f(x) = x+3σ and f(x) = x-3σ. In other words, in NReLU, a randomly selected value is added to the output. Outputs of the NReLU will predominantly lie between the two lines, for positive inputs. For some slightly negative inputs, the NReLU may return a positive output in case the addition of Gaussian noise increases causes the output to exceed zero. Thus the upper, x+3σ, line intersects the y-axis above the origin.

In the lower part of the figure, denoted by c), is a stochastic rectifier, SReLU. The output of the rectifier for positive x lies between the two lines, denoted as b1*x and b2*x. For negative x the output is zero. In other words, the output for positive input is obtained by multiplying the input with a randomly selected value. In terms of <FIG>, b <NUM> = <NUM> + a and b2 = <NUM> - a. Expressed another way, a positive output is produced from a positive input such that the positive output is a linear function of the positive input, a gradient of the linear function having a variability. The SReLU may be configured, as illustrated, to return a zero output from a negative or zero input.

The benefit of SReLU over NReLU may be understood with reference to the figure, since the range of variation in NReLU is constant, being so also for small input values. in SReLU, however, the range of variation decreases as the input approaches zero from the positive direction, which maintains signals in small-amplitude inputs better than NReLU. Furthermore, compared to NReLU, SReLU is computationally more efficient, since SReLU directly multiplies each activation unit with the multiplier selected from the range. NReLU, on the other hand, calculates an input variance from each layer, and then adds a bias selected from a Gaussian distribution to each activation unit. SReLU may, in general, be employed in an artificial convolutional neural network.

<FIG> illustrates an example apparatus. Illustrated is device <NUM>, which may comprise, for example, computing device such a server, node or cloud computing device. Device <NUM> may be configured to run a neural network, such as is described herein. Comprised in device <NUM> is processor <NUM>, which may comprise, for example, a single- or multi-core processor wherein a single-core processor comprises one processing core and a multi-core processor comprises more than one processing core. Processor <NUM> may comprise more than one processor. A processing core may comprise, for example, a Cortex-A8 processing core by ARM Holdings or a Steamroller processing core produced by Advanced Micro Devices Corporation. Processor <NUM> may comprise at least one Qualcomm Snapdragon and/or Intel Core processor, for example. Processor <NUM> may comprise at least one application-specific integrated circuit, ASIC. Processor <NUM> may comprise at least one field-programmable gate array, FPGA. Processor <NUM> may be means for performing method steps in device <NUM>. Processor <NUM> may be configured, at least in part by computer instructions, to perform actions.

Computer instructions in memory <NUM> may comprise a plurality of applications or processes. For example, machine learning algorithms, such as an AdaBoost algorithm with its classifiers, may run in one application or process, a camera functionality may run in another application or process, and an output of a machine learning procedure may be provided to a further application or process, which may comprise an automobile driving process, for example, to cause a braking action to be triggered responsive to recognition of a pedestrian in a camera view.

Device <NUM> may comprise a transmitter <NUM>. Device <NUM> may comprise a receiver <NUM>. Transmitter <NUM> and receiver <NUM> may be configured to transmit and receive, respectively, information in accordance with at least one communication standard. Transmitter <NUM> may comprise more than one transmitter. Receiver <NUM> may comprise more than one receiver. Transmitter <NUM> and/or receiver <NUM> may be configured to operate in accordance with wireless local area network, WLAN, Ethernet, universal serial bus, USB, and/or worldwide interoperability for microwave access, WiMAX, standards, for example. Alternatively or additionally, a proprietary communication framework may be utilized.

Device <NUM> may comprise user interface, UI, <NUM>. UI <NUM> may comprise at least one of a display, a keyboard, a touchscreen, a vibrator arranged to signal to a user by causing device <NUM> to vibrate, a speaker and a microphone. A user may be able to operate device <NUM> via UI <NUM>, for example to configure machine learning parameters and/or to switch device <NUM> on and/or off.

Device <NUM> may comprise further devices not illustrated in <FIG>. For example, where device <NUM> comprises a smartphone, it may comprise at least one digital camera. Some devices <NUM> may comprise a back-facing camera and a front-facing camera, wherein the back-facing camera may be intended for digital photography and the front-facing camera for video telephony. Device <NUM> may comprise a fingerprint sensor arranged to authenticate, at least in part, a user of device <NUM>. In some embodiments, device <NUM> lacks at least one device described above.

Processor <NUM>, memory <NUM>, transmitter <NUM>, receiver <NUM>, and/or UI <NUM> may be interconnected by electrical leads internal to device <NUM> in a multitude of different ways. For example, each of the aforementioned devices may be separately connected to a master bus internal to device <NUM>, to allow for the devices to exchange information. However, as the skilled person will appreciate, this is only one example and depending on the embodiment various ways of interconnecting at least two of the aforementioned devices may be selected without departing from the scope of the present invention.

<FIG> illustrates a neural network in accordance with the invention. The network comprises an input layer <NUM>, which may have dimensions of <NUM> x <NUM>, for example. Layers <NUM>, <NUM> and <NUM> may have dimensions <NUM> x <NUM>, with depth <NUM>. Layer <NUM> may run 3x3 convolutional kernels with SReLU output, layer <NUM> may likewise run 3x3 convolutional kernels with SReLU output, and layer <NUM> may run 1x1 convolutional kernels with SReLU output. Each of layers <NUM>, <NUM> and <NUM> outputs <NUM> feature channels.

Layers <NUM> and <NUM> may each have dimensions <NUM> x <NUM> with depth <NUM>, and run 3x3 convolutional kernels with SReLU output. Layer <NUM> may run 1x1 convolutional kernels, apply SReLU to output and implement a dropout, as described herein above.

Processing advances from layer <NUM> to layer <NUM> via a Max pooling procedure. Layers <NUM> and <NUM> may have dimensions <NUM> x <NUM>, with depth <NUM>, and they may run 3x3 convolutional kernels with SReLU output. Layer <NUM> may have <NUM> x <NUM> dimensions with <NUM> depth, with SReLU output and dropout. Processing advances from layer <NUM> to layer <NUM> via a Max pooling procedure. Layers <NUM> and <NUM> may have dimensions <NUM> x <NUM> with depth <NUM>, and they may run 3x3 convolutional kernels with SReLU output. Layers <NUM> and <NUM> may have dimensions <NUM> x <NUM> with depth <NUM> and <NUM>, respectively, running 1x1 convolutional kernels with SReLU and ReLU output, respectively. From layer <NUM>, which runs ten feature channels, processing may advance to a decision phase via an Average pooling procedure. Activations in each channel are averaged to generate one score for each category. The decision phase may comprise <NUM>-class softmax classifier, for example.

To generate a neural network with SReLU on accordance with the example in <FIG>, initially all convolutional layers may be provided with ReLU output, after which all except the last one may be replaced with SReLU output. The neural network may, in general, comprise an artificial convolutional neural network, for example.

<FIG> is a flow graph of a method in accordance with at least some embodiments of the present invention. The phases of the illustrated method may be performed in a device arranged to run the neural network, for example, by a control device of such a device.

Phase <NUM> comprises storing data defining, an artificial neural network. Phase <NUM> comprises training the artificial neural network by applying a test dataset to the artificial neural network with at least one stochastic rectified linear unit, the at least one stochastic rectified linear unit being configured to produce a positive output from a positive input by multiplying the input with a stochastically selected value.

In use, after training, the stochastic rectified linear unit is replaced in the artificial neural network with a rectified linear unit which returns an output f from input x according to f(x) = max(<NUM>, x).

be limited, except as by the claims set forth below.

At least some embodiments of the present invention find industrial application in optimizing machine recognition, to, for example, reduce traffic accidents in self-driving vehicles.

Claim 1:
An apparatus comprising:
- memory configured to store data defining, an artificial neural network, wherein the artificial neural network is configured to detect pedestrians from image data, and
- at least one processing core configured to:
train the artificial neural network by applying a test dataset comprising training samples, each training sample comprising an image, to the artificial neural network with at least one stochastic rectified linear unit, the at least one stochastic rectified linear unit being configured to produce a positive output from a positive input by multiplying the input with a stochastically selected value, wherein the stochastical selection comprises that the value is randomly or pseudorandomly selected from a range (<NUM> - a, <NUM> + a), wherein a is a number, wherein the at least one processing core is configured to randomly or pseudorandomly re-obtain the value plural times during the training; and
after training, use the artificial neural network, wherein the stochastic rectified linear unit is replaced with a rectified linear unit.