System and method for invertible wavelet layer for neural networks

An electronic device, method, and computer readable medium for an invertible wavelet layer for neural networks are provided. The electronic device includes a memory and at least one processor coupled to the memory. The at least one processor is configured to receive an input to a neural network, apply a wavelet transform to the input at a wavelet layer of the neural network, and generate a plurality of subbands of the input as a result of the wavelet transform.

TECHNICAL FIELD

This disclosure relates generally to neural networks. More specifically, this disclosure relates to an invertible wavelet layer for neural networks.

BACKGROUND

Deep learning or deep neural networks is a revolutionary force in artificial intelligence. Neural Networks help computers make sense of infinite amounts of data in the form of images, sound, and text. Using multiple layers of neural perceptrons, computers now have the capacity to see, learn, and react to complex situations as well as if not better than humans.

A pooling layer is a key component in modern deep neural networks. The purpose of a pooling layer is to reduce the resolution of the feature maps so that as the network goes deeper, the same sized convolutional kernel can cover an increasingly larger receptive field in the input image. As a result, the neural network can have an increasingly global understanding of the image.

However, pooling is a lossy operation that wastes a large portion of previously computed values. For a typical deep neural network with five 2×2 max pooling layers, 99.9% of the computation of the neural network is discarded due to the pooling operation.

An upsampling layer is the reverse operation of pooling layer. An upsampling layer introduces either zeroes or interpolated values into the feature maps so that the following layers need to compute on top of feature maps with redundancies.

SUMMARY

This disclosure provides a system and method for an invertible wavelet layer for neural networks.

In one embodiment, an electronic device is provided. The electronic device includes a memory and at least one processor coupled to the memory. The at least one processor is configured to receive an input to a neural network, apply a wavelet transform to the input at a wavelet layer of the neural network, and generate a plurality of subbands of the input as a result of the wavelet transform.

In another embodiment, a method of a neural network is provided. The method comprises receiving an input to the neural network, applying a wavelet transform to the input at a wavelet layer of the neural network, and generating a plurality of subbands of the input as a result of the wavelet transform.

In another embodiment, a non-transitory computer readable medium embodying a computer program for operating an electronic device including a memory and at least one processor is provided. The computer program comprises computer readable program code that, when executed by the at least one processor, causes the electronic device to receive an input to a neural network, apply a wavelet transform to the input at a wavelet layer of the neural network, and generate a plurality of subbands of the input as a result of the wavelet transform.

DETAILED DESCRIPTION

According to embodiments of the present disclosure, an invertible wavelet layer for improving neural networks is provided. Deep neural networks can perform various functions such as image recognition, data analysis, natural language processing, intent classification, or other functions. Neural networks can generate an output based on a weighted sum of inputs, which is then passed through an activation function. The activation function can provide an output after summing the inputs multiplied by the weights. It will be understood by those skilled in the art that various activation functions can be used depending on the configuration of the neural network and the result to be achieved by the neural network.

The inputs, weights, and outputs can be organized within a multilayer perceptron (MLP), wherein there is an input layer, one or more hidden layers, and an output layer. A plurality of inputs, or an input vector, make up the input layer, a plurality of hidden layer neurons reside in the hidden layer or layers, and one or more outputs can be generated for the output layer. The neural network can be a feedforward network where inputs are passed from the input layer to a hidden layer. The inputs can be processed through an activation or transfer function to provide new inputs to a next hidden layer, if the neural network has multiple hidden layers, from hidden layer to hidden layer until the final hidden layer passes the final outputs to the output layer. As a neural network is trained, the weights can be adjusted based on calculated error rates to increase the accuracy of the neural network.

Convolutional neural networks can be used for image or object recognition. A convolution layer performs convolutions between an image and a filter or kernel (a matrix of values) to weight sections of the image based on the kernel in order to emphasize features in the image. Convolutions can be performed on a subset of the image at a time until the full image is weighted by a kernel. Kernels using different weights can be used for additional convolutions, creating a feature map as a result of each convolution. Each feature map can then be passed to the next layer of the neural network. Other layers of a convolutional neural network can be batch normalization layers, or Bnorm layers, rectified linear units (ReLU) layers, pooling layers, or others. A convolutional neural network can perform any number of convolutions, batch normalizations, ReLU calculations, and pooling operations depending on the neural network. The image can be reduced down to a vector of values and a fully connected layer then takes the vector and provides one or more outputs, such as indicating whether the image matches a particular feature or object attempting to be detected. It will be appreciated that the present disclosure is not limited to any particular type of neural network and that this disclosure can be applied to any neural network.

The Bnorm layer can be used to normalize the activation of each convolution layer. The ReLU layer applies an activation function to increase the nonlinear properties of the network, such as by zeroing out negative values. The pooling layer downsamples images or feature maps to reduce the resolution of the feature maps so that, as the network goes deeper, the same sized kernel can be used to cover an increasingly larger receptive field, allowing for more efficient processing by subsequent layers. Max pooling is a common method of pooling that outputs the maximum value of a sub-region of an image or feature map. However, pooling is a lossy operation that wastes a large portion of previously computed values. For example, when using max pooling, only the highest value in a sub-region is retained, and all other values are discarded. For instance, in a 2×2 max pooling layer, out of four elements in the pooling window or sub-region, only one value is extracted and the other three are discarded, wasting 75% of the computations of the previous layers. For a neural network with five 2×2 pooling layers the resolution of the output feature map will be reduced by a factor of 1024 and ¾(1+¼+¼2+¼3+¼4), which is a 99.9% waste of computations because of the nature of how the pooling layer operates.

In some applications such as semantic segmentation, the input and output resolution of the neural network is the same. After the feature map resolution reduction process, a resolution restoration process is implemented using an upsampling layer and deconvolutional layers. The upsampling layer is the reverse operation of the pooling layer. For example, to implement a 2×2 upsampling layer, each element is duplicated 2×2 times or zeroes are inserted. The upsampling layer thus introduces redundant information into the neural network due to the lossy operation of the pooling layers in order to approximate the original image. The upsampling layers thus provide an image that is of lower quality than the original image. The present disclosure is directed to replacing pooling layers with wavelet layers and the upsampling layers with inverse wavelet layers to losslessly reduce and expand the resolution of the feature maps. Since the wavelet and inverse wavelet layers are lossless and invertible, no computations from the previous layers are wasted. When combining wavelet and inverse wavelet layers, an auxiliary auto-encoder structure can be implemented along with deep neural networks to visualize the network's learned knowledge.

FIG.1illustrates an example system100according to embodiments of this disclosure. The embodiment of the system100shown inFIG.1is for illustration only. Other embodiments of the system100could be used without departing from the scope of this disclosure.

The system100includes a network102that facilitates communication between various components in the system100. For example, network102can communicate Internet Protocol (IP) packets, frame relay frames, Asynchronous Transfer Mode (ATM) cells, or other information between network addresses. The network102includes one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), all or a portion of a global network such as the Internet, or any other communication system or systems at one or more locations.

The network102facilitates communications between various server(s)104and various client devices106-114. Server104can represent one or more servers. Each server104includes any suitable computing or processing device that can provide computing services for one or more client devices. Each server104could, for example, include one or more processing devices, one or more memories storing instructions and data, and one or more network interfaces facilitating communication over the network102.

Each client device106-114represents any suitable computing or processing device that interacts with at least one server or other computing device(s) over the network102. In this example, the client devices106-114include a desktop computer106, a mobile telephone or mobile devices108(such as a smartphone), a personal digital assistant (PDA)110, a laptop computer112, and a tablet computer114. However, any other or additional client devices could be used in the system100.

In this example, some client devices108-114communicate indirectly with the network102. For example, the client devices108and110(mobile devices108and PDA110, respectively) communicate via one or more base stations116, such as cellular base stations or eNodeBs (eNBs). Mobile devices108include both smart phones and feature phones. Smart phones represent a class of mobile devices108that are a handheld device with a mobile operating system and an integrated mobile broadband cellular network connection for voice, short message service (SMS), and internet data communication. Feature phones represent a class of mobile devices108that are a midway point between a basic phone and a smart phone. Feature phones generally have voice calling and text messaging functions in addition to basic multimedia and internet capabilities. Also, the client devices112and114(laptop computer and tablet computer, respectively) communicate via one or more wireless access points118, such as IEEE 802.11 wireless access points. Note that these are for illustration only and that each client device106-114could communicate directly with the network102or indirectly with the network102via any suitable intermediate device(s) or network(s).

In certain embodiments, the mobile device108(or any other client device106-114) can transmit information securely and efficiently to another device, such as, for example, the server104. The mobile device108(or any other client device106-114) can receive information to be processed as an input(s) into a neural network. Such information can include image data, voice/audio data, geolocation data, user information, or other data received by or stored on the mobile device108. The mobile device108(or any other client device106-114) can trigger the information transmission between itself and server104. The mobile device108(or any other client device106-114) can provide a real-time result generated by a neural network.

The processes and systems provided in this disclosure allow for a client device or a server to provide a result processed by a neural network. In certain embodiments, a client device (client device106-114) can determine the neural network result. In certain embodiments, a client device (client device106-114) receives the data to be included as inputs into a neural network and transmits the data over the network102to the server104, which determines the output(s) using the neural network.

FIGS.2and3illustrate example devices in a computing system in accordance with embodiments of the present disclosure. In particular,FIG.2illustrates an example server200, andFIG.3illustrates an example electronic device300. The server200could represent the server104inFIG.1, and the electronic device300could represent one or more of the client devices106-114inFIG.1.

Server200can represent one or more local servers or one or more neural network servers for processing received inputs through a trained neural network. As shown inFIG.2, the server200includes a bus system205that supports communication between at least one processor(s)210, at least one storage device(s)215, at least one communications interface220, and at least one input/output (I/O) unit225.

The processor210executes instructions that can be stored in a memory230. The processor210can include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. Example types of processor(s)210include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discreet circuitry.

The memory230and a persistent storage235are examples of storage devices215that represent any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, neural network inputs and other data, or other suitable information on a temporary or permanent basis). The memory230can represent a random access memory or any other suitable volatile or non-volatile storage device(s). The persistent storage235can contain one or more components or devices supporting longer-term storage of data, such as a ready only memory, hard drive, Flash memory, or optical disc.

The communications interface220supports communications with other systems or devices. For example, the communications interface220could include a network interface card or a wireless transceiver facilitating communications over the network102. The communications interface220can support communications through any suitable physical or wireless communication link(s).

The I/O unit225allows for input and output of data. For example, the I/O unit225can provide a connection for user input through a keyboard, mouse, keypad, touchscreen, or other suitable input device. The I/O unit225can also send output to a display, printer, or other suitable output device.

Note that whileFIG.2is described as representing the server104ofFIG.1, the same or similar structure could be used in one or more of the various client devices106-114. For example, a desktop computer106or a laptop computer112could have the same or similar structure as that shown inFIG.2.

FIG.3illustrates an electronic device300in accordance with an embodiment of this disclosure. The embodiment of the electronic device300shown inFIG.3is for illustration only and other embodiments could be used without departing from the scope of this disclosure. The electronic device300can come in a wide variety of configurations, andFIG.3does not limit the scope of this disclosure to any particular implementation of an electronic device. In certain embodiments, one or more of the devices104-114ofFIG.1can include the same or similar configuration as electronic device300.

In certain embodiments, the electronic device300is useable with data transfer applications, such as providing neural network inputs or activating a function based on a neural network result or output. For example, the electronic device300can receive information, such as voice data, transfer the data to the server200, receive a response from the server200indicating the result of processing the information through a neural network, and activate a function on the electronic device300in accordance with the result. The electronic device300can be a mobile communication device, such as, for example, a wireless terminal, a desktop computer (similar to desktop computer106ofFIG.1), a mobile device (similar to mobile device108ofFIG.1), a PDA (similar to PDA110ofFIG.1), a laptop (similar to laptop computer112ofFIG.1), a tablet (similar to tablet computer114), and the like.

As shown inFIG.3, the electronic device300includes an antenna305, a communication unit310, a transmit (TX) processing circuitry315, a microphone320, and a receive (RX) processing circuitry325. The communication unit310can include, for example, a RF transceiver, a BLUETOOTH transceiver, a WI-FI transceiver, ZIGBEE, infrared, and the like. The electronic device300also includes a speaker330, a processor340, an input/output (I/O) interface345, an input350, a display355, a memory360, a sensor(s)365, and a biometric scanner370. The memory360includes an operating system (OS)361, applications362, and user data363.

The communication unit310receives, from the antenna305, an incoming RF signal transmitted such as a BLUETOOTH or WI-FI signal from an access point (such as a base station, Wi-Fi router, Bluetooth device) of the network102(such as a Wi-Fi, Bluetooth, cellular, 5G, LTE, LTE-A, WiMAX, or any other type of wireless network). The communication unit310can down-convert the incoming RF signal to generate an intermediate frequency or baseband signal. The intermediate frequency or baseband signal is sent to the RX processing circuitry325that generates a processed baseband signal by filtering, decoding, or digitizing the baseband or intermediate frequency signal, or a combination thereof. The RX processing circuitry325transmits the processed baseband signal to the speaker330(such as for voice data) or to the processor340for further processing (such as for web browsing data and remittance).

The TX processing circuitry315receives analog or digital voice data from the microphone320or other outgoing baseband data from the processor340. The outgoing baseband data can include web data, e-mail, or interactive video game data. The TX processing circuitry315encodes, multiplexes, digitizes, or a combination thereof, the outgoing baseband data to generate a processed baseband or intermediate frequency signal. The communication unit310receives the outgoing processed baseband or intermediate frequency signal from the TX processing circuitry315and up-converts the baseband or intermediate frequency signal to an RF signal that is transmitted via the antenna305.

The processor340can include one or more processors or other processing devices and execute the OS361stored in the memory360in order to control the overall operation of the electronic device300. For example, the processor340could control the reception of forward channel signals and the transmission of reverse channel signals by the communication unit310, the RX processing circuitry325, and the TX processing circuitry315in accordance with well-known principles. The processor340is also capable of executing other applications362resident in the memory360, such as, one or more applications for remittance, fraud detection, and the like.

The processor340can execute instructions that are stored in a memory360. The processor340can include any suitable number(s) and type(s) of processors or other devices in any suitable arrangement. For example, in some embodiments, the processor340includes at least one microprocessor or microcontroller. Example types of processor340include microprocessors, microcontrollers, digital signal processors, field programmable gate arrays, application specific integrated circuits, and discreet circuitry.

The processor340is also capable of executing other processes and programs resident in the memory360, such as operations that receive, store, and timely instruct by providing image capturing and processing. The processor340can move data into or out of the memory360as required by an executing process. In some embodiments, the processor340is configured to execute plurality of applications362based on the OS361or in response to signals received from eNBs or an operator. The processor340is also coupled to the I/O interface345that provides the electronic device300with the ability to connect to other devices, such as client devices106-114. The I/O interface345is the communication path between these accessories and the processor340.

The processor340is also coupled to the input350and the display355. The operator of the electronic device300can use the input350to enter data or inputs into the electronic device300. Input350can be a keyboard, touch screen, mouse, track ball, voice input, or other device capable of acting as a user interface to allow a user in interact with electronic device300. For example, the input350can include voice recognition processing thereby allowing a user to input a voice command via microphone320. For another example, the input350can include a touch panel, a (digital) pen sensor, a key, or an ultrasonic input device. The touch panel can recognize, for example, a touch input in at least one scheme among a capacitive scheme, a pressure sensitive scheme, an infrared scheme, or an ultrasonic scheme. Input350can be associated with sensor(s) and/or a camera365by providing additional input to processor340. The camera can be used to capture images to be processed by a convolutional neural network. Such a convolutional neural network can be an application stored on the electronic device300, or on the server200, in which case the electronic device300can transmit a captured image to the server200to be processed by the neural network.

In certain embodiments, sensor365includes inertial sensors (such as, accelerometers, gyroscope, and magnetometer), optical sensors, motion sensors, cameras, pressure sensors, heart rate sensors, altimeter, breath sensors (such as microphone320), and the like. The input350can also include a control circuit. In the capacitive scheme, the input350can recognize touch or proximity. The display355can be a liquid crystal display (LCD), light-emitting diode (LED) display, optical LED (OLED), active matrix OLED (AMOLED), or other display capable of rendering text and/or graphics, such as from websites, videos, games, images, and the like.

The memory360can include persistent storage (not shown) that represents any structure(s) capable of storing and facilitating retrieval of information (such as data, program code, and/or other suitable information on a temporary or permanent basis). The memory360can contain one or more components or devices supporting longer-term storage of data, such as a ready only memory, hard drive, Flash memory, or optical disc. The memory360also can contain user data363that includes profile data and user history data. User data363can also contain data received from sensor365. User data363can biographical and biometric data.

Electronic device300further includes one or more sensor(s)365that can meter a physical quantity or detect an activation state of the electronic device300and convert metered or detected information into an electrical signal. In certain embodiments, sensor365includes inertial sensors (such as accelerometers, gyroscopes, and magnetometers), optical sensors, motion sensors, cameras, pressure sensors, heart rate sensors, altimeter, breath sensors (such as microphone320), and the like. For example, sensor365can include one or more buttons for touch input, (such as on a headset or the electronic device300), a camera, a gesture sensor, a gyroscope or gyro sensor, an air pressure sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a grip sensor, a proximity sensor, a color sensor, a bio-physical sensor, a temperature/humidity sensor, an illumination sensor, an Ultraviolet (UV) sensor, an Electromyography (EMG) sensor, an Electroencephalogram (EEG) sensor, an Electrocardiogram (ECG) sensor, an Infrared (IR) sensor, an ultrasound sensor, an iris sensor, a fingerprint sensor, and the like. The sensor365can further include a control circuit for controlling at least one of the sensors included therein. The sensor(s)365can be used to determine an orientation and facing direction, as well as geographic location of the electronic device300. Any of these sensor(s)365can be located within the electronic device300or another electronic device in communication with the electronic device300.

AlthoughFIGS.2and3illustrate examples of devices in a computing system, various changes can be made toFIGS.2and3. For example, various components inFIGS.2and3could be combined, further subdivided, or omitted and additional components could be added according to particular needs. As a particular example, the processor340could be divided into multiple processors, such as one or more central processing units (CPUs) and one or more graphics processing units (GPUs). In addition, as with computing and communication networks, electronic devices and servers can come in a wide variety of configurations, andFIGS.2and3do not limit this disclosure to any particular electronic device or server.

FIG.4illustrates a block diagram of an example convolutional neural network400including wavelet layers in accordance with embodiments of the present disclosure. The processor210of the server200or the processor340of electronic device300can execute the neural network400. Convolutional neural networks can include any number of convolutional layers, batch normalization or Bnorm layers, rectified linear units or ReLU layers, or other layers. In other neural networks, pooling layers are used to downsample the feature maps produced from the other previous layers. In the neural network illustrated inFIG.4, the layers where the pooling layers would be located in other neural networks are replaced with wavelet layers. The wavelet layers losslessly reduce the size of feature maps. For processing by the next series of layers.

In the example illustrated inFIG.4, the neural network400is a neural network for performing semantic segmentation. Semantic segmentation includes inputting an input image into a convolutional neural network to detect features in the image, upsampling the image back to its original size, and displaying a segmented image in which the features of the image are highlighted and differentiated. For example, as shown inFIG.4, an RGB input image402is received by the processor. The processor passes the input image402through the neural network and returns the input image402to its original size as output image404. In certain embodiments, the processor outputs the output image404with highlighted sections of different colors that differentiate the features detected in the image from other detected features.

The neural network400includes a series of convolutional layers406in a feature detection portion405of the neural network. For purposes of illustration, the convolutional layers406inFIG.4can also represent any number of Bnorm and/or ReLU layers. A wavelet layer408follows each set of convolutional layers406(including any Bnorm and/or ReLU layers). The processor at each wavelet layer408applies a wavelet transform to each image or feature map provided by the previous layers, until the features in the image402can be accurately detected. Once the processor reaches the final set of layers and can distinguish the features in the image402, the features maps produced by the processor at the final layers are passed at reference numeral410to a segmentation portion411of the neural network400. The first layer in the segmentation portion411of the neural network400is an inverse wavelet layer412. The processor at the inverse wavelet layer412applies the inverse of the wavelet transform used in the previous wavelet layers408to begin restoring the image to the original resolution.

After the processor applies the inverse wavelet transform at the inverse wavelet layer412, the feature maps are passed through another series of convolutional (or deconvolutional) layers414, which may also include any number of Bnorm or ReLU layers. The processor continues this process, passing the feature maps through a series of inverse wavelet layers412and convolutional layers414, which in some embodiments can be of the same number of convolutional layers406and wavelet layers408applied in the feature detection portion405of the neural network400. Once the image is restored to its original resolution, the processor at a final activation function layer416, such as a layer including a Softmax activation function or other functions, outputs the segmented output image404.

Referring now toFIGS.5A and5B,FIG.5Aillustrates an example wavelet input image502and an example transformed image504in accordance with embodiments of the present disclosure, andFIG.5Billustrates a block diagram of an example wavelet transform process500in accordance with embodiments of the present disclosure. The processor210of the server200or the processor340of electronic device300can execute the wavelet transform process500. It will be understood that there are a number of existing wavelet transforms, such as Haar, Daubechie, Mallot, and other wavelet transforms, and this disclosure is not limited to any single wavelet transform. The processor receives an image502to be transformed. For grayscale images, the processor can apply the wavelet transform to the image502to produce a transformed image504. For color or RGB images, the processor can apply the wavelet transform to each color layer separately.

As one example, the processor can apply a Haar wavelet transform to the image502. A Haar wavelet transform applies a series of operations on pixel values of an image, arranged in a matrix A. For example, if the first row of matrix A is r1=(88 88 89 90 92 94 96 97), the Haar wavelet transform can group the columns in pairs, as follows: [88, 88], [89, 90], [92, 94], [96, 97]. The first four columns of r1are replaced by the average of each of these pairs (approximation coefficients), and the second four columns of r1are replaced by ½ the difference of these pairs (detail coefficients), such that r1h1=(88 89.5 93 96.5 0 −0.5 −1 −0.5). The processor repeats this process for the first four values of r1h1, while leaving the last four values alone, such that r1h1h2=(88.75 94.75 −0.75 −1.75 0 −0.5 −1 −0.5). The processor then repeats this process on the first two values of r1h1h2, while leaving the last six values alone, such that r1h1h2h3=(91.75−3 −0.75 −1.75 0 −0.5 −1 −0.5). The processor performs this process on every row of the matrix, and then on every column of the matrix, outputting a compressed image.

The processor can also accomplish the transform by matrix multiplication using a defined transform matrix. Multiplying matrix A by the defined matrix can accomplish the previously described steps. For example, if matrix A is an 8×8 matrix, the transform matrix can be defined as

H=[1/81/81/401/20001/81/81/40-1/20001/81/8-1/4001/2001/81/8-1/400-1/2001/8-1/801/4001/201/8-1/801/400-1/201/8-1/80-1/40001/21/8-1/80-1/4000-1/2],
and the product of AH will be the wavelet transform applied to the rows of matrix A. To apply the wavelet transform to the columns, A is multiplied on the left side by HT. Therefore, the entire process can be achieved by the product of HTAH. This wavelet transform H is invertible, such that the original image values can be recreated by applying the inverse of this process to the transformed image. If B=HTAH, then A=(HT)−1MH−1, where B represents the transformed image of A. Since the transform H is invertible, the wavelet transform is lossless, and has an efficient implementation of N log N. Applying such a wavelet transform in place of pooling thus allows for calculations and data to be retained, whereas pooling would discard the data. It will be understood that this is but one example of a Haar wavelet transform for illustrative purposes, and any type of wavelet transform can be applied to the input image502without deviating from the present disclosure.

As shown inFIGS.5A and5B, the processor applies a two-dimensional discrete wavelet transform to separate the output image into four subbands, a low-low (LL) subband506, a low-high (LH) subband508, a high-low (HL) subband510, and a high-high (HH) subband512. The subbands are created by the process500illustrated inFIG.5B. The processor receives the image502and applies both a high pass filter H(z)514and a low pass filter L(z)516to the image502, each to the rows of the image502. The high pass filter514produces a high subband section518of the image502, and the low pass filter516produces a low subband section520. For example, if the image502is an 8×8 image (H×W), the high subband section518and the low subband section520would be 8×4 images. The processor then applies the high pass filter514and the low pass filter516to the columns of the low subband section520, resulting in the LL subband506and LH subband508. The processor also applies the high pass filter514and the low pass filter516to the columns of the high subband section518, resulting in the HL subband510and HH subband512. Each of the subbands506-512are arranged in the image504as shown inFIGS.5A and5B. The quadrants506-512can be used as separate inputs for another layer of a neural network, the processor stacking the quadrants along the channel dimension. It will be understood that low and high pass filters could be applied to the columns first, to create 4×8 images, and then to the rows. This would result in the positions of the LH subband508and the HL subband510being switched in the image504, but this would not affect the performance or operation of the wavelet neural networks described herein. It will also be understood that the low and high pass filters applied to the columns could be different filters from the filters applied to the image502.

FIG.6illustrates an example series of wavelet and inverse wavelet layers600in accordance with embodiments of the present disclosure. The wavelet and inverse wavelet layers600can be executed by the processor210of the server200or the processor340of electronic device300. As described with respect toFIGS.5A and5B, at a wavelet layer of a neural network, the processor applies a two-dimensional discrete wavelet transform to an image602received by the processor. As a result of the wavelet transform applied in the wavelet layer, the processor generates four subbands604of the image602. The processor stacks the subbands along the channel dimension. The processor can further transform the four subbands604by applying high and low pass filters to each of the four subbands604at another wavelet layer of the neural network. It will be understood that between each wavelet layer of the neural network, the processor can convolve the subbands with filters at one or more convolutional layers, and apply batch normalization or ReLU operations to the convolved subbands before applying the wavelet transform of the next wavelet layer.

As a result of applying the wavelet transform to the subbands604, subbands606are created by the processor, which includes 16 different subbands of the image. Any number of wavelet layers may be used depending on the design of the neural network, with the processor increasing the number of subbands by a factor of four at each wavelet layer. It will also be understood that, depending on the number of filters used in the neural network at the convolutional layers, the number of feature maps or subbands can be different. For example, if after subbands604are created the processor convolves each of the subbands at a convolutional layer with multiple filters, then, for example, the convolutional layer could produce eight feature maps from the subbands. After the processor processes the feature maps at the next wavelet layer, the subbands606would then include 32 subbands, with the processor creating four subbands being created from each of the eight feature maps. Thus, the dimensional relationship between inputs and outputs of wavelet layers and inverse wavelet layers is H×W×C−H/2×W/2×4C, such that the resolution is reduced by a factor of 2 in H and W, and the channel depth is expanded by a factor of 4 at the wavelet layers.

In the example illustrated inFIG.6, after the subbands606are created by the processor, the processor executes an inverse wavelet layer, creating subbands608, which are the same subbands as subbands604previously created by the processor. The processor at another inverse wavelet applies an inverse wavelet transform on the subbands608to create an output image610. Since the wavelet transforms and inverse wavelet transforms are lossless operations, the output image610is an accurate recreation of the input image602. As described herein, in other neural networks that use pooling layers, since pooling is a lossy operation and calculations are discarded, during upsampling redundant and/or approximated data is used to recreate the image, which results in a lower quality image. The wavelet and inverse wavelet layers of the present disclosure thus provide for an output image610of the same quality as the input image602.

FIG.7illustrates a block diagram of an example configuration of a convolutional neural network700including wavelet layers in accordance with embodiments of the present disclosure. The processor210of the server200or the processor340of electronic device300can execute the neural network700. An image702is received by the processor. The processor applies a wavelet transform to the image702at a wavelet layer704, creating a plurality of subbands706. The processor convolves the plurality of subbands with one or more filters at a convolutional layer708, creating a plurality of feature maps710. The number of the plurality of feature maps710can depend on the number of filters applied to the subbands706by the processor at the convolutional layer708. For example, if the plurality of subbands706includes four subbands and the processor at convolutional layer708applies one filter to each subband, the plurality of feature maps710includes four feature maps. If the processor applies two filters to each subband, the plurality of feature maps includes eight feature maps, for example.

It will be understood the neural network700can include additional wavelet layers or convolutional layers, and that other layers such as Bnorm and ReLU layers can be included, depending on the design of the neural network and the purpose of the neural network. For example, if the neural network700is designed to classify objects in an image, the neural network700can include additional wavelet and convolutional layers, as well as Bnorm and ReLU layers, to further emphasize features in the image702and to further decrease the resolution of the image. The processor can then flatten the feature maps and input the feature maps into one or more fully connected layers at which the processor determines a classification for one or more objects in the image702. In other embodiments, the neural network700can be designed for semantic segmentation that includes one or more inverse wavelet layers, deconvolutional layers, and other layers, for restoring the original resolution of the image702and segmenting the output image with the features detected by the processor. It will also be understood that the wavelet layer704could come after the convolutional layer708, such that the processor first convolves the image702with one or more filters at the convolutional layer708, and then the processor at the wavelet layer704applies a wavelet transform to the feature maps previously created at the convolutional layer708.

FIG.8illustrates a flowchart of an example convolutional neural network process800including wavelet layers in accordance with embodiments of the present disclosure.FIG.8does not limit the scope of this disclosure to any particular embodiments. While process800depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. For ease of explanation, the process800is described with respect to processor210of the server200ofFIG.2and processor340of the electronic device300ofFIG.3. However, the process800can be used with any other suitable system.

At block802, the processor receives an input image. At block804, the processor applies a wavelet transform to the input image at a wavelet layer of the neural network, generating a plurality of image subbands at block806. At block808, the processor convolves the plurality of subbands with one or more filters at a convolutional layer, generating a plurality of feature maps or weighted subbands at block810. The number of the plurality of feature maps can depend on the number of filters applied to the subbands by the processor at the convolutional layer. For example, if the plurality of subbands includes four subbands and the processor at the convolutional layer applies one filter to each subband, the plurality of feature maps includes four feature maps. If the processor applies two filters to each of the four subbands, the plurality of feature maps includes eight feature maps, for example.

At block812, the processor applies a wavelet transform to the weighted image subbands at another wavelet layer of the neural network, generating another plurality of image subbands at block814. At decision block816, the processor determines if there are additional layers in the neural network, and, if so, the processor performs additional convolutions and wavelet transforms at blocks808-814. If at decision block816the processor determines that there are no additional layers in the neural network, the processor outputs a result818. For example, the processor can output a classification result classifying one or more objects in the image, or other results depending on the design of the neural network.

It will be understood that the process800can include additional wavelet layers or convolutional layers, and that other layers such as Bnorm and ReLU layers can be included and executed by the processor, depending on the design of the neural network and the purpose of the neural network. For example, if the neural network is designed to classify objects in an image, the process800can include additional wavelet and convolutional layers, as well as Bnorm and ReLU layers, to further emphasize features in the input image and to further decrease the resolution of the input image. The processor can then flatten the feature maps and input the feature maps into one or more fully connected layers at which the processor determines a classification for one or more objects in the input image. In other embodiments, the neural network can be designed for semantic segmentation that includes one or more inverse wavelet layers, deconvolutional layers, and other layers, executed by the processor to restore the original resolution of the input image and segment the output image with the features detected by the processor. It will also be understood that the input image can be convolved with the one or more filters first before a wavelet transform is applied, such that the processor first convolves the image with one or more filters at the convolutional layer, and then the processor at the wavelet layer applies a wavelet transform to the feature maps previously created at the convolutional layer.

FIG.9illustrates a block diagram of an example configuration of a convolutional neural network900including wavelet layers in accordance with embodiments of the present disclosure. The processor210of the server200or the processor340of electronic device300can execute the neural network900. An image902is received by the processor. The processor applies a wavelet transform to the image902at a wavelet layer904, creating a LL subband906and a plurality of subbands908. The plurality of subbands908includes HL, HH, and LH subbands. In the example ofFIG.9, the processor convolves the LL subband with one or more filters at a convolutional layer910, creating one or more feature maps912, depending on the number of filters applied at the convolutional layer910. The processor concatenates the plurality of subbands908with the one or more feature maps912at a concatenation layer914, the processor creating a plurality of output feature maps916.

As concatenation can be less resource intensive than performing multiple convolutions, convolving one of the subbands such as the LL subband906and concatenating the subband with the other subbands at the concatenation layer914, rather than convolving all the subbands with filters at a convolutional layer, can result in a more efficient neural network, depending on the design and task to be accomplished by the neural network. For example, if the purpose of the neural network900is to detect and classify objects in an image, and output a classification result, concatenation can be used to increase the speed of the neural network. If the neural network is a pixel-to-pixel or semantic segmentation network, all subbands could be convolved at a convolutional layer, such as illustrated inFIG.7. However, it will be understood that concatenation could be used in pixel-to-pixel, semantic segmentation, or other neural networks, depending on the purpose of the neural network and resources used by the neural network.

It will be understood the neural network900can include additional wavelet layers, convolutional layers, or concatenation layers, and that other layers such as Bnorm and ReLU layers can be included, depending on the design of the neural network and the purpose of the neural network. For example, if the neural network900is designed to classify objects in an image, the neural network900can include additional wavelet, convolutional, and concatenation layers, as well as Bnorm and ReLU layers, to further emphasize features in the image902and to further decrease the resolution of the image. The processor can then flatten the feature maps and input the feature maps into one or more fully connected layers at which the processor determines a classification for one or more objects in the image902.

In other embodiments, the neural network900can be designed for semantic segmentation that includes one or more inverse wavelet layers, deconvolutional layers, and other layers, for restoring the original resolution of the image902and segmenting the output image with the features detected by the neural network900. It will also be understood that the wavelet layer904could come after the convolutional layer910, such that processor first convolves the image902with one or more filters at the convolutional layer910, and then the processor at the wavelet layer904applies a wavelet transform to the feature maps previously created at the convolutional layer910. The processor could then pass the LL subband906to another convolutional layer before concatenation.

FIG.10illustrates a flowchart of an example convolutional neural network process1000including wavelet layers in accordance with embodiments of the present disclosure.FIG.10does not limit the scope of this disclosure to any particular embodiments. While process1000depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. For ease of explanation, the process1000is described with respect to processor210of the server200ofFIG.2and processor340of the electronic device300ofFIG.3. However, the process1000can be used with any other suitable system.

At block1002, the processor receives an input image. At block1004, the processor applies a wavelet transform to the input image at a wavelet layer of the neural network, generating a plurality of image subbands at block1006. At block1008, the processor convolves one of the plurality of subbands, such as the LL subband, with one or more filters at a convolutional layer, generating one or more feature maps or weighted subbands at block1010. At block1012, the processor concatenates the one or more weighted subbands with the other ones of the plurality of image subbands at a concatenation layer to create a plurality of concatenated subbands. The number of the plurality of feature maps or concatenated subbands can depend on the number of filters applied to the subband by the processor at the convolutional layer. For example, if the processor at the convolutional layer applies one filter to the subband, the one or more feature maps includes one feature map. If the processor applies four filters to the subband, the one or more feature maps include four feature maps, for example.

At block1014, the processor applies a wavelet transform to the concatenated subbands at another wavelet layer of the neural network, generating another plurality of image subbands at block1016. At decision block1018, the processor determines if there are additional layers in the neural network, and, if so, the processor performs additional convolutions, concatenations, and wavelet transforms at blocks1008-1016. If at decision block1018the processor determines that there are no additional layers in the neural network, the processor outputs a result. For example, the processor can output a classification result classifying one or more objects in the image, or other results depending on the design of the neural network.

It will be understood that the process1000can include additional wavelet layers, convolutional layers, or concatenation layers, and that other layers such as Bnorm and ReLU layers can be included and executed by the processor, depending on the design of the neural network and the purpose of the neural network. For example, if the neural network is designed to classify objects in an image, the process1000can include additional wavelet, convolutional, and concatenation layers, as well as Bnorm and ReLU layers, to further emphasize features in the input image and to further decrease the resolution of the input image. The processor can then flatten the feature maps and input the feature maps into one or more fully connected layers at which the processor determines a classification for one or more objects in the input image. In other embodiments, the neural network can be designed for semantic segmentation that includes one or more inverse wavelet layers, deconvolutional layers, and other layers, executed by the processor to restore the original resolution of the input image and segment the output image with the features detected by the processor. It will also be understood that the input image can be convolved with the one or more filters first before a wavelet transform is applied, such that processor first convolves the image with one or more filters at the convolutional layer, and then the processor at the wavelet layer applies a wavelet transform to the feature maps previously created at the convolutional layer. The processor could then pass a subband, such as the LL subband, to another convolutional layer before concatenation.

FIG.11illustrates a block diagram of an example neural network1100including an auxiliary encoder1102for feature visualization in accordance with embodiments of the present disclosure. The processor210of the server200or the processor340of electronic device300can execute the neural network1100. The neural network1100can be a network for classifying objects detected in images. For example, the neural network1100can be configured to detect handwritten digits in an image1104received by the processor. The processor convolves the image1104with one or more filters at a convolutional layer1106. The processor applies a wavelet transform to one or more feature maps at a wavelet layer1108, creating a plurality of subbands1110. The number of the plurality of subbands1110can depend on the number of filters applied to the image1104by the processor at the convolutional layer1106. The processor convolves the plurality of subbands with one or more filters at another convolutional layer1112, and applies a wavelet transform to one or more feature maps at another wavelet layer1114, creating a plurality of subbands1116. The number of the plurality of subbands1116can depend on the number of filters applied in the previous layers.

The processor can execute any number of additional convolutional and wavelet layers depending on the design and purpose of the neural network1100. The neural network1100could also include concatenation layers, such as that described with respect toFIGS.9and10. It will be understood that the wavelet layer1108could come before the convolutional layer1106, such that processor first separates the image1104into subbands before convolving the subbands with one or more filters at the convolutional layer1106. It will be understood the neural network1100can include other layers such as Bnorm and ReLU layers, depending on the design of the neural network and the purpose of the neural network. Once the processor executes all the convolutional and wavelet layers in the neural network1100, the processor flattens the feature maps at block1118into an input vector1120and inputs the input vector into one or more fully connected layers1122to generate an output vector1124. At block1126the processor can perform a cross entropy error function on the output vector1124to, for example, define the loss function for network training. The processor can apply one or more labels1128to the output result of the neural network1100to classify the result. For example, if the neural network1100is designed to recognize hand-written digits between 0 and 9, the output vector may include 10 values between 0 and 1, each representing a probability for each of the digits 0-9. For example, an output vector with values [0, 0, 0.1, 0, 0.3, 0, 0, 0, 0, 0.6] indicates a 10% probability that the image1104includes a 2, a 30% probability that the image1104includes a 4, and a 60% probability that the image1104includes a 9. In this case, the processor can label the output as a 9, as the 9 digit has the highest probability.

The neural network1100includes an auxiliary encoder1102as a separate branch of the neural network1100. The auxiliary encoder1102can be used to output an image emphasizing the features that caused the processor executing the neural network1100to reach the output result. The auxiliary encoder1102projects the feature maps at each depth back to the input image resolution so that patterns the neural network has learned can be visualized. The processor applies an inverse wavelet transform to the plurality of subbands1116, or to the subbands created at a subsequent layer if the neural network1100includes additional layers, at an inverse wavelet layer1130to merge the subbands and convolves the subbands at a convolutional layer1132to create a plurality of higher resolution subbands1134. The plurality of higher resolution subbands1134correspond to the subbands1110in the example illustrated inFIG.11. The processor then performs L2 regularization on the plurality of higher resolution subbands1134and on the plurality of subbands1110at block1136.

The processor applies another inverse wavelet transform to the plurality of higher resolution subbands1134at an inverse wavelet layer1138, to merge the subbands, and convolves the subbands at convolutional layer1140to create an image1142at the original resolution of image1104. The processor then performs L2 regularization on the image1104and the image1142at block1144. The image1142includes the original image, with emphasized features of the original image that caused the processor to output the result. For instance, in the example where the neural network1100is configured to recognize hand-written digits between 0 and 9, once the neural network training is complete, the processor can back-propagate vectors through the auxiliary encoder branch1102of the neural network1100to output a visualization of the important features of the input image. For example, the processor can back-propagate a vector of [1, 0, 0, 0, 0, 0, 0, 0, 0, 0] to provide a visualization for the digit 0.FIG.12illustrates a series of example auxiliary encoder output images1202for each of digits 0 through 9, if 10 vectors for each associated with one of the digits are back-propagated through the encoder branch1102. As shown inFIG.12, each of the series of images1202includes white sections and darker sections. The white sections indicate areas where features indicating the digit are present, and the darker sections indicate areas where no content is detected that causes the processor determine the image is of a certain digit. The auxiliary encoder1102thus can be used both to test and optimize a neural network, and to output images indicating detected feature areas, which can be useful for applications such as semantic segmentation.

FIGS.13A and13Billustrate a flowchart of an example wavelet layer convolutional neural network and auxiliary encoder process1300in accordance with embodiments of the present disclosure.FIGS.13A and13Bdo not limit the scope of this disclosure to any particular embodiments. While process1300depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. For ease of explanation, the process1300is described with respect to processor210of the server200ofFIG.2and processor340of the electronic device300ofFIG.3. However, the process1300can be used with any other suitable system.

The neural network with respect to process1300can be a network for classifying objects detected in images. For example, the neural network can be configured to detect handwritten digits in an image received by the processor. At block1302, the processor receives an input image to the neural network. At block1304, the processor convolves one or more input channels of the input image with one or more filters at a convolutional layer to generate one or more output feature maps. At block1306, the processor applies a wavelet transform to the one or more feature maps at a wavelet layer, creating a plurality of subbands. The number of the plurality of subbands can depend on the number of filters applied to the image by the processor at the convolutional layer. At decision block1308, the processor determines if there are additional convolutional or wavelet layers in the neural network. If so, the processor executes blocks1304and1306to convolve the plurality of subbands with one or more filters at another convolutional layer, and to apply a wavelet transform to one or more feature maps at another wavelet layer, creating another plurality of subbands.

The processor can execute any number of additional convolutional and wavelet layers depending on the design and purpose of the neural network. The neural network can also include concatenation layers such as that described with respect toFIGS.9and10. It will be understood that the wavelet layer could come before the convolutional layer, such that the processor first separates the image into subbands before convolving the subbands with one or more filters at the convolutional layer. It will be understood the neural network can include other layers such as Bnorm and ReLU layers, depending on the design of the neural network and the purpose of the neural network. The processor loops the process1300at blocks1304-1308until the processor executes all convolutional and wavelet layers in the neural network. Once the processor executes all the convolutional and wavelet layers in the neural network, at block1310the processor flattens the feature maps into an input vector and inputs the input vector into one or more fully connected layers, generating an output vector at block1312. At block1314, the processor can perform a cross entropy error function on the output vector. The processor can also apply one or more labels to the output result of the neural network to classify the result. For example, if the neural network is designed to recognize hand-written digits between 0 and 9, the output vector may include 10 values between 0 and 1, each representing a probability for each of the digits 0-9.

The neural network of the process1300includes an auxiliary encoder that can be used to output an image emphasizing the features that caused the processor to reach the output result. At block1316, the processor retrieves the feature maps output before the flattening operation performed at block1310. The processor applies an inverse wavelet transform to the feature maps at an inverse wavelet layer of the auxiliary encoder branch of the neural network, to merge the subbands, and, at block1320, convolves the subbands at a convolutional layer to create a plurality of higher resolution subbands. At block1322, the processor performs L2 regularization on the plurality of higher resolution subbands and on the feature maps of the corresponding wavelet layer of the neural network, where the corresponding wavelet layer is the wavelet layer that produced subbands of the same resolution as the features maps created at block1320. At decision block1324, the processor determines if the processor has recreated the original image at the original resolution.

If not, the process1300loops back to blocks1318-1324, where the processor applies another inverse wavelet transform to the plurality of higher resolution subbands at an inverse wavelet layer, convolves the subbands at a convolutional layer and performs L2 regularization on the plurality of higher resolution subbands and on the feature maps of the corresponding wavelet layer of the neural network. If at block1322the original image has been recreated, the processor performs L2 regularization on the recreated image and on the input image. If at decision block1324the processor determines that, as a result of the inverse wavelet transforms and the convolutions, the processor has recreated the original image, at block1326the processor outputs the image emphasizing detected features in the image, as described herein with respect toFIGS.11and12.

Once the network is trained, at block1328, the processor can back-propagate a vector through the auxiliary encoder branch of the neural network to output at block1330a visualization of the features that the network is trained to detect to reach that particular result. In the example herein in which the neural network is configured to recognize hand-written digits between 0 and 9, the processor can back-propagate a vector of [1, 0, 0, 0, 0, 0, 0, 0, 0, 0] to provide a visualization for the digit 0. As shown inFIG.12, each of the series of images1202includes white section and darker sections. The white sections indicate areas where content indicating the digit is present, and the darker sections indicate areas where no content is detected. The auxiliary encoder thus can be used both to test and optimize a network based, and to output images indicating detected areas, which can be useful for applications such as semantic segmentation.