Patent ID: 12243281

DETAILED DESCRIPTION

FIG.1Ashows a schematic view of an image processing arrangement100according to an embodiment of the present invention. The image processing arrangement comprises a controller101, a memory102, an image data receiving device112, such as for example a camera or image sensor, an image streaming device (such as a communication interface) or an image data reading device arranged to read image data from the memory102. The controller101is configured to receive at least one image data file, corresponding to at least an image, from the image data receiving device112, and to perform filtering such as by applying a number of ISPs to the image data file and subsequently to perform object detection (and/or image classification or segmentation) on the resulting (portion of the) image. The image data receiving device112may be comprised in the image processing arrangement100by being housed in a same housing as the image processing arrangement, or by being connected to it, by a wired connection or wirelessly.

It should be noted that the image processing arrangement100may comprise a single device or may be distributed across several devices and apparatuses.

The controller101is also configured to control the overall operation of the image processing arrangement100. In one embodiment, the controller101is a graphics controller. In one embodiment, the controller101is a general purpose controller. In one embodiment, the controller101is a combination of a graphics controller and a general purpose controller. As a skilled person would understand there are many alternatives for how to implement a controller, such as using Field-Programmable Gate Arrays circuits, AISIC, GPU, etc. in addition or as an alternative. For the purpose of this application, all such possibilities and alternatives will be referred to simply as the controller101.

The memory102is configured to store graphics data and computer-readable instructions that when loaded into the controller101indicates how the image processing arrangement100is to be controlled. The memory102may comprise several memory units or devices, but they will be perceived as being part of the same overall memory102. There may be one memory unit for a display arrangement storing graphics data, one memory unit for image capturing device storing settings, one memory for the communications interface (see below) for storing settings, and so on. As a skilled person would understand there are many possibilities of how to select where data should be stored and a general memory102for the image processing arrangement100is therefore seen to comprise any and all such memory units for the purpose of this application. As a skilled person would understand there are many alternatives of how to implement a memory, for example using non-volatile memory circuits, such as EEPROM memory circuits, or using volatile memory circuits, such as RAM memory circuits. For the purpose of this application all such alternatives will be referred to simply as the memory102.

It should be noted that the teachings herein find use in arrangements for object detection, segmentation and/or image classification and image filtering in many areas of computer vision, including object detection in mixed or augmented reality systems, image retrieval, industrial use, robotic vision and video surveillance where a basic image processing arrangement100such as inFIG.1Amay be utilized. In one embodiment, the image processing arrangement100is a digital camera or other image sensor device (or comprised in such device). In one embodiment, the image processing arrangement100is connected to a digital camera or other image sensor device.

FIG.1Bshows a schematic view of an image processing arrangement being a viewing device100according to an embodiment of the present invention. In this embodiment, the viewing device100is a smartphone or a tablet computer. In such an embodiment, the viewing device further comprises a display arrangement110, which may be a touch display, and the image data receiving device112may be a series of cameras of the smartphone or tablet computer. In such an embodiment the controller101is configured to receive an image from the camera (or other image receiving device)112, detect objects in the image and display the image on the display arrangement110along with virtual content indicating or being associated with the detected object(s). In the example embodiment ofFIG.1B, the camera112is arranged on a backside (opposite side of the display110, as is indicated by the dotted contour of the camera112) of the image processing arrangement100for enabling real life objects behind the image processing arrangement100to be captured and shown to a user (not shown inFIG.1B) on the display110along with any displayed virtual content. The displayed virtual content may be information and/or graphics indicating and/or giving information on detected objects.

FIG.1Cshows a schematic view of an image processing arrangement being or being part of an optical see-through (OST) viewing device100according to an embodiment of the present invention. The viewing device100is a see-through device, where a user looks in through one end, and sees the real-life objects in the line of sight at the other end of the viewing device100. The viewing device100is in one embodiment a virtual reality device.

In one embodiment the viewing device100is a head-mounted viewing device100to be worn by a user (not shown explicitly inFIG.1C) for looking through the viewing device100. In one such embodiment the viewing device100is arranged as glasses, or other eye wear including goggles, to be worn by a user.

The viewing device100is in one embodiment arranged to be hand-held, whereby a user can hold up the viewing device100to look through it.

The viewing device100is in one embodiment arranged to be mounted on for example a tripod, whereby a user can mount the viewing device100in a convenient arrangement for looking through it. In one such embodiment, the viewing device100may be mounted on a dashboard of a car or other vehicle.

The viewing device comprises a display arrangement110for presenting virtual content to a viewer and an image data receiving device112for identifying or detecting objects. As disclosed above with reference toFIG.1A, the image data receiving device112may be remote and comprised in the image processing arrangement through a connection to the image processing arrangement100.

It should also be noted that even if only one image data receiving device112is discussed in the above, the image data receiving device is arranged to receive image data relating to more than one image, such as a video sequence or from parallel image sources, for advanced photo manipulation effects.

FIG.1Dshows a schematic view of an image processing arrangement100according to an embodiment of the present invention. In this embodiment, the image processing arrangement100is a computer. In one such embodiment, the image processing arrangement is arranged to operate as a second image processing arrangement arranged to receive an image from another (first) image processing arrangement as will be discussed in the below with reference toFIG.2. A first image processing arrangement100is generally seen as an image receiver and the second image processing arrangement100is generally seen as an image processor. It should be noted that even though the description of the image processing arrangement100ofFIG.1Dis aimed at a computer, all embodiments of image processing arrangements given with reference toFIGS.1A,1Bmay also be arranged to operate as a second image processing arrangement. Specific examples of embodiments operating as a second image processing arrangement100D are computers, laptop computers, tablet computers and/or smartphones. In one embodiment, the computer100is comprised in a cloud service and may thus be a collection of computers or other computational resources, commonly referred to herein as a cloud computer.

In the following, simultaneous reference will be made to the image processing arrangements100ofFIGS.1A,1B,1C and1D.

In one embodiment the image processing arrangement100further comprises a communication interface103. The communication interface may be wired and/or wireless. The communication interface103may comprise several interfaces.

In one embodiment the communication interface comprises a USB (Universal Serial Bus) interface. In one embodiment the communication interface comprises a HDMI (High Definition Multimedia Interface) interface. In one embodiment the communication interface comprises a Display Port interface. In one embodiment the communication interface comprises an Ethernet interface. In one embodiment the communication interface comprises a MIPI (Mobile Industry Processor Interface) interface. In one embodiment the communication interface comprises an analog interface, a CAN (Controller Area Network) bus interface, an I2C (Inter-Integrated Circuit) interface, or other interface.

In one embodiment the communication interface comprises a radio frequency (RF) communications interface. In one such embodiment the communication interface comprises a Bluetooth™ interface, a WiFi™ interface, a ZigBee™ interface, a RFID™ (Radio Frequency IDentifier) interface, Wireless Display (WiDi) interface, Miracast interface, and/or other RF interface commonly used for short range RF communication. In an alternative or supplemental such embodiment the communication interface comprises a cellular communications interface such as a fifth generation (5G) cellular communication interface, an LTE (Long Term Evolution) interface, a GSM (Global Systéme Mobile) interface and/or other interface commonly used for cellular communication. In one embodiment the communications interface is configured to communicate using the UPnP (Universal Plug n Play) protocol. In one embodiment the communications interface is configured to communicate using the DLNA (Digital Living Network Appliance) protocol.

In one embodiment, the communications interface103is configured to enable communication through more than one of the example technologies given above. As an example, a wired interface, such as MIPI could be used for establishing an interface between the display arrangement, the controller and the user interface, and a wireless interface, for example WiFi™ could be used to enable communication between the image processing arrangement100and an external host device (not shown).

The communications interface103is configured to enable the image processing arrangement100to communicate with other devices, such as other image processing arrangements100and/or smartphones, Internet tablets, computer tablets or other computers, media devices, such as television sets, gaming consoles, video viewer or projectors (not shown), or image capturing devices for receiving the image data streams. In particular, the communications interface103is configured to enable the image processing arrangement100to communicate with a second image processing arrangement.

A user interface104may be comprised in or be connected to the image processing arrangement100(only shown inFIG.1BandFIG.1D). Additionally or alternatively, (at least a part of) the user interface104may be comprised remotely in the image processing arrangement100through the communication interface103, the user interface then (at least a part of it) not being a physical means in the image processing arrangement100, but implemented by receiving user input through a remote device (not shown) through the communication interface103. One example of such a remote device is a game controller, a mobile phone handset, a tablet computer or a computer.

FIG.2shows a schematic view of an object detection system200according to an embodiment herein. The object detection system200comprises a first image processing arrangement100A and at least one second image processing arrangement100B. As noted above, the first image processing arrangement100A is arranged to operate as an image receiver and the second image processing arrangement100B is arranged to operate as an image processor. By enabling a first image processing arrangement100A to transmit an image to a second image processing arrangement100B for processing, in particular for performing tasks such as object detection, segmentation and/or image classification, the first image processing arrangement100A may be relieved of heavy computations which enables for producing smaller and/or cheaper image processing arrangements.

In the example ofFIG.2, the first image processing arrangement100A is receiving an image that contains a number of objects. In the example ofFIG.2, the image is received through the image receiving device112, for example a camera that captures an image of three objects201-203. It should be noted that this is only an example and an image may contain any (including zero) number of objects and there are many different ways for an image to be received. As stated and as would be understood, an image may be received by being captured by a camera, by being fetched or retrieved from a memory102(local or remote) or being received through the communication interface103, such as when streaming.

In a specific example, the first image processing arrangement100A is a viewing device (such as disclosed with reference toFIG.1C) arranged for virtual, augmented or other manipulated realities, and the second image processing arrangement100B is a smartphone (such as disclosed with reference toFIG.1B) or a computer (such as disclosed with reference toFIG.1D).

The general principle of the teachings herein will now be disclosed using specific example with simultaneous reference to the system ofFIG.2and the method ofFIG.3.

FIG.3shows a flowchart of a general method according to an embodiment of the teachings herein. The method utilizes an image processing arrangement100as taught herein. The method comprises a (first) image processing arrangement100(A) receiving310an image, as is also disclosed above with reference toFIG.2, the first image processing arrangement100A receiving an image containing the three objects201-203. As is also stated above, the image may be received from an external source or from the memory102, or from a camera112of the image processing arrangement100. For the purpose of the teachings herein there will be made no difference between the mage and the data file representing the image and the two will be used interchangeably and it should be noted that a skilled person would understand when reference is made to the actual image and when reference is made to the data file (or a conversion of it) representing the image.

As a next step, the image would be compressed and transmitted to the second image processing arrangement100B for (further) processing. A joint (end-to-end) training of compression and processing, such as autoencoder and inference of networks, to achieve a better combination of inference accuracy and compression ratio is beneficial. However, as realized by the inventors, and hinted at above, this means that the first and second image processing arrangements100A and100B are jointly trained for specific tasks, and the resulting weights and resulting compressed bit stream are then specific for such a trained image processing arrangement pair100A-100B. The image processing arrangement pair will thus operate efficiently for the scenario it was trained for, however, for other scenarios, where potentially other objects should be detected, other tasks are to be performed or where the models or training have been further optimized, the resulting weights and bit-stream (compressed image) would be different. A first image processing arrangement100A trained for a certain model or task-set, producing a data-stream DA, might not be correctly decoded or processed with a second image processing arrangement100B trained for another model or task-set. In today's society where adaptability and compatibility are of utmost important for devices to be successful, this is a real problem that the inventors have realized. As VR goggles have been researched since the early nineties and as the first commercially successful VR goggle (Oculus VR) paired with a smartphone was initiated already in 2012 the problem of how to achieve sufficiently efficient VR goggles is a long-standing problem and the inventors have realized a significant manner of solving this overall long-standing problem.

Furthermore, there are several reasons that the inventors have insightfully identified through inventive reasoning why different weights, and even different neural network models, should be available for performing different tasks. Examples of such reasons are: further optimizations of a system (such as the system200inFIG.2) provides revisions with better accuracy/compression, or other (or additional) objects are to be detected or identified. It should be pointed out that in object detection (as well as classification and segmentation) if a system is not trained to recognize for example dogs, it will not be able to detect dogs and the system thus has to be trained explicitly to do so. Furthermore, in a system, several trained models/weights might co-exist: one image retrieving device112, such as a camera sensor, might switch between weight-sets depending on scenario (e.g. suitable to detect different types of objects), or there might be several different sensors producing data and these have differing models/weights. Another aspect is if the system toggles between scenarios where the bit-stream should be visible at the decoder side (i.e. at the second image processing arrangement100B) vs. when only a machine shall consume the data—then it would be natural to switch between encoding/decoding tasks. Furthermore, if the compressed bit-stream is stored in a file, it is imperative that it is understood with which decoder such data should be later decoded and analyzed. Contemporary techniques do not describe these problems, nor do they provide solutions. However, the inventors have both realized these situations, the associated problem(s) and are proposing a solution to overcome or at least mitigate these shortcomings.

As discussed briefly above, image compression with end-to-end training of neural network has achieved a significant progress during recent years. A notable technique is to approximate the quantization process with an addition of uniform noise in the latent variable space. This enables a back-propagation through the quantization process. Another technique that can be used to render the end-to-end possible, is to estimate the entropy of the latent variables to approximate their bit-rate for compression as the entropy is the lower bound of the achievable rate in average. An approach with a variational autoencoder has outperformed the-state-of the-art in terms of image coding. By introducing additional latent variables conditioned on the variables for the compressed representation, the spatial dependencies left are modeled.

Compression can be conducted through an end-to-end training manner. The inference network for a given task can then be cascaded to the autoencoder for an end-to-end training to obtain an accurate inference while minimizing the compression bit-rate. Because such encoder-decoder pairs are jointly trained and optimized for certain tasks, the data flow becomes also dependent on this encoder-decoder pair (or image processing arrangement pair).

The proposed solution addresses the problem of multiple networks/weights, i.e., to switch the weights of the neural network for a specific task by using a task identification (ID) number.

The first image processing arrangement100A is thus configured to, prior to compressing330the image, select320which task is to be performed, or rather how the task is to be performed by selecting the model and the parameters to be used, and to select325a task identification (ID) (for example a number) that identifies or indicates the selected task.

This task ID is communicated between the two image processing arrangements prior to the second image processing arrangement100B executing the task. The task ID may be communicated by being transmitted along with the compressed image, or by being (possibly) implicitly indicated through the task selection process. As will be discussed below, the task selection320may be performed through a hand-shake protocol between the first image processing arrangement100A and the second image processing arrangement100B.

Based on the task ID, the corresponding weights or other parameters are loaded on both image processing arrangements for consequent execution. The task parameters may be retrieved from a local memory, such as the memory102, or from a remote storage, such as through a cloud service. For the purpose of the teachings herein, there will not be made any difference from where the task parameters are retrieved. The task parameters may be seen to be retrieved as part of selecting325the task ID by the first image processing arrangement100A.

As the task (and the corresponding task ID) has been selected and the task parameters have been retrieved, the first image processing arrangement100A compresses330the image based on the task parameters. As is indicated inFIG.3, an optional (pre-) processing335may also be performed by the first image processing arrangement100A for increasing the accuracy of the task, by performing some processing that is beneficially performed while compressing the image. The compressed image is then transmitted340from the first image processing arrangement100A to the second image processing arrangement100B. The task ID is also transmitted340, as discussed above, possibly along with the image.

As the second image processing arrangement100B receives350the compressed image, the second image processing arrangement100B will retrieve350the task parameters based on the received task ID (possibly received along with the image, or indicated through the task selection process). The task parameters may be retrieved as part of receiving (345) the task ID by the second image processing arrangement100B, or when selecting (320A) the task. As the task parameters have been retrieved, the task is performed, i.e. the second image processing arrangement100B processes355the image based on the task parameters. In the example of object detection, the second image processing arrangement100B thereby detects360the object(s)201-203in the image and transmits370indications of the detected object(s)201-203to the first image processing arrangement100A. For other tasks, the second image processing arrangement100B analyzes360the results of the processing and transmits370indications of the results to the first image processing arrangement100A. Optionally the second image processing arrangement100B may also display or otherwise utilize the results of the processing at its end. For example, the image may be displayed along with the detected objects on a display110or through other user interface104of the second image processing arrangement100B.

A second image processing arrangement100B might support multiple sets of weights and be capable of receiving alternative streams from different first image processing arrangements100A with different weight-sets, and the communication of task IDs guides the selection of as well as switching of weights or other task parameters.

As discussed above, the relation between task ID and task parameters (such as weights and/or model descriptions), i.e. task data, can be stored in a location available to all devices, such as in a cloud server. The task data can be stored as pure data or as encrypted data for security perspectives. Alternatively or additionally, the task data can be stored at each image processing arrangement. In one embodiment some task data is stored remotely and some task data is stored locally.

Returning to the first image processing arrangement100A, it receives380the result(s) of the processing and utilizes390the results. In the example of object detection, the first image processing arrangement100A receives380the object(s) or at least indication(s) of the object(s) and displays them, or otherwise indicates390the object(s), for example on the display device110or through other user interface104.

As has been discussed in the above, the first image processing arrangement100A selects which task is to be used. In order for an optimum (or at least beneficial) performance to be achieved and to enable for a higher adaptability for different image processing arrangement pairs, the first image processing arrangement100A may be configured to select which task parameters, i.e. specific task, that is to be used for an overall task. As the overall task has been selected, the first image processing arrangement100A may thus communicate this to the second image processing arrangement100B whereby a handshake protocol is initiated between the first image processing arrangement100A and the second image processing arrangement100B for selecting320/320A the task data that provides the better performance for both image processing arrangements. For example, if the first image processing arrangement100A wants the bitrate to be constrained to a certain threshold, but the second image processing arrangement100B also wants to minimize its cost (energy usage, etc., as reloading the weight may consume extra energy for the devices), one interesting outcome could be that the second image processing arrangement100B prefers to keep the weights that are already loaded into local memory102(thereby for example saving energy and time by not loading the task data) if such a decision also meets the bitrate constraint of the source.

As a generalization of the handshake process for multiple task operations, task scheduling among the devices can be introduced, i.e., to determine which task to run first. For example, the first image processing arrangement100A may stipulate its task with some constraints (delay, for example). Then, the second image processing arrangement100B (potentially capable of hosting multiple models, weight-sets) should optimize for two metrics simultaneously: minimizing its energy usage (e.g. by keeping current weights in memory as long as possible) and minimizing the maximum delay experienced by client tasks.

As indicated above, the networks are trained for a specific task, such as a specific object detection task. When switching task, for example for performing object detection for a different set of objects, firstly, the task ID of the corresponding weights for the neural networks is communicated between image processing arrangements. Thereafter, the weights and/or other task parameters (i.e. the task data) are loaded into the neural networks from the local memory. The task data of the neural network are stored after training in the local memory and synced between both image processing arrangements. Additionally or alternatively, the task data can be stored in a network database, such as a cloud service. For example, the task data are only stored in an online database and are synced to one or both of the image processing arrangements when switching tasks. It should be noted that the first image processing arrangement100A may store a different set of task data locally, than the second image processing arrangement100B.

Returning to the specific example ofFIG.2, where a user wearing AR goggles (first image processing arrangement100A, such as the viewing device disclosed with reference toFIG.1C) is watching traffic (objects201-203) flow by on a highway. As the controller101of the AR goggles100A receives310an image of cars (objects)201-203through the camera112, a task is selected320and a task ID is selected325along with retrieving task parameters from the local memory102(assuming locally stored task parameters) or possibly through the communication interface103of the AR goggles100A (assuming remotely stored parameters). Thereafter the image is compressed330(and possibly (pre-)processed335) by the controller101before being transmitted340to a smartphone of the user (the second image processing arrangement100B, such as the smartphone disclosed with reference toFIG.1B) through the communication interface103of the AR goggles100A. The controller101of the smartphone100B receives345the (compressed) image through the communication interface103of the smartphone100B and retrieves350the task ID and associated task parameters from the memory102of the smartphone100B (assuming locally stored parameters) or possibly through the communication interface103of the smartphone100B (assuming remotely stored parameters). The controller101of the smartphone100B then performs object detection355(i.e. processes) on the image and detects three cars (objects201-203). The three cars201-203are identified as one being a police car and the two others being civilian or non-descript cars. An indication of the detected cars201-203and their classifications are transmitted370through the communication interfaces103of the smartphone100B and the AR goggles103to the AR goggles100A that receives380the indications of the cars201-203and indicates390the classifications of the cars201-203as one being a police car, by marking the car being a police car with a square and the text label “POLICE” overlaying the portion of the image representing the police car on the display device110of the AR goggles100A.

In one embodiment, the task ID is encrypted to provide for an easy manner of providing security. By only encrypting the task ID, a minimum of computational resources are used to provide sufficient security as the compressed image data is unintelligible without knowing which task parameters associated with the task ID was used when compressing the image. Prior to the communication of the task ID, a public key and a private key for an encryption are generated and deployed in the respective image processing arrangements. An example is to deploy the public key at the first image processing arrangement and the private key at the second image processing arrangement. When the task selecting operation is to be performed, for example, the first image processing arrangement determines the appropriate model/weight, i.e. task data, to use, encrypt its task ID and sends the encrypted task ID to the second image processing arrangement100B. Upon the second image processing arrangement100B receiving the encrypted ID, the task ID is decrypted using the private key by the second image processing arrangement100B. The corresponding task parameters are then chosen for performing the task. In the case when a handshake is performed. A feedback will be generated from the second image processing arrangement100B to the first image processing arrangement100A to confirm or reject the task selection request320/320A.

Additionally or alternatively, the second image processing arrangement100B can also suggest a task ID, which is encrypted and sent to the first image processing arrangement100A based on the optimization outcome mentioned in the handshake process.

FIG.4shows a component view for a software component (or module) arrangement400according to an embodiment of the teachings herein. The software component arrangement400is adapted to be used in an image processing arrangement system200as taught herein for providing image processing possibly object detection, segmentation and/or image classification as taught herein. The image processing arrangement system200comprises a first image processing arrangement100A and a second image processing arrangement100. The software component arrangement400comprises a software component module for receiving410an image in the first image processing arrangement100A and a software component module for selecting420a task and425a task identifier associated with task data in the first image processing arrangement100A. The software component arrangement400also comprises a software component module for compressing430the image based on the task data in the first image processing arrangement100A and a software component module for transmitting440the compressed image to the second image processing arrangement100B for processing. The software component arrangement400also comprises a software component module for receiving445the compressed image and task identifier in the second image processing arrangement100B and a software component module for retrieving450task parameters associated with the task identifier in the second image processing arrangement100B. The software component arrangement400also comprises a software component module for processing455the compressed image based on the task parameters in the second image processing arrangement100B, a software component module for determining460results and a software component module for transmitting470the at least indications of the determined results to the first image processing arrangement100A. The software component arrangement400also comprises a software component module for receiving480at least indications of a result of the processing from the second image processing arrangement100B in the first image processing arrangement100A and a software component module for indicating490the result.

FIG.5shows a component view for an image processing arrangement500according to an embodiment of the teachings herein. The image processing arrangement500is adapted to be used in an image processing arrangement system200as taught herein for providing image processing possibly object detection, segmentation and/or image classification as taught herein. The image processing arrangement system200comprises a first image processing arrangement100A and a second image processing arrangement100. The image processing arrangement500comprises a circuitry for receiving510an image in the first image processing arrangement100A and a circuitry for selecting520a task and525a task identifier associated with task data in the first image processing arrangement100A. The image processing arrangement500also comprises a circuitry for compressing530the image based on the task data in the first image processing arrangement100A and a circuitry for transmitting540the compressed image to the second image processing arrangement100B for processing. The image processing arrangement500also comprises a circuitry for receiving545the compressed image and task identifier in the second image processing arrangement100B and a circuitry for retrieving550task parameters associated with the task identifier in the second image processing arrangement100B. The image processing arrangement500also comprises a circuitry for processing555the compressed image based on the task parameters in the second image processing arrangement100B, a circuitry for determining560results and a circuitry for transmitting570the at least indications of the determined results to the first image processing arrangement100A. The image processing arrangement500also comprises a circuitry for receiving580at least indications of a result of the processing from the second image processing arrangement100B in the first image processing arrangement100A and a circuitry for indicating590the result.

FIG.6shows a schematic view of a computer-readable medium120carrying computer instructions121that when loaded into and executed by a controller of an image processing arrangement100enables the image processing arrangement100to implement the present invention.

The computer-readable medium120may be tangible such as a hard drive or a flash memory, for example a USB memory stick or a cloud server. Alternatively, the computer-readable medium120may be intangible such as a signal carrying the computer instructions enabling the computer instructions to be downloaded through a network connection, such as an internet connection.

In the example ofFIG.6, a computer-readable medium120is shown as being a computer disc120carrying computer-readable computer instructions121, being inserted in a computer disc reader122. The computer disc reader122may be part of a cloud server123—or other server—or the computer disc reader may be connected to a cloud server123—or other server. The cloud server123may be part of the internet or at least connected to the internet. The cloud server123may alternatively be connected through a proprietary or dedicated connection. In one example embodiment, the computer instructions are stored at a remote server123and be downloaded to the memory102of the image processing arrangement100for being executed by the controller101.

The computer disc reader122may also or alternatively be connected to (or possibly inserted into) an image processing arrangement100for transferring the computer-readable computer instructions121to a controller of the image processing arrangement (presumably via a memory of the image processing arrangement100).

FIG.6shows both the situation when an image processing arrangement100receives the computer-readable computer instructions121via a wireless server connection (non-tangible) and the situation when another image processing arrangement100receives the computer-readable computer instructions121through a wired interface (tangible). This enables for computer-readable computer instructions121being downloaded into an image processing arrangement100thereby enabling the image processing arrangement100to operate according to and implement the invention as disclosed herein.

A detailed description will now be given as to how the processing may be performed over the first and the second image processing arrangements.FIG.7shows a schematic view of a system, such as the system200ofFIG.2, according to one embodiment of the teachings herein.

The system comprises a first image processing arrangement100A and a second image processing arrangement100B, such as any of the image processing arrangements disclosed in reference toFIGS.1A-1Dabove. The first image processing arrangement100and the second image processing arrangement100B constitutes a compression and -interference framework, in an embodiment where the processing is inference for later object detection, segmentation or classification. The first image processing arrangement100A comprises encoding (or compressing) neural networks710for performing the compression (and possibly (pre-) processing) including encoding, quantization (Q), and decoding. The encoding neural networks710are arranged to receive an image X possibly from the camera112, whereby the encoding neural network will compress and encode the image into a compressed format C which then undergoes a quantization process Q. The first image processing arrangement100A also comprises an encoder720possibly a part of or at least connected to a communication interface103for transmitting the quantized and encoded bit stream to the second image processing arrangement100B. The second image processing arrangement100B comprises a decoder730possibly a part of or at least connected to a communication interface103for receiving the encoded bit stream from the first image processing arrangement100A. The second image processing arrangement100B also comprises a processing neural network740, which may comprise a decoding or decompressing network740′ and a task network740″, such as a network for inference and later object detection, segmentation or classification.

As discussed above, the networks710and740are trained end-to end to provide as high accuracy of the task to be performed as possible. The objective is to minimize the bit-rate (R) and the detection loss (L) function (i.e. task accuracy) at the same time. Without the loss of generality, a weight λ1 is also added to take into account the distortion (D(X, X′)) between the original signal (X) and the reconstructed signal (X′). Setting λ1 to zero eliminates the effect of this constrain. The training block750-1indicates one alternative for training the neural networks end-to-end, where the bit rate (R) is expressed as a function of the quantization process Q(Y), the distortion D(X, X′) is expressed as a function of the original signal X and the reconstructed signal X′ and where the detection (or other task) accuracy loss function L is expressed as a function L(P,G) of the prediction (or task) results P and a factor G, the ground truth [be the “actual” information provided by direct measurement or observation. Instead of the result provided by the inference (prediction). For this kind of task, the ground truth would be the label (e.g., class and bounding box) provided for training the object detector (the image processing arrangement). As indicated, various elements may be weighted by weights □. In the example shown, the distortion D and the accuracy loss L are weighted by weights □1 and □2 respectively.

Alternatively, a variation as discussed above, in which the decoding network and the object detection (or task) network are merged together into a composite task network740. This can possibly reduce the number of layers and parameters for the decoding and object detection network when the perceptual quality is not considered at all, as is the case if the second image processing arrangement100B is not to display the resulting image and only to provide task results to the first image processing arrangement100A. In such an alternative, the training block750-2indicates one alternative for training the neural networks end-to-end, where the bit rate (R) is expressed as a function of the quantization process Q(Y) and where the detection accuracy loss function L is expressed as a function L(P,G) of the prediction (or task) results P and a factor G, the ground truth. As indicated, various elements may be weighted by weights □. In the example shown, the accuracy loss L are weighted by weight □2.

The deployment of the system200after training is shown inFIG.7, where the encoder network710resides in the first image processing arrangement100A, i.e. the capture device (for example a light-weight head-set as in the example ofFIG.7), and the decoder plus the object detector network740work in a separate more powerful processing device, i.e. the second image processing arrangement100B, such as a smartphone as in the example ofFIG.7. The system200may be arranged to switch task and load new weights and/or other task parameters based on the task ID number as discussed above with reference toFIGS.2and3. InFIG.7there is also shown that the first image processing arrangement100A and the second image processing arrangement100B comprises a task data database760, possibly stored locally in the memory102of the image processing arrangement and/or stored remotely and accessible through the communication interface103.

Returning to the method disclosed through the flowchart ofFIG.3and discussed above, an image is received310and compressed330by being passed to the encoding network710and quantized (Q) into latent representation for encoding possibly through entropy encoding and340transmitted to the second image processing arrangement100B through the encoder720(encoding the image for transmission) and received345by the decoder730(decoding the image after transmission) of the second image processing arrangement100B, where the received image is passed to the object detection network740for performing345of the task that has been previously selected320, and which task ID has also been previously communicated or communicated along with the image for proper selection and retrieval of task parameters based on the task ID by the first image processing arrangement100A (320,325) and by the second image processing arrangement100B (350) respectively from the respective data bases760so that the second image processing arrangement100B can process355the image properly to produce360the detection result. The detection results, or at least indications of them, are transmitted370to the first image processing arrangement100A where they are received380and utilized390.

In one embodiment, the encoding network710consists of at least one layer of Convolutional Neural Network (CNN), and at least one non-linear layer, e.g., Rectified Linear Units (ReLUs), Generalized Divisive Normalization (GDN). A down-sampling process can be performed in such an embodiment in the CNN operation by adjusting the stride. This has one benefit in that the parameter size is reduced and the bit rate is also reduced.

In one embodiment, the decoding network740is the inverse process of the encoding network710, e.g., at least one layer of CNN and at least one non-linear layer. An up-sampling process can similarly be conducted in the CNN operation by adjusting the stride.

In one embodiment, the encoding network710and the decoding network740can take the form of autoencoders and variational autoencoders. This has a benefit in that it benefits from a proven performance of autoencoders and variational autoencoders in image compression, compared to the state-of-the-art conventional methods. In addition, these encoders can reconstruct images/parameters for a full image at once versus block-based conventional encoders. This also enable the end-to-end training

In one embodiment, the quantization process Q is approximated with an addition of noise of uniform distribution during training, and the original quantization process is used during inference/encoding. This has a benefit in that the generalization of the end to end model is enhanced because introducing noise acts as a regularization mechanism.

In one embodiment, the bit rate of the compressed representation is approximated by the entropy. The density of the variables after the approximated quantization can be estimated with Non-parametric models, e.g. piece-wise smooth functions, or parametric models, e.g., a mixture of Laplacian model. The probability of the quantized variable is equal to an integral of the estimated density within the corresponding quantization bin. This has a benefit in that the process estimates the entropy of the source in an accurate way

In one embodiment, the object detection network740can be a state-of-the-art object detector, e.g., YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), or faster R-CNN (Recursive Convolutional Neural Network).

In one embodiment, when the decoding network740′ and the objection detection network740″ are merged, the decoding network740is simplified. This simplification can be done by reducing the number of layers as well as number of filter channels.

In one embodiment, the detection accuracy loss, L, is a weighted sum of the confidence loss and the localization loss from the detected object class and its associated bounding box, respectively. This has a benefit in that it captures the contribution of loss from both classification and regression sub-tasks and the tradeoff for both the Loss from inference task and constraints.

In one embodiment, the distortion metric Mean Square Error MSE can be used to measure the distortion, D. This has a benefit of simplicity and wide adaptation in the community.

In one embodiment, the weighted parameters λ1 and λ2 are determined empirically. An increase of λ1 enforces the reconstructed signal to resemble the original input signal.

In one embodiment, the entropy coding process utilizes an arithmetic encoder to produce the compressed bits. In another embodiment, the Content Adaptive Binary Arithmetical Coding (CABAC) can be applied to encode and decode the bit stream after binarization.

In one such embodiment, the task ID includes a field indicator identifying it as encrypted to allow systems combining bit streams with encrypted as well as open task IDs. In some embodiments, the encryption/decryption key is stored in both devices. In another embodiment, the bit stream is aimed for a specific decoder (second) image processing arrangement. In such systems, the ID might be encrypted with the second image processing arrangement's public key.