Patent Description:
Different solutions exist for providing surveillance image data or a vision system for vehicles.

For example, a camera system where a wide field-of-view is generated by a camera mounted to a motorized gimbal which combines images captured at different times and different directions into a single aggregate image. This system relies on covering a wide field-of-view by changing the direction of the camera and is able to simultaneously capture images from the multiple cameras.

Patent application <CIT> discloses vehicle-mountable imaging systems and methods that combine warping, fusing and stitching images from imaging sensors with partially overlapping FOVs.

Patent application <CIT> discloses that images may be stitched using one of static steam-based stitching, dynamic seam-based stitching or dynamic warp stitching, or by applying different on of these stitching methods on different sub-areas of the images.

Article "<NPL>, discloses a military driver night sight periscope with image fusion.

Internet publication "<NUM>° Advanced Situational Awareness Day & Night for Army Applications" by Wärtsilä Euroatlas, retrieved from internet URL: https://cdn. com/docs/default-source/euroatlas-files/vision-systems/b6429-<NUM><NUM> advance-situational-awareness-leaflet-<NUM>. pdf?sfvrsn=5de412446 discloses a situational awareness systems that performs image fusion and image stitching.

Internet publication "<NUM>° Day & Night Vision System for Army Applications" by Wärtsilä, retrieved from internet URL: https://cdn. com/docs/default-source/euroatlas-files/vision-systems/b6429-<NUM><NUM>-day-night-vision-system-leaflet-<NUM>. pdf?sfvrsn=4ee41244 <NUM> discloses a military driver sight system that has high resolution camera sensors and long wave infrared.

Internet publication "Vision systems" by Wärtsilä Euroatlas, retrieved from internet URL: https://www. de/product-services/products/vision-systems discloses night vision devices wherein two images are stitched in software to provide a wide field of view.

However, there are several aspects and demanding requirements that need to be considered for the design, development, and qualification of a camera system for military vehicles, for example.

A highly reliable and precisely engineered camera system is essential in meeting the mission objectives, and ensuring the safety and survival of the vehicle's crew in different operating modes and circumstances.

Thus, a solution is needed to enable accurate, efficient, and reliable method for providing surveillance image data or a vision system for vehicles.

According to a first example aspect of the present invention, there is provided a surveillance apparatus carried by a vehicle for generating surveillance image data, comprising:.

In an embodiment, static stitching components include at least one of following: vertical adaption information; perspective correction information; and cutting information.

In an embodiment, dynamic stitching components include at least one of following: contrast adaptation information; color adaptation information; and blending of image transition information.

In an embodiment, in the stitching mode, image stitching of different channels is configured to be arranged simultaneously.

In an embodiment, the apparatus is further configured to determine a backup operating mode, wherein a multiplexer function is activated to provide a video signal bypass image data for the display without processing the image data with the wide field of view model.

In an embodiment, the at least two camera devices associated to the first channel are arranged vertically alongside each other.

In an embodiment, the at least two camera devices associated to the second channel are arranged vertically alongside each other.

In an embodiment, the at least two camera devices associated to the first channel are arranged horizontally alongside the at least two camera devices associated to the second channel.

In an embodiment, the camera device comprises at least one of the following: LWIR sensor, SWIR sensor, CMOS sensor, image Intensified CMOS, and CCD sensor.

In an embodiment, in the second operating mode, combining the image data of both channels comprises at least one of stitching and fusing of the image data.

In an embodiment, in the second operating mode, combining the image data of both channels comprises a plurality of different fusion modes for the image data.

In an embodiment, the image data comprises video data and/or still image data.

In an embodiment, the output mode comprises at least one of the following: full screen (FS) mode, split screen (SS) mode, picture-in-picture (PIP) mode and fusion mode (FUS).

According to a second example aspect of the present invention, there is provided a computer-implemented method for generating surveillance image data by at least one processor of an apparatus carried by a vehicle, wherein a plurality of camera devices carried by the vehicle are configured to provide image data on different channels, and wherein each channel is associated with at least two camera devices with substantially same field-of-view (FOV) angles but different line-of-sight (LOS) angles, the method comprising:.

According to a third example aspect of the present invention, there is provided a computer program embodied on a computer readable medium comprising computer executable program code, which code, when executed by at least one processor of an apparatus carried by a vehicle, causes the apparatus to:.

The embodiments in the foregoing are used merely to explain selected aspects or steps that may be utilized in implementations of the present invention as defined in the claims.

In the following description, like numbers denote like elements.

A military vehicle camera system needs to contend with harsh environmental conditions, such as gunfire shock, muzzle flash, vibration, high- and low- temperatures, solar load, water, sand, dust and mud. Different and fast changing light conditions while the vehicle is moving from dark shadow areas to bright sunlight requires an imaging sensor, which can quickly adapt and integrate. When firing the howitzer or machine gun, a strong muzzle flash can cause image blur when using common standard dynamic range CMOS or CCD imaging sensors. During fast climate change from cold to hot, fogging inside the camera housing could limit the camera view.

Climate and visibility conditions define further challenges, and weather conditions vary. There can be early morning rain or snow, desert sunlight with dust, sunset conditions, and extremely poor visibility.

No single sensor technology can cover all these operational conditions. Only a combination of two or more different image sensor technologies, such as Long Wavelength Infra-Red (LWIR), and Complementary Metal-Oxide-Semiconductor (optical sensor) CMOS or Charge Coupled Device (optical sensor) CCD sensors may provide an image under all such different conditions.

Horizontal field of view & detection recognition range define their own challenges. These parameters may actually be in contradiction. The larger the horizontal field of view for a single sensor camera system, the lower the Detection Recognition Identification (DRI) range may be. A single camera and lens optic with a wide horizontal field of view (HFOV) may have the disadvantage of optical distortion and a fisheye effect, which leads to poor depth perception. When combining two of the same cameras with lenses, and using an image stitching algorithm, the DRI range may be significantly better with a larger HFOV. A stitching algorithm, however, causes no dead zones on the vehicle, compared to different single camera installations.

Safety and reliability are key factors. The digital camera system needs to be available under any conditions, since freezing camera images could cause a crash or threaten the vehicle's operation. Thus, a modular software with different threats for each process is required. Each process needs to be monitored with a software watchdog. Additionally, a built-in hardware test is required in order to display system failure immediately.

System latency plays an important role on the system performance. The overall system latency should be as low as possible from glass to glass, as different studies have shown that a latency of more than <NUM> milliseconds causes the vehicle crew to suffer "sea sickness" while viewing the camera system monitor.

Human stress factor in combat situations are important too. During operations and combat situations the stress factor of the crew is high. Thus, the HMI of the vision system, including the software menu, needs to be simple and operating failure proof. Input keys needs to be illuminated, ergonomically arranged, and in compliance with relevant standards.

Resolution of the sensors and display panel have their own effects. As the imaging sensor resolution increases, the pixel size decreases, as does the low light sensitivity. For a low light sensitive camera system, a sensor pixel size of between <NUM> and <NUM> at <NUM> x <NUM> up to <NUM> x <NUM> sensor resolution is the combination of one example embodiment. The display panel resolution and display ratio should be the same as, or very similar to, the imaging sensor resolution and sensor ratio, otherwise the monitor display may become a limiting factor for the vision system.

<FIG> shows a schematic picture of a vehicle <NUM>, vehicle system <NUM> and a control apparatus <NUM> according to an example embodiment.

The vehicle system <NUM> comprises a control apparatus <NUM> configured to provide and operate a wide field of view model (WFOV) <NUM>. The control apparatus <NUM> is also called as a surveillance apparatus when combined with the cameras <NUM> and a display <NUM>.

When planning a route between endpoints or waypoints, for example, route plan information may be determined. The wide field of view model (WFOV) <NUM> is maintained and operated by the control apparatus <NUM> and may receive route plan information for a dedicated route. The route plan information may be generated by the control apparatus <NUM> or received by the control apparatus <NUM>. The route plan information is generated using information from navigation system <NUM> that is configured to provide route plan related information based on weather conditions, time schedule, safety aspects and fuel consumption (e.g. based on estimated fuel consumption and weather forecast), for example. As part of the planning steps an estimate of the resources available and possible constraints to the route plan are needed as well. Topographic and soil information associated to the dedicated route may be determined using the route plan information. Furthermore, energy consumption information associated to the dedicated route may be determined using the route plan information. The route plan information may be adjusted based on the wide field of view model (WFOV) <NUM>, the topographic and soil information and the operational characteristic information, automatically.

The control apparatus <NUM> may be configured to receive environmental information via communication link (COM) <NUM> or via vehicle sensors (SEN) <NUM>, for example. The environmental information comprises at least one of the following: weather information; obstacle information; topography information; and brightness information.

Gun system (GUN) <NUM> may comprise any guns, weapons, or ammunition relating to the firepower of the vehicle <NUM>. The gun system (GUN) <NUM> may provide operational characteristics for the control apparatus <NUM>, such as offensive information. The offensive information may comprise, for example, currently active gun and ammunition related information of the vehicle.

In an embodiment, the operational characteristics received by the control apparatus <NUM> may also comprise defensive information. The defensive information may comprise, for example, detected enemy threat related information of the vehicle. Such information may be determined based on information received via sensors <NUM> including radars, camera system <NUM> or via communication link <NUM>, for example.

A display system (DISP) <NUM> may be configured to provide overall status information of the vehicle <NUM>. The display <NUM> may be external to the control apparatus <NUM>, integrated as part of it, or both.

In an embodiment, an advanced vehicle system <NUM> of the vehicle <NUM> is configured to operate in degraded visual environments (DVE) and is modular and can be integrated within various versions of Main Battle Tank(s), Armoured Personnel Carrier (s) (APC) and other special purpose vehicles. A number of different advanced cameras <NUM>, configurations and packages are possible. Various embedded processing units <NUM> can be connected to operate in a cluster as a full <NUM>° system.

In an embodiment, by establishing a wide field of view model (WFOV) <NUM> for communicating between systems <NUM>-<NUM> it is possible for the on-board systems to negotiate an optimal solution for the route or some activity. Top priority for optimization may be defined to be safety, and second and third priority can be set by the vehicle operator (optimal offensive or defensive position, energy efficiency, fuel consumption, speed/time, etc.), for example. The wide field of view model (WFOV) <NUM> may operates as a virtual pilot for a route.

The wide field of view model (WFOV) <NUM> solution will allow different levels of automation within the vehicle <NUM>. In first operation mode, the wide field of view model (WFOV) <NUM> may be configured to provide a route plan, which the crew can use for scheduling their activities. In second operation mode, the wide field of view model (WFOV) <NUM> may be configured to provide an embedded solution, wherein the sub-systems can notify the crew based on the plan, when to perform certain tasks or be switched on or set to standby. This notification may be repeated on the main control display or remote-control station. In third operation mode, wide field of view model (WFOV) <NUM> may be configured to provide a solution to be fully automated and automatically executing the plan of the wide field of view model (WFOV) <NUM> with merely notification provided to the crew or remote-control station when performing different automated tasks.

No matter a plurality of different elements <NUM>-<NUM> is disclosed in <FIG>, not all are mandatory for embodiments of the invention. Only mandatory features are control apparatus <NUM> and camera system <NUM>.

<FIG> shows a schematic picture of a camera system <NUM> according to an example embodiment.

<FIG> shows a schematic picture of a vision system <NUM> according to an example embodiment. The vision system <NUM> is also understood as a surveillance apparatus <NUM>.

The digital vision system <NUM> comprises a <NUM>° camera housing <NUM>, image acquisition, and processing software (wide field of view model (WFOV) <NUM> of <FIG>) that supports multiple cameras <NUM>-<NUM> (e.g. four). The system <NUM> has <NUM> x identical camera pairs <NUM>-<NUM>, <NUM>-<NUM> and associated lenses. Additional cameras with camera link interfaces can be easily integrated without changing the software <NUM>.

In an embodiment, a <NUM>° rear view camera system <NUM> may be based on the same embedded processing unit <NUM>, and use the same image processing software <NUM>, as the day & night sight driver periscope system, for example.

In an embodiment, all components <NUM>, <NUM>-<NUM>, <NUM> are selected to fulfil shock and vibration requirements according to MIL-STD-<NUM> tracked vehicle vibrations. The electronics and sensors are housed in a modular body, which is protection class IP68K.

The camera housing <NUM> is all weather housing. The camera windows can be cleaned with water and air when sand & dust or mud are limiting the view. The operator can activate the cleaning via monitor keyboard if required. The <NUM>° camera housing <NUM> is designed to withstand a sand storm acc. MIL-STD-810F, Method <NUM>. The housing material and surface treatment comply with salt and fog requirements of MIL-STD-810F, Method <NUM>. The <NUM>° camera system provides high performance, low latency, with easy and intuitive operation.

In an embodiment, the housing <NUM> comprises a front surface <NUM>, wherein the plurality of camera devices <NUM>-<NUM> are arranged.

In an embodiment, a plurality of camera devices <NUM>-<NUM> are configured to provide image data on different channels, wherein each channel is associated with at least two camera devices with substantially same field-of-view (FOV) angle but different line-of-sight (LOS) angle.

The camera devices <NUM>-<NUM> are configured to operate on a first channel that is a longwave infrared (LWIR) channel. The camera devices <NUM>-<NUM> are configured to operate on a second channel that is a visible infrared (VIS) channel.

In an embodiment, the at least two camera devices <NUM>-<NUM> associated to the first channel are arranged vertically alongside each other. Correspondingly, the at least two camera devices <NUM>-<NUM> associated to the second channel are arranged vertically alongside each other.

Furthermore, the at least two camera devices <NUM>-<NUM> associated to the first channel are arranged horizontally alongside the at least two camera devices <NUM>-<NUM> associated to the second channel.

Various system features support the chief of section for clearing the vehicle surrounding during day and night. Software features like digital zoom, special image contrast enhancement algorithm which display image details under difficult environmental conditions.

In an embodiment, different operating modes are determined. In split screen mode LWIR and CMOS images are displayed simultaneously. In fusion mode, both sensor images LWIR and CMOS are fused into one full screen image.

A recording function may be integrated with several days of recording capability, embedded video date & time stamp. This function can be used for mission analysis and training purposes.

In an embodiment, the camera device <NUM>-<NUM> comprises at least one of the following: IR sensor, LWIR sensor, CMOS sensor, and CCD sensor. The at least two camera devices <NUM>-<NUM> associated to the first channel may comprise IR camera devices with resolution of <NUM> x <NUM>, for example. The at least two camera devices <NUM>-<NUM> associated to the second channel may comprise CMOS camera devices with resolution of <NUM> x <NUM> for example.

In an embodiment, the housing <NUM> is rotatably and/or tiltably arranged to a vehicle. The rotation and/or tilting of the housing <NUM> may be controlled by the control apparatus or its operator.

In an embodiment, a field-of-view (FOV) angle of each camera <NUM>-<NUM> is <NUM>°. Thus, image data for the first channel extends over the field-of-view (FOV) angle of <NUM>° when the camera devices <NUM>-<NUM> are arranged as in <FIG> where the angular difference of line of sight (LOS) between camera devices <NUM>-<NUM> is <NUM>°. Correspondingly, image data for the second channel extends over the field-of-view (FOV) angle of <NUM>° when the camera devices <NUM>-<NUM> are arranged as in <FIG> where the angular difference of line of sight (LOS) between camera devices <NUM>-<NUM> is <NUM>°. By doing that, it is possible to provide field-of-view (FOV) angle of the output data to be <NUM>°.

In an embodiment, the wide field of view model (WFOV) <NUM> of the control apparatus <NUM> operates as follows. The control apparatus <NUM> receives first image data from a first camera device <NUM> and second image data from a second camera device <NUM> of a first channel (LWIR). The control apparatus <NUM> also receives third image data from a third camera device <NUM> and fourth image data from a fourth camera device <NUM> of a second channel (VIS). Then, the wide field of view model <NUM> is generated based on the first, second, third and fourth image data. The wide field of view model <NUM> is configured to utilize a stitching mode, wherein image data is stitched per channel, by combination of dynamic and static stitching, a first operating mode, wherein image data of one channel is configured to be processed, a second operating mode, wherein image data of both channels are configured to be combined, and an output mode, wherein the output mode defines configuration for image data to be provided on the display. Once selection information for the modes is received, output data is generated for the display <NUM> based on the selection information and the wide field of view model <NUM>. Selection information may be received from operator or determined automatically by the model <NUM>.

In an embodiment, the images from the LWIR cameras <NUM>-<NUM> are stitched in the software model <NUM>, which results in an image covering a <NUM>° horizontal field of view and a high total resolution of <NUM> x <NUM> pixels for the LWIR image, and <NUM> x <NUM> pixels for the VIS (visible) CMOS image.

The use of two camera pairs <NUM>-<NUM>, <NUM>-<NUM> with <NUM>° HFOV optical lens results in a higher Detection Recognition Identification (DRI) range than with a single camera having a wide HFOV lens and enables a <NUM>° horizontal field of view. A single camera and an optical lens with a wide HFOV has the disadvantages of optical distortion and the fisheye effect among others.

The vision system <NUM> overcomes this by stitching two camera images to provide the highest DRI range in its class.

The stitched LWIR and VIS-CMOS images can be displayed in split screen mode simultaneously in first mode, full screen mode LWIR, or CMOS in second mode, and image fusion mode in third mode.

The software of the wide field of view model (WFOV) <NUM> may be divided into several modules. Each module is responsible for a defined set of functionalities. Multiple processes are used to group and execute a subset of modules, while each process can execute several threads.

Communication between the processes is established by message based IPC protocols and shared memory <NUM> (see <FIG>). Communication between the modules <NUM>-<NUM>, <NUM>-<NUM>, <NUM> inside the same process is implemented using shared memory <NUM>.

The reason for grouping modules <NUM>-<NUM>, <NUM>-<NUM>, <NUM> into a single process is to decrease the latency of communication between the modules. Using a message based IPC protocol for communication between the processes allows separating functionality, which keeps the parts simple and avoids complexity.

Transmission of image data between the camera control processes and the main process is carried out using shared memory to reduce the overall latency.

For logging and watchdog functionality, systemd may be used. Each process has to regularly notify the system process, otherwise it will be restarted. All process actions are logged into the system's database.

A wide dynamic range CMOS camera sensor may be integrated within camera devices <NUM>-<NUM>, for example, for the <NUM>° camera system. The camera sensor signal response is logarithmic through more than <NUM> decades (><NUM> dB). The logarithmic sensor provides an almost infinite dynamic range, which results in no saturation, a see through halo, and no loss of details.

When driving on bumpy roads the horizon line changes. With a traditional camera sensor, when shifting from a dominant bright sky to a dominant dark background, linear sensors either lose information at the ground or saturate in the sky. With a logarithmic sensor and the wide field of view model (WFOV) <NUM>, all information is always captured.

In order to increase the low light sensitivity to night level <NUM>, an image intensifier tube can be combined with the wide dynamic range CMOS camera sensor. This WDR CMOS image intensified camera may be integrated also into the driver sight periscope and may be supported by the developed image processing software.

<FIG> presents an example block diagram of a control apparatus <NUM> in which various embodiments of the invention may be applied. The control apparatus <NUM> is configured to maintain and/or operate the wide field of view model (WFOV).

The general structure of the control apparatus <NUM> comprises a user interface <NUM>, a communication interface <NUM>, a processor <NUM>, and a memory <NUM> coupled to the processor <NUM>. The control apparatus <NUM> further comprises software <NUM> stored in the memory <NUM> and operable to be loaded into and executed in the processor <NUM>. The software <NUM> may comprise one or more software modules and can be in the form of a computer program product, such as the wide field of view model (WFOV) <NUM> of <FIG>. The control apparatus <NUM> may further comprise a user interface controller <NUM>. The user interface <NUM> may comprise e.g. the display <NUM> of <FIG>, <FIG>.

The processor <NUM> may be, e.g., a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a graphics processing unit, or the like. <FIG> shows one processor <NUM>, but the apparatus <NUM> may comprise a plurality of processors.

The memory <NUM> may be for example a non-volatile or a volatile memory, such as a read-only memory (ROM), a programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), a random-access memory (RAM), a flash memory, a data disk, an optical storage, a magnetic storage, a smart card, or the like. The apparatus <NUM> may comprise a plurality of memories. The memory <NUM> may be constructed as a part of the apparatus <NUM> or it may be inserted into a slot, port, or the like of the apparatus <NUM> by a user. The memory <NUM> may serve the sole purpose of storing data, or it may be constructed as a part of an apparatus serving other purposes, such as processing data. A proprietary application, such as computer program code for the wide field of view model (WFOV), route related data, vehicle related data, gun data, sensor data or environmental data may be stored to the memory <NUM>.

In an embodiment, the apparatus <NUM> is configured to perform a computer-implemented method for generating surveillance image data, wherein a plurality of camera devices are configured to provide image data on different channels, and wherein each channel is associated with at least two camera devices with substantially same field-of-view (FOV) angle but different line-of-sight (LOS) angle, the method comprising: receiving first image data and second image data of a first channel, wherein the first channel is a longwave infrared (LWIR) channel; receiving third image data and fourth image data of a second channel, wherein the second channel is a visible spectrum (VIS) channel; generating a wide field of view model based on the first, second, third and fourth image data; determining a stitching mode, wherein image data is stitched per channel, by combination of dynamic and static stictching, determining a first operating mode, wherein image data of one channel is configured to be processed; determining a second operating mode, wherein image data of both channels are configured to be combined; determining output mode, wherein the output mode defines configuration for image data to be provided on the display; receiving selection information for the operating mode, the stitching mode and the output mode; and generating output data for a display based on the selection information and the wide field of view model.

The user interface controller <NUM> or the user interface <NUM> may comprise circuitry for receiving input from a user of the control apparatus <NUM> (an operator), e.g., via a keyboard, graphical user interface shown on the display of the user interfaces <NUM> of the control apparatus <NUM>, speech recognition circuitry, or an accessory device, such as a headset, and for providing output to the user via, e.g., a graphical user interface or a loudspeaker.

The communication interface module <NUM> implements at least part of data transmission. The communication interface module <NUM> may comprise, e.g., a wireless or a wired interface module. The wireless interface may comprise such as a WLAN, Bluetooth, infrared (IR), radio frequency identification (RF ID), GSM/GPRS, CDMA, WCDMA, LTE (Long Term Evolution) or <NUM> radio module. The wired interface may comprise such as RS-<NUM>, SPI, I2C, Ethernet, universal serial bus (USB) or any other standard for example. The communication interface module <NUM> may be integrated into the control apparatus <NUM>, or into an adapter, card or the like that may be inserted into a suitable slot or port of the control apparatus <NUM>. The communication interface module <NUM> may support one radio interface technology or a plurality of technologies. The control apparatus <NUM> may comprise a plurality of communication interface modules <NUM>.

In an embodiment, the control apparatus <NUM> is configured to connect with the camera devices via the communication interface <NUM>.

A skilled person appreciates that in addition to the elements shown in <FIG>, the control apparatus <NUM> may comprise other elements, such as microphones, extra displays, as well as additional circuitry such as input/output (I/O) circuitry, memory chips, application-specific integrated circuits (ASIC), processing circuitry for specific purposes such as source coding/decoding circuitry, channel coding/decoding circuitry, ciphering/deciphering circuitry, and the like. Additionally, the control apparatus <NUM> may comprise a disposable or rechargeable battery (not shown) for powering when external power if external power supply is not available.

In an embodiment, the control apparatus <NUM> comprises speech recognition means. Using these means, a pre-defined phrase may be recognized from the speech and translated into control information for the apparatus <NUM>, for example.

External devices or sub-systems (e.g. elements <NUM>-<NUM> of <FIG>) may be connected to the control apparatus <NUM> using communication interface <NUM> of the apparatus <NUM> or using a direct connection to the internal bus of the apparatus <NUM>.

<FIG> shows a schematic picture of a wide field of view model (WFOV) <NUM> and related information flows according to an example embodiment.

Elements <NUM>-<NUM> may have alternative ways to connect with each other and <FIG> only shows one example embodiment. Furthermore, only connections that relate somehow to the WFOV <NUM> are illustrated. For example, environmental information <NUM> may be also used for route planning and thus for the route plan information <NUM> but direct connection between blocks <NUM> and <NUM> is not shown for simplifying the <FIG>.

The WFOV <NUM> can be configured to operate as a stand-alone solution or as an integrated part of the control/management system of the vehicle. The WFOV <NUM> may enable automation of vision system <NUM> control, and further enables a higher degree of autonomous operation on board conventional vehicles and paves the way for vision and route management for autonomous vehicles.

Vision system information <NUM> provided to the direction of the wide field of view model (WFOV) <NUM> comprises at least image data of different channels and camera devices.

In an embodiment, the wide field of view model (WFOV) <NUM> may be interfaced with the camera system, navigation system, automation system, power management system and sub-systems like gun solutions, as shown in <FIG>, for example.

The wide field of view model (WFOV) <NUM> may be arranged to receive route plan information <NUM> including information like weather forecasts, navigation information for the dedicated route, waypoint information for the dedicated route, environmental restrictions and other relevant information. The route plan information <NUM> may be received from the navigation system or the route plan information <NUM> may be generated by the control apparatus <NUM>. The route plan information <NUM> may comprise at least one of the following: navigation information; and environmental information. The navigation information may comprise at least one of the following: destination information of the dedicated route; remaining travel time of the dedicated route; remaining distance of the dedicated route; navigation information for the dedicated route; waypoint information for the dedicated route; and environmental restriction information of the dedicated route.

Topography information <NUM> associated to the dedicated route may be determined using the route plan information <NUM>. The topography information <NUM> may comprise information on topographical details of the route or nearby area, as well as details of soil, for example. Furthermore, it may comprise information on possible obstacles or other objects around the vehicle that may affect either to movements, cover or visibility of the vision system <NUM> including cameras.

In an embodiment, the wide field of view model (WFOV) <NUM> may be configured to automate interaction between navigational route planning and setting up different vision system <NUM> modes or camera parameters.

In an embodiment, the control apparatus <NUM> may be configured to determine a control task relating to the route plan information <NUM> automatically based on the wide field of view model (WFOV) <NUM>. Thus, the route plan information <NUM> that is determined for a dedicated route, may be dynamically adjusted automatically using the wide field of view model (WFOV) <NUM>.

In an embodiment, the control apparatus <NUM> may be configured to dynamically adjust navigation information of the route plan information. Furthermore, the control apparatus <NUM> may be configured to dynamically adjust navigation information for the dedicated route, and, for example, dynamically adjusting waypoint information for the dedicated route. For example, the route with optimal visibility and/or cover may be determined.

In an embodiment, the control apparatus <NUM> may be configured to dynamically adjust destination information or remaining travel time of the dedicated route.

The wide field of view model (WFOV) <NUM> is further arranged to receive characteristic information <NUM> representing at least one operating characteristic of the vehicle. The operating characteristic information <NUM> of the vehicle may comprise at least one of the following: currently active gun system and ammunition; status of search light system; and status information of energy storage sub-system, such as a battery system.

The wide field of view model (WFOV) <NUM> may further be arranged to receive environmental information <NUM> separate or in addition to possible environmental information included in the route plan information <NUM>. The environmental information <NUM> may represent at least one current environmental characteristic of the vehicle, such as soil characteristic information, weather information; wind information; air pressure information; ice information; and roll or pitch information of the vehicle.

In an embodiment, if there has not been identified any violations of possible constraints, the wide field of view model (WFOV) <NUM> may generate at least one task for controlling vision system <NUM> (or <NUM> in <FIG>) with or without some associated search light within the vehicle automatically based on the wide field of view model (WFOV) <NUM> and control the associated element based on the determined task.

For example, the route planning system may carry out following procedures: A) Calculate and balance to what degree a route deviation will benefit the overall vision system. B) Generate an operational plan for when to change between different vision system operation modes during the planned route. C) If the preferred target time is known, calculate the optimal speed profile including staying longer in optimal areas for vision system and avoiding waiting time in low quality vision areas. Additionally, the system may collect real operating data, compare it with the original prediction/recommendation, and automatically improve the recommendation for later routes operated by the wide field of view model (WFOV) <NUM>.

In an embodiment, if there has not been identified any violations of possible constraints, the wide field of view model (WFOV) <NUM> may generate vision plan (VP) <NUM> and utilize the vision plan (VP) <NUM> for determining control tasks relating to vision system <NUM> or route plan <NUM> within the vehicle automatically based on the wide field of view model (WFOV) <NUM>. The vision plan may comprise, for example, settings when different operating modes (split, single, fusion) is activated, rotation/tilting angle for the camera housing, activation of search light or laser light, cleaning camera lenses, and detailed camera device settings or image data processing parameters.

While cruising and performing transit during the route, the wide field of view model (WFOV) <NUM> may maintain a dynamic and up-to-date situational awareness in relation to the executed route (navigation) and vision plan and the status from all camera devices and sensors, for example. If the situation changes and a system changes health status, the wide field of view model (WFOV) <NUM> may be configured to update the vision plan <NUM> including tasks and automatically notifying the navigation system to allow the navigation system to modify the route plan information accordingly.

Because the wide field of view model (WFOV) <NUM> has access to information about optimal operation conditions of the sub-systems, the model can help to determine optimal operating mode, wherein only confirmed request from the operator is needed, and the wide field of view model (WFOV) <NUM> may allow running sub-systems outside the optimal operation conditions.

The vision plan <NUM> information can be provided in a first mode as a schedule made available to the crew to follow. The crew may perform the scheduled tasks for the vision system <NUM> based on the vision plan <NUM>. In a second mode, the vision plan <NUM> may be embedded in the main display of the vehicle, for example. The system may be further configured to provide an integrated guidance tool to prompt the operator when a task should take place and by acknowledgement from the operator enable and perform the task and end the task when performed. A third mode allows a fully automated solution, where the operator may only be informed about the vision plan <NUM> or the tasks determined by the wide field of view model (WFOV) <NUM>. Optionally, current status of the model and next steps may be informed to the operator but the wide field of view model (WFOV) <NUM> is configured to control elements automatically. In such embodiment the vision plan <NUM> may be optional.

It is possible to override the wide field of view model (WFOV) <NUM> by changing it to standby mode and allowing a manual operation of the visual system <NUM> and the sub-systems. At the third mode, the wide field of view model (WFOV) <NUM> can operate autonomously together with the vision system <NUM> and/or navigation system and all the sub-systems. Instead of notifying the operator, the wide field of view model (WFOV) <NUM> may log (e.g. using the vision plan <NUM>) the activities and events and will only request assistance from the mission controller or a human operator in case the wide field of view model (WFOV) <NUM> is facing a situation it cannot handle or it is not available for operation.

In an embodiment, the energy vision plan <NUM> may also comprise automatic information being sent to mission control room. The information being sent may relate to, for example, estimate of enemy equipment or persons detected, or supplies needed when at next service, for example. By doing that the mission control can make a better estimate of the overall situation. The mission control system may have a dynamic wide field of view model (WFOV) <NUM> of its own that receives inputs from multiple vehicles.

In an embodiment, the wide field of view model (WFOV) <NUM> is configured to receive input from an operator (USR) <NUM> either on-board the vehicle or remote at other vehicle, for example. In certain pre-defined operating modes or tasks, it may be required that operator acknowledgement is received from the operator (USR) <NUM> for the determined task by the wide field of view model (WFOV) <NUM> before controlling an automation element of the vehicle based on the determined task in response to the received operator acknowledgement.

In an embodiment, the wide field of view model (WFOV) <NUM> may be updated in real-time using the route plan information <NUM>, the environmental information <NUM> and the operational characteristics <NUM>. In an embodiment, when receiving confirmation from the operator <NUM> of the task being performed, the wide field of view model (WFOV) <NUM> is updated in response to the received confirmation.

In an embodiment, in autonomous vehicle operation mode, automatic route planning may be executed to provide the route plan information <NUM> for a safe and optimized route taking into account planned destination and ETA, up to date chart data from the electronic chart library, predicted environmental conditions as well as status information of different sub-systems. Furthermore, a contingency plan to stop the vehicle safely in case of emergency is generated along the route for every leg or even leg segment, for example. The approval mechanisms of the route plan <NUM> may vary depending on autonomy level in use, authority rule sets and customer specifications. Once the route plan is activated and being executed, the control system is permanently monitoring and adapting the route execution with regards to track- and schedule keeping) if necessary. Reasons for adaptation can be, for example: new destination and/or new ETA, differences between predicted and real environmental conditions, collision avoidance maneuvers, and unexpected changes in the vehicle (i.e. unforeseen equipment failure).

In an embodiment, the route plan information <NUM> comprises at least one of the following: navigation information for a waypoint or a destination; target time or arrival information for the waypoint or the destination; and environmental information associated to at least one route of the route plan information.

<FIG> shows a schematic picture of a system <NUM> according to an example embodiment. A vehicle <NUM> comprises a control apparatus <NUM> for controlling vision system.

The control apparatus <NUM> is capable of downloading and locally executing software program code. The software program code may be a client application of a service whose possible server application is running on a server apparatus <NUM>, <NUM> of the system <NUM>. The control apparatus <NUM> may comprise a capturing device, such a sensor device, for providing vehicle related signals and data. The sensor device may comprise an accelerometer, an inclinometer, a gyroscope, a wind sensor, a positioning sensor, a temperature sensor, a pressure sensor, or a camera, for example. The camera may also be used to provide video data and a microphone may be used for providing audio data, for example. The sensor device may also provide environmental signals and data.

The control apparatus <NUM> is configured to generate surveillance image data, comprising: a plurality of camera devices configured to provide image data on different channels, wherein each channel is associated with at least two camera devices with substantially same field-of-view (FOV) angle but different line-of-sight (LOS) angle, and the apparatus <NUM> is configured to: receive first image data and second image data of a first channel; receive third image data and fourth image data of a second channel; generate a wide field of view model based on the first, second, third and fourth image data; determine a stitching mode, wherein image data of at least one channel is configured to be combined using a stitching algorithm; determine a first operating mode, wherein image data of one channel is configured to be processed; determine a second operating mode, wherein image data of both channels are configured to be combined; determine output mode, wherein the output mode defines configuration for image data to be provided on the display; receive selection information for the operating mode, the stitching mode and the output mode; and generate output data for the display based on the selection information and the wide field of view model.

Environmental status changes around the vehicle on different areas when moving around. There may be obstacles, different lighting conditions, shades, raining, snowing, fog, dust etc. that may affect visibility around the vehicle.

In an embodiment, between waypoints, the vehicle <NUM> may choose different routes over the area and depending on the route there may be a plurality of areas or regions <NUM>-<NUM> with different conditions. The vehicle <NUM> route may be optimized using the wide field of view model (WFOV) <NUM> so that the vehicle <NUM> stays longer periods on areas <NUM>-<NUM> more optimal for visibility or cover, for example. Also the operation mode of the vision system may be automatically determined based on different areas <NUM>-<NUM> under control of the wide field of view model (WFOV) <NUM>.

The control apparatus <NUM> may be configured to be connectable to a public network <NUM>, such as Internet, directly via local connection or via a wireless communication network <NUM> over a wireless connection <NUM>. The wireless connection <NUM> may comprise a mobile cellular network, a satellite network or a wireless local area network (WLAN), for example. The wireless communication network <NUM> may be connected to a public data communication network <NUM>, for example the Internet, over a data connection <NUM>. The apparatus <NUM> may be configured to be connectable to the public data communication network <NUM>, for example the Internet, directly over a data connection that may comprise a fixed or wireless mobile broadband access. The wireless communication network <NUM> may be connected to a server apparatus <NUM> of the system <NUM>, over a data connection.

The control apparatus <NUM> may set up local connections within the vehicle system <NUM> with at least one capturing device and at least one automation device. The capturing device, such as a sensor, may be integrated to the apparatus <NUM>, attached to the body of the vehicle and connected to the vehicle control system or arranged as separate sensor device and connectable to the network <NUM> over separate connection.

The apparatus <NUM> and its client application may be configured to log into a vehicle data service run on a server <NUM>, for example. The server apparatus <NUM>, <NUM> may be used to maintain any data, such as image data, route plan information, environmental data, or task related information, for example.

In particular, real-time interaction may be provided between the apparatus <NUM> and the server <NUM> to collaborate for dynamic vision system related data over a network <NUM>. Real-time interaction may also be provided between the apparatus <NUM> and the remote user device <NUM> to collaborate for any wide field of view model (WFOV) <NUM> related data over a network <NUM>, <NUM>.

The apparatus <NUM> may be connected to a plurality of different capturing devices and instruments and the apparatus <NUM> may be configured to select which sensor devices is actively collaborated with.

A user/operator of the apparatus <NUM> or the remote user device <NUM> may need to be logged in with user credentials to a chosen service of the network server <NUM>.

The system <NUM> may comprise a server apparatus <NUM>, which comprises a storage device <NUM> for storing service data, service metrics and subscriber information, over data connection <NUM>. The service data may comprise wide field of view model (WFOV) <NUM> related data, route related data, environmental data, image data, waypoint properties related data, vehicle related data, navigation information, configuration data, task information for the automation system, sensor data, user input data, real-time collaboration data, predefined settings, and attribute data, for example.

In particular, configuration information or application download information for any apparatus may be automatically downloaded and configured by the server <NUM>. Thus, the user of the devices may not need to do any initialization or configuration for the service. The system server <NUM> may also take care of account creation process for the service, such sensor devices, apparatuses and users. Timing of the download may also be configured to be automatic and optimized in view of the vehicle travel plan. For example, download may be automatically taking place when the vehicle is under service in base.

The association of the devices can be one-time or stored persistently on any of the devices or the server <NUM>.

In particular, authentication of a sensor device or apparatus <NUM> on a system server <NUM> may utilize hardware or SIM credentials, such as International Mobile Equipment Identity (IMEI) or International Mobile Subscriber Identity (IMSI). The sensor device or apparatus <NUM> may transmit authentication information comprising IMEI and/or IMSI, for example, to the system server <NUM>. The system server <NUM> authenticates the device by comparing the received authentication information to authentication information of registered users / devices / vehicles / apparatuses stored at the system server database <NUM>, for example. Such authentication information may be used for pairing the devices and/or apparatuses to generate association between them for a vehicle data connection.

A service web application may be used for configuration of a system. The service web application may be run on any user device, admin device, or a remote-control device <NUM>, such as a personal computer connected to a public data network, such as Internet <NUM>, for example. The control apparatus <NUM> may also be connected locally to the apparatus <NUM> over a local connection <NUM> and may utilize the network connections of the apparatus <NUM> for configuration purposes. The service web application of the control apparatus may provide searching/adding instruments, determining attributes, device setup and configuration, for example. The service web application of the control apparatus <NUM> may be a general configuration tool for tasks being too complex to be performed on the user interface of the apparatus <NUM>, for example.

A remote-control apparatus <NUM> may be authenticated and configuration data sent from the control apparatus <NUM> to the system server <NUM>, <NUM>, wherein configuration settings may be modified based on the received data. In an embodiment, the modified settings may then be sent to the apparatus <NUM> over the network <NUM> and the local connection or the wireless operator. The modified settings may also be sent to external devices correspondingly, through the apparatus <NUM> or directly over the network <NUM>, for example.

The sensor device may be wireless or wired.

The system <NUM> may also comprise a plurality of satellites <NUM> in orbit about the Earth. The orbit of each satellite <NUM> is not necessarily synchronous with the orbits of other satellites and, in fact, is likely asynchronous. A global positioning system receiver apparatus such as the ones described in connection with preferred embodiments of the present invention is shown receiving spread spectrum Global Navigation Satellite System global positioning system (GNSS) satellite signals <NUM> from the various satellites <NUM>.

The remote-control apparatus <NUM> may be configured to be operated by a remote operator of the vehicle system <NUM>. The remote-control apparatus <NUM> may be arranged on a remote station, on the vehicle or on another vehicle, for example.

The input for an automatic route plan may come from a Remote Control Centre (RCC), the Remote Operation Centre (ROC) or the Fleet Operation Centre (FOC), depending on the level of autonomy. A mission manager process may receive the order and provide it to the route planning and execution process of the apparatus <NUM>. The mission order contains at least destination and planned arrival time. Additional parameters i.e. driven by cargo (avoiding of areas with predicted danger state above a certain level) can be part of it. Based on input from <NUM>) and <NUM>) above and defined safety precautions / margins (i.e. safety corridor) an automatic routing algorithm will find in the first instance a geometrically optimal route from A to B. Geometric adaptations as well as the generation of the schedule by means of considering information's from <NUM>), <NUM>) and <NUM>) will be performed by an optimization engine afterwards.

<FIG> presents an example block diagram of a server <NUM>.

The general structure of the server apparatus <NUM> comprises a processor <NUM>, and a memory <NUM> coupled to the processor <NUM>. The server apparatus <NUM> further comprises software <NUM> stored in the memory <NUM> and operable to be loaded into and executed in the processor <NUM>. The software <NUM> may comprise one or more software modules and can be in the form of a computer program product.

The processor <NUM> may be, e.g., a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a graphics processing unit, or the like. <FIG> shows one processor <NUM>, but the server apparatus <NUM> may comprise a plurality of processors.

The memory <NUM> may be for example a non-volatile or a volatile memory, such as a read-only memory (ROM), a programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), a random-access memory (RAM), a flash memory, a data disk, an optical storage, a magnetic storage, a smart card, or the like. The server apparatus <NUM> may comprise a plurality of memories. The memory <NUM> may be constructed as a part of the server apparatus <NUM> or it may be inserted into a slot, port, or the like of the server apparatus <NUM> by a user. The memory <NUM> may serve the sole purpose of storing data, or it may be constructed as a part of an apparatus serving other purposes, such as processing data.

The communication interface module <NUM> implements at least part of radio transmission. The communication interface module <NUM> may comprise, e.g., a wireless or a wired interface module. The communication interface module <NUM> may be integrated into the server apparatus <NUM>, or into an adapter, card or the like that may be inserted into a suitable slot or port of the server apparatus <NUM>. The communication interface module <NUM> may support one radio interface technology or a plurality of technologies.

The e-mail server process <NUM>, which receives e-mail messages sent from control apparatuses <NUM> and computer apparatuses <NUM> via the network <NUM>. The server <NUM> may comprise a content analyzer module <NUM>, which checks if the content of the received message meets the criteria that are set for new activity data item of the service. The content analyzer module <NUM> may for example check whether the e-mail message contains a valid activity data item to be used for wide field of view model (WFOV) <NUM> processing, for example. The valid data item received by the e-mail server is then sent to an application server <NUM>, which provides application services e.g. relating to the user accounts stored in a user database <NUM> and content of the content management service. Content provided by the service system <NUM> is stored in a content database <NUM>.

A skilled person appreciates that in addition to the elements shown in <FIG>, the server apparatus <NUM> may comprise other elements, such as microphones, displays, as well as additional circuitry such as input/output (I/O) circuitry, memory chips, application-specific integrated circuits (ASIC), processing circuitry for specific purposes such as source coding/decoding circuitry, channel coding/decoding circuitry, ciphering/deciphering circuitry, and the like.

<FIG> presents an example block diagram of a computer apparatus <NUM>. The computer apparatus <NUM> may be a user equipment (UE), user device or apparatus, such as a mobile terminal, a smart phone, a laptop computer, a desktop computer or other communication device.

The general structure of the computer apparatus <NUM> comprises corresponding elements as earlier discussed for the apparatus <NUM>.

<FIG> shows a flow diagram showing operations in accordance with an example embodiment of the invention.

In step <NUM>, the computer-implemented method for generating surveillance image data by at least one processor of a surveillance apparatus carried by a vehicle, wherein a plurality of camera devices carried by the vehicle are configured to provide image data on different channels, and wherein each channel is associated with at least two camera devices with substantially same field-of-view (FOV) angles but different line-of-sight (LOS) angles, is started.

In step <NUM>, first image data and second image data of a first channel, wherein the first channel is a longwave infrared channel, are received. In step <NUM>, third image data and fourth image data of a second channel, wherein the second channel is a visible spectrum infrared channel are received.

In step <NUM>, a wide field of view model is generated based on the first, second, third and fourth image data. In step <NUM>, a stitching mode is determined, wherein image data of each channel is configured to be combined using a stitching algorithm.

In an embodiment, in the stitching mode, image data is stitched per channel, and realized as combination of dynamic and static stitching. The static stitching components include at least some of the following: vertical adaption information; perspective correction information; and cutting information.

In an embodiment, dynamic stitching components include at least some of the following: contrast adaptation information; color adaptation information; and blending of image transition information.

In an embodiment, in the stitching mode, image stitching of various number of channels is possible at the same time.

In an embodiment, the method may further comprise determining a backup operating mode, wherein a multiplexer function is activated to provide a video signal bypass image data for the display without processing the image data with the wide field of view model.

In step <NUM>, a first operating mode is determined, wherein image data of one channel is configured to be processed. In step <NUM>, a second operating mode is determined, wherein image data of both channels are configured to be combined.

In step <NUM>, output mode is determined, wherein the output mode defines configuration for image data to be provided on the display.

In step <NUM>, selection information is received for the operating mode, the stitching mode and the output mode. In step <NUM>, output data is generated for the display based on the selection information and the wide field of view model. The method is ended in step <NUM>.

In an embodiment, different operating modes are provided. In the stitching mode, image data of different channels are configured to be combined using a stitching algorithm as disclosed in relation to <FIG> and associated description. In the first operating mode, image data of one channel is configured to be processed. That may be used, for example, when processing only one of the camera devices (e.g. VIS1 or VIS2) of certain channel or processing different image streams for different camera device. In the second operating mode, image data of both channels (VIS and LWIR) are configured to be combined.

In an embodiment, operating modes of the first, the second and the stitching mode may be configured to be used for different output modes that comprise at least one of the following modes: Full screen visual mode (FS-VIS), full screen thermal mode (FS-LWIR), split screen with visual image on top (SS-VIS), split screen with thermal image on top (SS-LWIR), Picture-in-Picture mode with thermal image as main image (PIP-VIS) and Picture-in-Picture mode with visual image as main image (PIP-LWIR).

In an embodiment, in full screen visual output mode (FS-VIS), CMOS or CCD camera image data is displayed in full screen resolution. In full screen thermal mode (FS-LWIR), LWIR camera image data is displayed at <NUM> x <NUM> (depending on the monitor) resolution. In split screen mode (Display: <NUM> x <NUM>) with visual image on top (SS-VIS), the LWIR image data is displayed on top position at <NUM> x <NUM> resolution, and the VIS image data is displayed below at <NUM> x <NUM> resolution. In split screen mode with thermal image on top (SS-LWIR), the vertical positions of the image data are changed respectively.

A zoom mode is provided as option, both in full screen mode and in in split screen mode. Zoom mode is configured to be enabled also individually for each image, VIS and LWIR. A PAN mode is also enabled after zooming the image.

In an embodiment, an additional combined visual and thermal image mode (FUSION) may be used, for example, to replace the Picture-in-Picture modes. In FUSION mode, image data of two channels are combined to fused image data, wherein stitching mode is used for providing the image data of different channels for fusion.

<FIG> shows a flow diagram showing operations related to stitching in accordance with an example embodiment of the invention.

In the stitching mode, image data is stitched per channel, and realized as combination of dynamic and static stitching using the stitching algorithm.

In an embodiment, stitching process and algorithm is a combination of dynamic and static components and may comprise following phases carried at least partially by the wide field of view model (WFOV) <NUM>, for example.

In an embodiment, when in the stitching mode, image data of each channel <NUM>-<NUM> (VIS channel) and <NUM>-<NUM> (LWIR channel) are configured to be combined using a stitching algorithm S1, S2 of the wide field of view model (WFOV) <NUM>. Not all elements and functionalities of the wide field of view model (WFOV) <NUM> or the stitching algorithms S1, S2 are shown but only some main elements to illustrate the stitching mode.

In visual (VIS) channel, first image data <NUM> and second image data <NUM> of the VIS channel are received from respective camera devices, for example. The image data <NUM>-<NUM> are provided in stitching mode to the wide field of view model (WFOV) <NUM> for processing by the stitching algorithm S1.

In composing step <NUM>, first and second image data <NUM>, <NUM> are composed, wherein vertical image offset between the first and second image data <NUM>, <NUM> is corrected to provide composed image data. The composed image data comprises composed first and second image data with corrected vertical offset so that when arranging the composed first and second image data horizontally aligned, the combined image has no vertical offset.

In cropping step <NUM>, the composed image data is cropped and scaled to correspond with the monitor/display <NUM> resolution to provide scaled image data. The scaled image data comprises scaled first and second image data with adjusted resolution based on resolution input from the monitor <NUM> or set by the user so that when arranging the scaled first and second image data horizontally aligned, the combined image matches the monitor resolution.

In enhancing step <NUM>, the scaled image data is processed to enhance contrast of the image data to provide enhanced image data. The enhanced image data comprises enhanced first and second image data with enhanced contrast.

In rotating step <NUM>, the enhanced image data is corrected for perspective to provide rotated image data. A rotation matrix may be used for correction, for example. The rotated image data comprises rotated first and second image data with adjusted rotation using rotation function so that when arranging the rotated first and second image data horizontally aligned, linear objects in the combined image are linear over the borderline of the rotated first and second image data. Rotation function may be applied for each image of the image data.

In conversion step <NUM>, the rotated image data is conversed using Trapez conversion into rectangular image data. The rotated image data comprising the first and second rotated image may not be rectangular in shape and that is corrected for both first and second image to provide resulting rectangular image data. The rectangular image data comprises rectangular first and second image data so that when arranging the rectangular first and second image data horizontally aligned, the combined image is also rectangular and fitting to the monitor resolution.

In adaptation step <NUM>, the rectangular image data is composed to provide adapted image data comprising a single image, wherein selected part of the rectangular image data in the transition area (in the borderline area where first and second image data from different cameras are combined) are extracted and average value of the image brightness is calculated for that area. Image gain adaptation may also be applied for harmonizing the adapted image data. The adapted image data comprises single image data and do not comprise anymore separate first and second data. The adapted image is provided to be displayed by the monitor or combined with other channels, for example.

In mode selection step <NUM>, the adapted image data may be combined with other channel(s) (LWIR channel) before providing data to the monitor <NUM>, or providing the adapted data without combination to the monitor <NUM>.

In an embodiment, instead or in addition to providing output data to display/monitor <NUM>, the output data may be used as input for controlling a sub-system of the vehicle <NUM> as shown in <FIG>, <FIG>, for example. Automatic or autonomous operations may thus be enabled.

In infrared (LWIR) channel, third image data <NUM> and fourth image data <NUM> of the LWIR channel are received from respective camera devices, for example. The third and fourth image data <NUM>-<NUM> are provided in stitching mode to the wide field of view model (WFOV) <NUM> for processing by the stitching algorithm S2. The stitching algorithms S1 and S2 may be a single algorithm or two separate algorithms.

In an embodiment, steps <NUM>-<NUM> may correspond to the steps <NUM>-<NUM> as disclosed above.

<FIG> shows a schematic picture of a wide field of view model (WFOV) <NUM> and related data according to an example embodiment.

As disclosed in <FIG>, in visual (VIS) channel, first image data <NUM> and second image data <NUM> of the VIS channel are received from respective camera devices, for example. Correspondingly, in infrared (LWIR) channel, third image data <NUM> and fourth image data <NUM> of the LWIR channel are received from respective camera devices, for example.

In an embodiment, the wide field of view model (WFOV) <NUM> is configured to receive input from an operator (USR) <NUM> either on-board the vehicle or remote at other vehicle or remote control apparatus, for example. In certain pre-defined operating modes or tasks, it may be required that operator acknowledgement is received from the operator (USR) <NUM> for the determined task by the wide field of view model (WFOV) <NUM> before controlling an automation element of the vehicle based on the determined task in response to the received operator acknowledgement.

In an embodiment, the user input information <NUM> may comprise a plurality of inputs. Stitching mode input <NUM> is configured to set stitching mode active (on) or inactive (off). As default, the value is on. Operation mode input <NUM> is configured to set operation mode between first operation mode and second operation mode. In the first operation mode image data of one channel (VIS or LWIR) is configured to be processed, and in the second operating mode image data of both channels (VIS and LWIR) are configured to be processed by the model <NUM>. Output mode input <NUM> is configured to set the mode how image data is provided via the display/monitor <NUM>. The output mode input <NUM> may be configured to set at least some of the following modes: Full screen visual mode (FS-VIS), full screen thermal mode (FS-LWIR), split screen with visual image on top (SS-VIS), split screen with thermal image on top (SS-LWIR), Picture-in-Picture mode with thermal image as main image (PIP-VIS), Picture-in-Picture mode with visual image as main image (PIP-LWIR), and fusion mode (FUS) wherein full screen combined image using both channels and four camera data are utilized with stitching mode in. All the output modes can be provided in stitching mode but depending on the installation configuration, user may select also any of the output modes (except fusion mode) without stitching mode on, wherein the image data <NUM>-<NUM>, or <NUM>-<NUM> are not stitched but provided to the monitor <NUM> as unstitched image data.

Various embodiments have been presented. It should be appreciated that in this document, words comprise, include and contain are each used as open-ended expressions with no intended exclusivity.

Without in any way limiting the scope, interpretation, or application of the claims appearing below, a technical effect of one or more of the example embodiments disclosed herein is improved method and apparatus for vision system of a vehicle. Another technical effect of one or more of the example embodiments disclosed herein is improved method and apparatus for vehicle vision mode control.

Another technical effect of one or more of the example embodiments disclosed herein is that it enables performing the vehicle related tasks automatically in the safest and most efficient way possible. Optionally, while the operator may have oversight, the wide field of view model based automation may be principally handled by software in autonomous mode.

Another technical effect of one or more of the example embodiments disclosed herein is that safety is improved since there is less likelihood of human error, and systems are efficiently utilized, and greater efficiency that allows reduced operating costs.

Claim 1:
A surveillance apparatus carried by a vehicle for generating surveillance image data, comprising:
a plurality of camera devices (<NUM>, <NUM>, <NUM>, <NUM>) configured to provide image data on different channels, wherein each channel is associated with at least two camera devices (<NUM>, <NUM>; <NUM>, <NUM>) with substantially same field-of-view (FOV) angles but different line-of-sight (LOS) angles;
a communication interface (<NUM>, <NUM>, <NUM>) for transceiving data;
at least one processor (<NUM>, <NUM>, <NUM>); and
at least one memory (<NUM>, <NUM>, <NUM>) including computer program code (<NUM>, <NUM>, <NUM>);
the at least one memory (<NUM>, <NUM>, <NUM>) and the computer program code (<NUM>, <NUM>, <NUM>) configured to, with the at least one processor (<NUM>, <NUM>, <NUM>), cause the apparatus to:
receive (<NUM>) first image data and second image data of a first channel, wherein the first channel is a longwave infrared (LWIR) channel;
receive (<NUM>) third image data and fourth image data of a second channel, wherein the second channel is a visible spectrum (VIS) channel;
generate (<NUM>) a wide field of view model based on the first, second, third and fourth image data;
determine (<NUM>) a stitching mode, wherein image data of at least one channel is configured to be combined using a stitching algorithm;
determine (<NUM>) a first operating mode, wherein image data of one channel is configured to be processed;
determine (<NUM>) a second operating mode, wherein image data of both channels are configured to be combined;
determine (<NUM>) output mode, wherein the output mode defines configuration for image data to be provided on the display (<NUM>);
receive (<NUM>) selection information for the operating mode, the stitching mode and the output mode; and
generate (<NUM>) output data for the display (<NUM>) based on the selection information and the wide field of view model,
characterized in that in the stitching mode, image data is stitched per channel, by combination of dynamic and static stitching.