Patent Description:
When a surveillance system provides an image received from a camera photographing a surveillance area through a screen, a controller may visually recognize the image through a user interface and adjust a rotation direction or a zoom magnification of the camera.

A surveillance system may monitor a plurality of areas at the same time through a plurality of cameras or may monitor one area in various directions through a camera equipped with a plurality of image sensors.

At this time, the camera classifies and processes a plurality of pieces of image data obtained by the plurality of image sensors through one processor. In the future, as the demand for high-resolution surveillance images increases, it is necessary to develop a surveillance system that may efficiently process high-resolution image data.

<CIT> and <CIT> disclose an image processing system coworking with a plurality of camera modules.

The present disclosure provides an image processing device and a method for efficiently processing a plurality of pieces of image data obtained by a network camera.

According to an aspect of the present disclosure, a surveillance system is provided as defined in claim <NUM>.

In the present embodiment, the plurality of camera modules may constitute one network camera, and the multiple image data request may be a request for simultaneously receiving a plurality of pieces of image data obtained by the one network camera.

In the present embodiment, the communication module may receive a plurality of pieces of image data encoded by the plurality of camera modules from a network switch.

In the present embodiment, the processor may include a decoder configured to decode the plurality of pieces of image data; a scaler configured to scale a plurality of pieces of decoded image data; a multiplexer configured to generate the multiple image data by combining a plurality of pieces of scaled image data; and an encoder configured to encode the multiple image data, wherein the processor may include multiple image data encoded by the encoder in the multiple image data response.

In the present embodiment, the scaler may scale the plurality of pieces of decoded image data, such that the sum of the resolutions of the plurality of pieces of scaled image data is lower than or equal to a resolution required by the client terminal.

According to another aspect of the present disclosure, an image processing method is provided as defined in claim <NUM>.

In the present embodiment, the scaling of the plurality of pieces of image data may include decoding, by a decoder, the plurality of pieces of image data; and scaling, by a scaler, a plurality of pieces of decoded image data, and the generating of the multiple image data may include generating, by a multiplexer, the multiple image data by combining a plurality of pieces of scaled image data; and encoding, by an encoder, the multiple image data.

In the present embodiment, in the scaling of the plurality of pieces of decoded image data, the plurality of pieces of decoded image data may be scaled by the scaler, such that the sum of the resolutions of the plurality of pieces of scaled image data is lower than or equal to a resolution required by the client terminal.

In the present embodiment, the receiving of the multiple image data request and the transmitting of the multiple image data response may be performed by the host device, and the receiving of the plurality of pieces of image data, the scaling of the plurality of pieces of image data, and the generating of the multiple image data may be performed by the FPGA device distinct from the host device.

In the present embodiment, the plurality of pieces of image data received by the FPGA device from the plurality of camera modules may be raw data prior to image processing.

According to embodiments of the present disclosure, it is possible to efficiently process and transmit a plurality of pieces of high resolution image data.

Also, as a master camera module controls the operation of a slave camera module and processes and provides a plurality of pieces of image data, limited resources may be utilized efficiently and the performance of a surveillance system may be improved.

Also, since multiple image data and single image data may be freely provided by a master camera module, user convenience may be improved.

Also, even when an error occurs in the operation of any one camera module from among a plurality of camera modules, image data of the other normally operating camera modules is collected and provided, thereby providing a surveillance system that is highly error-resistant.

Also, by processing a plurality of pieces of image data having high resolutions through a host device provided separately from a camera module, a high-performance surveillance system may be provided.

Also, when an error occurs in the operation of a host device itself, authorizations may be set to a camera module to collect and provide image data, thereby providing a surveillance system that is highly error-resistant.

Also, by using an FPGA device separately from a host device, it is possible to prevent a time difference between a plurality of pieces of image data and to increase the image processing speed of the host device.

The present disclosure provides a surveillance system including a communication module configured to receive a plurality of pieces of image data obtained by a plurality of camera modules, receive a multiple image data request from a client terminal, and transmit a multiple image data response including multiple image data to the client terminal; and a processer configured to generate the multiple image data by scaling the plurality of pieces of image data in response to the multiple image data request and combining a plurality of pieces of scaled image data, wherein the plurality of camera modules share one internet protocol (IP) address.

The present disclosure may include various embodiments and modifications, and embodiments thereof will be illustrated in the drawings and will be described herein in detail. However, this is not intended to limit the present disclosure to particular modes of practice, and it is to be appreciated that all changes that do not depart from the technical scope of the inventive concept as defined by the appended claims are encompassed in the present disclosure. In the description of the present disclosure, certain detailed explanations of the related art are omitted when it is deemed that they may unnecessarily obscure the essence of the present disclosure.

While such terms as "first," "second," etc., may be used to describe various elements, such elements must not be limited to the above terms. The above terms may be used only to distinguish one element from another.

The terms used in the present specification are merely used to describe particular embodiments, and are not intended to limit the inventive concept. In the present specification, it is to be understood that the terms such as "including" or "having," etc., are intended to indicate the existence of the features, numbers, operations, actions, components, parts, or combinations thereof disclosed in the specification, and are not intended to preclude the possibility that one or more other features, numbers, operations, actions, components, parts, or combinations thereof may exist or may be added.

Some embodiments may be described in terms of functional block components and various processing operations. Such functional blocks may be realized by any number of hardware and/or software components configured to perform the specified functions. For example, the present disclosure may employ various integrated circuit (IC) components, e.g., memory elements, processing elements, logic elements, look-up tables, and the like, which may carry out a variety of functions under the control of one or more microprocessors or other control devices. Similarly, where the elements of the present disclosure are implemented using software programming or software elements, the invention may be implemented with any programming or scripting language such as C, C++, Java, assembler, or the like, with the various algorithms being implemented with any combination of data structures, objects, processes, routines or other programming elements. Functional blocks may be implemented in algorithms that are executed on one or more processors. Furthermore, the disclosure may employ any number of conventional techniques for electronics configuration, signal processing, and/or data processing. The words "mechanism", "element", "means", and "configuration" are used broadly and are not limited to mechanical or physical embodiments, but may include software routines in conjunction with processors, etc..

Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

The operation of a surveillance system <NUM> for image data of a camera module <NUM> described below is also applicable to audio data, setting data, metadata, etc. of the camera module <NUM>.

Audio data may refer to sound that may be obtained in a surveillance area of the camera module <NUM>.

Setting data may refer to profile information related to the operation of the camera module <NUM> itself and profile information related to the operation of the camera module <NUM> to obtain and transmit image data or audio data.

Metadata may refer to information related to image data or audio data obtained by the camera module <NUM>.

First, referring to <FIG> and <FIG>, the overall operation of the surveillance system <NUM> according to an embodiment will be described.

<FIG> is a diagram for explaining the surveillance system <NUM> according to an embodiment.

<FIG> is a diagram for describing a network camera CAM according to an embodiment.

Referring to <FIG> and <FIG>, the surveillance system <NUM> according to an embodiment includes a camera module <NUM>, a network switch <NUM>, a network <NUM>, and a client terminal <NUM>.

The surveillance system <NUM> according to an embodiment may provide a configuration that, when information regarding the camera module <NUM> collected by the network switch <NUM> is transmitted to the client terminal <NUM> through the network <NUM>, a user may monitor information transmitted to the client terminal <NUM>.

The network camera CAM may be used to photograph a surveillance area in real time for surveillance or security purposes, and one or more network cameras CAM may be provided.

The network camera CAM may include one or more camera modules <NUM>.

For example, first to fourth camera modules <NUM> to <NUM> may be provided in one network camera CAM. At this time, the network camera CAM may simultaneously obtain four pieces of image data for a surveillance region by simultaneously capturing images of the surveillance region in four different directions.

The camera module <NUM> captures images of a surveillance area to obtain image data for the surveillance area.

The camera module <NUM> may obtain image data through an image sensor, such as a charge-coupled device (CCD) sensor or a complementary metal-oxide-semiconductor (CMOS) sensor.

The image data may be real-time image data and/or recorded image data.

Meanwhile, the image data may be video data and / or still image data.

The camera module <NUM> may be a low-power camera powered by a battery.

A low-power camera normally maintains a sleep mode and periodically wakes up to check whether an event is occurring. The low-power camera is switched to an active mode when an event is occurring and returns to the sleep mode when no event is occurring. In this way, the low-power camera may reduce power consumption by maintaining the active mode only when an event is occurring.

The camera module <NUM> may be a PTZ camera capable of panning and tilting and having a lens with adjustable magnification of zooming.

The camera module <NUM> may encode image data and transmit encoded image data to the client terminal <NUM> connected to the network <NUM> through the network switch <NUM>. Here, the image data may be single image data or multiple image data.

The camera module <NUM> may store image data and image data address corresponding to the image data. An image data address may be information indicating a location where image data is stored.

A plurality of image data addresses may be different from one another. For example, first to fourth image data addresses address may be different from one another.

An image data address may be a real time streaming protocol (RTSP) address. When image data is stored in the camera module <NUM>, an image data address may indicate the camera module <NUM>. At this time, the client terminal <NUM> may access the camera module <NUM> in which image data is stored, based on an RTSP address.

The network switch <NUM> provides a path to access the network camera CAM.

For example, the network switch <NUM> may provide one internet protocol (IP) address indicating an access path to one network camera CAM. Therefore, the first to fourth camera modules <NUM> to <NUM> may share one IP address. In other words, the network switch <NUM> may operate as an IP router.

Meanwhile, the network switch <NUM> may access the camera module <NUM> based on an RTSP address requested by the client terminal <NUM>. In other words, the network switch <NUM> may transmit a request of the client terminal <NUM> only to the camera module <NUM> indicated by an RTSP address.

The network <NUM> may include a wired network or a wireless network. The wireless network may be a 2nd generation (<NUM>) or 3rd generation (<NUM>) cellular communication system, a 3rd generation partnership project (3GPP), a 4th generation (<NUM>) communication system, a long-term evolution (LTE) network, a world interoperability for microwave access (WiMAX) network, etc..

The client terminal <NUM> may display and store image data transmitted from the network switch <NUM>. The client terminal <NUM> may receive a user input and transmit the user input to the network switch <NUM>.

The client terminal <NUM> may include at least one processor. The client terminal <NUM> may be driven while being included in other hardware devices, such as a microprocessor or general purpose computer system. The client terminal <NUM> may be a personal computer or a mobile terminal.

<FIG> is a block diagram showing the configuration of an image processing device <NUM> according to an embodiment.

<FIG> and <FIG> are flowcharts for describing a method of operating the image processing device <NUM> according to embodiments.

Referring to <FIG>, the image processing device <NUM> according to an embodiment includes an image sensor <NUM>, a first communication module <NUM>, a first processor <NUM>, and a first memory <NUM>.

The image processing device <NUM> according to the embodiment of <FIG> may be implemented as the camera module <NUM> and the network switch <NUM> of <FIG>.

For example, at least some of the image sensor <NUM>, the first communication module <NUM>, the first processor <NUM>, and the first memory <NUM> of <FIG> may be implemented in a first camera module <NUM>, and the other components may be implemented in the network switch <NUM>.

Meanwhile, the image processing device <NUM> according to the embodiment of <FIG> may be implemented as the first camera module <NUM>.

In other words, the image sensor <NUM>, the first communication module <NUM>, the first processor <NUM>, and the first memory <NUM> of <FIG> may be included in the first camera module <NUM>.

Hereinafter, the operation of the image processing device <NUM> according to an embodiment implemented in the first camera module <NUM> will be described in detail.

The image sensor <NUM> obtains image data by capturing images of a surveillance area. For example, the image sensor <NUM> may obtain first image data by capturing images of the surveillance area of the first camera module <NUM>.

The first communication module <NUM> receives a plurality of pieces of image data from a plurality of camera modules <NUM>. For example, the first communication module <NUM> may receive second to fourth image data from second to fourth camera modules <NUM> to <NUM>.

The first communication module <NUM> receives a multiple image data request transmitted from the client terminal <NUM> and transmits a multiple image data response including multiple image data to the client terminal <NUM>. At this time, the first communication module <NUM> may communicate with the client terminal <NUM> through the network switch <NUM>.

The client terminal <NUM> may generate a multiple image data request by receiving a user input for receiving a plurality of images captured by images of one surveillance area in multiple directions. In other words, the multiple image data request may be, for example, a request for receiving first to fourth image data of the first to fourth camera modules <NUM> to <NUM> in one screen image.

The multiple image data request may include identification information indicating the network camera CAM.

The multiple image data responses may be generated in response to the multiple image data request. The multiple image data response may include multiple image data, identification information indicating the network camera CAM, etc..

The first processor <NUM> may generate multiple image data by scaling a plurality of pieces of image data in response to the multiple image data request and combining a plurality of pieces of scaled image data.

The first processor <NUM> may scale a plurality of pieces of image data to provide the plurality of pieces of image data in one screen image.

For example, the first processor <NUM> may scale the resolution of each of the first to fourth image data from <NUM> mega pixels to <NUM> mega pixels.

The first processor <NUM> may generate multiple image data by combining a plurality of pieces of image data to provide the plurality of pieces of image data at once. At this time, the first processor <NUM> may combine a plurality of scaled images.

For example, the first processor <NUM> may generate multiple image data having a resolution of <NUM> mega pixels by combining the first to fourth image data of which the resolutions are each scaled to <NUM> mega pixels.

The first processor <NUM> may generate a multiple image data response including multiple image data having the resolution of <NUM> mega pixels. At this time, the resolution of <NUM> mega pixels may be the resolution required by the client terminal <NUM>. In other words, the first processor <NUM> may perform image processing such as scaling on the first to fourth image data to implement a resolution adaptive for the client terminal <NUM>.

As such, according to the present embodiment, even a plurality of pieces of image data having high resolutions may be efficiently generated and transmitted through image processing such as scaling and combining.

The first memory <NUM> stores image data and an image data address.

For example, the first memory <NUM> may store the first to fourth image data and first to fourth image data addresses corresponding thereto.

When a single image data request indicating predetermined image data is received by the first communication module <NUM>, the first processor <NUM> may extract a predetermined image data address stored in the first memory <NUM>.

The single image data request may include identification information indicating predetermined image data. Identification information indicating predetermined image data may be identification information indicating a predetermined camera module <NUM>, but is not limited thereto. For example, a single image data request may indicate second image data by including identification information indicating a second camera module <NUM>.

The first processor <NUM> may generate a predetermined image data address response including a extracted predetermined image data address and may transmit the predetermined image data address response to the client terminal <NUM>.

For example, when the first communication module <NUM> receives a single image data request indicating the second image data, the first processor <NUM> may extract a second image data address from the first memory <NUM>, generate a second image data address response including the second image data address, and transmit the second image data address response to the client terminal <NUM>.

The first communication module <NUM> may download an upgrade program from a server (not shown). At this time, the first processor <NUM> may upgrade each of the first to fourth camera modules <NUM> to <NUM> based on the upgrade program.

For example, the upgrade program may be a program for upgrading a profile related to image acquisition of the image sensor <NUM>, a profile related to image processing of the camera module <NUM>, an event detection function of the camera module <NUM>, or a network connection function.

Referring to <FIG>, when the client terminal <NUM> receives a user input (operation S101), the client terminal <NUM> transmits a multiple image data request to the network switch <NUM> (operation S103).

At this time, the user input may be, for example, an user input for receiving first to fourth image data of the first to fourth camera modules <NUM> to <NUM> in one screen image.

At this time, the multiple image data request may include identification information indicating the network camera CAM.

Subsequently, the network switch <NUM> transmits the multiple image data request to the first camera module <NUM> included in the network camera CAM (operation S105).

The first camera module <NUM> may be designated in advance as a master camera module of the network camera CAM.

Meanwhile, the first camera module <NUM> obtains first image data through the image sensor <NUM> (operation S107) and receives second to fourth image data from the second to fourth camera modules <NUM> to <NUM> through the first communication module <NUM> (operation S109).

The first camera module <NUM> generates multiple image data in response to the multiple image data request (operation S111).

At this time, the first camera module <NUM> may scale the first to fourth image data and combine scaled first to fourth image data to generate multiple image data.

The first image data obtained by the image sensor <NUM> and the second to fourth image data received through the first communication module <NUM> may each have image quality of, for example, a resolution of <NUM> mega pixels, a frame rate of <NUM> frames per second, and a bit rate of <NUM> megabits per second (bps).

The scaled first to fourth image data may each have, for example, a resolution of <NUM> mega pixels and a frame rate of <NUM> fps.

The multiple image data is a result of encoding after the scaled first to fourth image data are combined, and may have the quality of a resolution of <NUM> mega pixels, a frame rate of <NUM> fps, and a bit rate of <NUM> mega bps.

When the first camera module <NUM> transmits a multiple image data response including the multiple image data to the network switch <NUM> (operation S113), the network switch <NUM> transmits the multiple image data response to the client terminal <NUM> (operation S115).

Subsequently, the client terminal <NUM> reproduces the multiple image data through a screen (operation S117).

At this time, the client terminal <NUM> may reproduce the multiple image data having the quality of a resolution of <NUM> mega pixels, a frame rate of <NUM> fps, and a bit rate of <NUM> mega bps.

As described above, according to the present embodiment, by controlling the operation of the camera module <NUM> provided in the network camera CAM and selecting a master camera module to process a plurality of pieces of image data simultaneously, limited resources may be efficiently utilized, and thus the performance of the surveillance system <NUM> according to an embodiment may be improved.

Referring to <FIG>, the first camera module <NUM>, which is the master camera module, stores second to fourth image data addresses in the first memory <NUM> (operation S301).

In other words, the first camera module <NUM> may store not only the first image data address corresponding to the first image data obtained by itself, but also the second to fourth image data addresses received from the second to fourth camera modules <NUM> to <NUM> in the first memory <NUM>.

Meanwhile, when the client terminal <NUM> receives a user input indicating the second image data (operation S303), the client terminal <NUM> transmits a single image data request indicating the second image data to the network switch <NUM> (operation S305).

At this time, the single image data request may include identification information indicating the second camera module <NUM>.

The network switch <NUM> transmits a second image data address request to the first camera module <NUM> (operation S307) and receives a second image data address response from the first camera module <NUM> (operation S309). At this time, the second image data address request does not include an RTSP address of the second image data, whereas the second image data address response may include the RTSP address of the second image data.

The network switch <NUM> transmits the second image data address response to the client terminal <NUM> (operation S311) and receives a second image data request from the client terminal <NUM> (operation S313). At this time, the second image data request may include the RTSP address of the second image data.

The network switch <NUM> transmits the second image data request to the second camera module <NUM> (operation S315).

At this time, the network switch <NUM> may transmit the second image data request to the second camera module <NUM> based on the RTSP address of the second image data included in the second image data request.

Meanwhile, the second camera module <NUM> obtains second image data (operation S317) and transmits a single image data response including the second image data to the client terminal <NUM> (operation S319).

The client terminal <NUM> reproduces the single image data (operation S321). At this time, the client terminal <NUM> may reproduce the second image data having the quality of a resolution of <NUM> mega pixels, a frame rate of <NUM> fps, and a bit rate of <NUM> mega bps.

As described above, according to the present embodiment, since the first camera module <NUM> functions as the master camera module, multiple image data and single image data may be freely provided, thereby providing user convenience.

Hereinafter, descriptions identical to those given above will be omitted or briefly given.

Next, with reference to <FIG>, the overall operation of a surveillance system <NUM>' according to another embodiment will be described.

<FIG> is a diagram for explaining the surveillance system <NUM>' according to another embodiment.

Referring to <FIG>, the surveillance system <NUM>' according to another embodiment includes the camera module <NUM>, the network switch <NUM>, the network <NUM>, the client terminal <NUM>, and a host device <NUM>.

The surveillance system <NUM>' according to another embodiment may provide a configuration that, when information of the camera module <NUM> collected by the host device <NUM> through the network switch <NUM> is transmitted to the client terminal <NUM> through the network <NUM>, a user may monitor information transmitted to the client terminal <NUM>.

One or more camera modules <NUM> may be provided.

The camera module <NUM> may obtain image data through an image sensor, encode obtained image data, and transmit encoded image data to the host device <NUM> through the network switch <NUM>.

The network switch <NUM> transmits a plurality of pieces of image data received from a plurality of camera modules <NUM> to the host device <NUM>. At this time, the plurality of pieces of image data transmitted and received through the network switch <NUM> may be data respectively encoded by the plurality of camera modules <NUM>.

For example, the network switch <NUM> may transmit encoded first to fourth image data received from the first to fourth camera modules <NUM> to <NUM> to the host device <NUM>. At this time, the encoded first to fourth image data may not be combined and may be separated from one another.

The network switch <NUM> may operate as an IP router for the plurality of camera modules <NUM>. In other words, the plurality of camera modules <NUM> may share one IP address. Therefore, the client terminal <NUM> may access a plurality of pieces of image data obtained by the plurality of camera modules <NUM> using one IP address.

The network <NUM> may include a wired network or a wireless network. The client terminal <NUM> may transmit and receive data to and from the host device <NUM> through the network <NUM>.

The client terminal <NUM> may display and store image data transmitted from the host device <NUM>.

The client terminal <NUM> may receive a user input and transmit the user input to the host device <NUM>.

The host device <NUM> receives a plurality of pieces of image data obtained by the plurality of camera modules <NUM> from the network switch <NUM> and transmits multiple image data generated by combining the plurality of pieces of image data to one another to the client terminal <NUM> according to a user's request.

For example, the host device <NUM> may decode encoded first to fourth image data received from the network switch <NUM>, scale decoded first to fourth image data, generate multiple image data by combining scaled first to fourth image data, encode the multiple image data, and transmit encoded multiple image data to the client terminal <NUM>.

When an operation error is detected, the host device <NUM> may select one camera module <NUM> from among the plurality of camera modules <NUM> as a master camera module and set authorization to the master camera module to perform the operation of the host device <NUM>.

For example, when it is determined that the operation of the host device <NUM> is impossible due to an overload or an external attack, the host device <NUM> may select the first camera module <NUM> from among the plurality of camera modules <NUM> as the master camera module and transmit a network service provision request to the first camera module <NUM>, thereby transferring authorization for providing multiple image data to the first camera module <NUM>.

Meanwhile, the host device <NUM> may perform a reboot by itself after transferring authorization for providing multiple image data to the first camera module <NUM>. The host device <NUM> that is successfully rebooted may thereafter perform a function of providing a path to access the network camera CAM together with the network switch <NUM>.

According to the present embodiment, even when an error occurs in the operation of any one camera module <NUM>, the host device <NUM> collects and provides image data from the other normally operating camera modules <NUM>, thereby providing the surveillance system <NUM>' according to another embodiment that is highly error-resistant.

Also, when an error occurs in the operation of the host device <NUM> itself, authorizations may be set to the camera module <NUM> to collect and provide image data, thereby providing the surveillance system <NUM>' according to another embodiment that is highly error-resistant.

<FIG> is a block diagram showing the configuration of an image processing device <NUM> according to another embodiment.

<FIG> are flowcharts for describing a method by which the image processing device <NUM> generates multiple image data according to another embodiment.

Referring to <FIG>, the image processing device <NUM> according to another embodiment includes a second communication module <NUM>, a second processor <NUM>, and a second memory <NUM>.

The image processing device <NUM> according to another embodiment of <FIG> may be implemented as the host device <NUM> of <FIG>. In other words, the second communication module <NUM>, the second processor <NUM>, and the second memory <NUM> of <FIG> may be included in the host device <NUM>.

Hereinafter, the operation of the image processing device <NUM> according to another embodiment implemented in the host device <NUM> will be described.

The second communication module <NUM> receives a plurality of pieces of image data obtained by the plurality of camera modules <NUM>, receives a multiple image data request transmitted from the client terminal <NUM>, and transmits a multiple image data response including multiple image data to the client terminal <NUM>.

In detail, the second communication module <NUM> may receive a plurality of pieces of image data encoded by the plurality of camera modules <NUM> from the network switch <NUM>.

For example, the second communication module <NUM> may receive encoded first to fourth image data from the network switch <NUM>. In this case, the encoded first to fourth image data may each have <NUM> mega pixels resolution.

The multiple image data request may be a request for simultaneously receiving a plurality of pieces of image data obtained by one network camera CAM.

For example, the multiple image data request may be a request for receiving first to fourth image data of the first to fourth camera modules <NUM> to <NUM> in one screen image.

The second processor <NUM> generates multiple image data by scaling a plurality of pieces of image data in response to the multiple image data request and combining a plurality of pieces of scaled image data.

In detail, the second processor <NUM> includes a decoder <NUM>, a scaler <NUM>, a first multiplexer <NUM>, and a first encoder <NUM>.

The decoder <NUM> may decode a plurality of pieces of encoded image data, the scaler <NUM> may scale a plurality of pieces of decoded image data, the first multiplexer <NUM> may generate multiple image data by combining a plurality of pieces of scaled image data, and the first encoder <NUM> may encode the multiple image data.

For example, the decoder <NUM> may decode encoded first to fourth image data, and the scaler <NUM> may scale decoded first to fourth image data.

At this time, the scaler <NUM> may scale a plurality of pieces of decoded image data, such that the sum of the resolutions of the plurality of pieces of scaled image data is adaptive for a resolution required by the client terminal <NUM>.

For example, the scaler <NUM> may scale the plurality of pieces of decoded image data, such that the sum of the resolutions of the plurality of pieces of scaled image data is lower than or equal to the resolution required by the client terminal <NUM>.

The resolution required by the client terminal <NUM> may be the highest resolution that may be displayed by the client terminal <NUM> or may be a resolution set by a user input, but is not limited thereto.

For example, the resolution required by the client terminal <NUM> may be <NUM> mega pixels, and the scaled first to fourth image data may each have a resolution of <NUM> mega pixels.

The first multiplexer <NUM> may combine the plurality of pieces of scaled image data to generate one multiple image data. The one multiple image data may refer to image data that may be displayed in one screen.

The first encoder <NUM> may encode multiple image data.

At this time, the resolution of encoded multiple image data may satisfy the resolution required by the client terminal <NUM>. In other words, the resolution of encoded multiple image data may lower than or equal to the resolution required by the client terminal <NUM>.

For example, the encoded multiple image data may each have a resolution of <NUM> mega pixels.

The second processor <NUM> may include the multiple image data encoded by the first encoder <NUM> in a multiple image data response.

As described above, according to the present embodiment, by processing a plurality of pieces of image data having high resolutions through the host device <NUM> provided separately from the camera module <NUM>, the surveillance system <NUM>' according to another embodiment with high performance may be provided.

Meanwhile, when an operation error of the host device <NUM> is detected, the second processor <NUM> may select one camera module <NUM> from among the plurality of camera modules <NUM> as a master camera module.

For example, the second processor <NUM> may select the first camera module <NUM> as the master camera module from among the first to fourth camera modules <NUM> to <NUM> in response to an operation error of the host device <NUM>.

When an operation error of the master camera module is detected, the second processor <NUM> may re-select one camera module <NUM> from among the remaining camera modules <NUM> as the master camera module.

For example, when the second processor <NUM> receives a message about an operation error such as a request to stop a network service from the first camera module <NUM>, the second processor <NUM> may select a second camera module <NUM> from among the second to fourth camera modules <NUM> to <NUM> as the master camera module.

At this time, the second processor <NUM> may select the camera module <NUM> of the predetermined highest priority from among the plurality of camera modules <NUM> or the camera module <NUM> with the least load from among the plurality of camera modules <NUM> as the master camera module.

The camera module <NUM> with the least load from among the plurality of camera modules <NUM> may be the camera module <NUM> with the least load when an operation error of the host device <NUM> is detected or the camera module <NUM> with the least load measured during a predetermined period, but is not limited thereto.

When the master camera module is selected by the second processor <NUM>, the second communication module <NUM> may transmit a network service provision request to the master camera module.

The network service provision request may be a request for a network service providing operation performed by the host device <NUM>. For example, the network service provision request may include at least one of a request for image transmission performed by the second communication module <NUM> and a request for image processing performed by the second processor <NUM>.

The master camera module may perform functions of the second communication module <NUM> and the second processor <NUM> in response to a network service provision request. In other words, the master camera module may efficiently process high-resolution image data by performing the operation of the first camera module <NUM> described above with reference to <FIG> in response to a network service provision request.

The second memory <NUM> may store data received through the second communication module <NUM> and/or data processed by the second processor <NUM>.

The second memory <NUM> may store an upgrade program downloaded from a server (not shown).

According to the present embodiment, even when an operation error of the camera module <NUM> or the host device <NUM> is detected, high-resolution image data obtained by the camera module <NUM> may be processed and provided to a user.

Referring to <FIG>, the host device <NUM> receives a multiple image data request from the client terminal <NUM> through the second communication module <NUM>.

In detail, when the client terminal <NUM> receives a user input for receiving a plurality of images on one screen (operation S501), the client terminal <NUM> transmits a multiple image data request to the host device <NUM> (operation S503).

At this time, the multiple image data request may include identification information indicating the network camera CAM provided with the first to fourth camera modules <NUM> to <NUM>.

Meanwhile, the host device <NUM> receives a plurality of pieces of image data obtained by the plurality of camera modules <NUM> through the second communication module <NUM>.

Here, the plurality of camera modules <NUM> shares one IP address. For example, the first to fourth camera modules <NUM> to <NUM> constituting one network camera CAM may share one IP address.

Specifically, the network switch <NUM> receives encoded first to fourth image data from the first to fourth camera modules <NUM> to <NUM> (operation S505) and transmits the encoded first to the fourth image data to the host device <NUM> (operation S507).

Subsequently, the host device <NUM> generates multiple image data in response to the multiple image data request (operation S509).

In detail, the host device <NUM> generates multiple image data by scaling a plurality of pieces of image data in response to the multiple image data request and combining a plurality of pieces of scaled image data, through the second processor <NUM>.

Referring to <FIG>, the second processor <NUM> of the host device <NUM> may include the decoder <NUM>, the scaler <NUM>, the first multiplexer <NUM>, and the first encoder <NUM>.

For example, the client terminal <NUM> may request multiple image data having the quality of a resolution of <NUM> mega pixels, a frame rate of <NUM> fps, and a bit rate of <NUM> mega bps.

The decoder <NUM> may decode encoded first image data E1, encoded second image data E2, encoded third image data E3, and encoded fourth image data E4 to output decoded first image data D1, decoded second image data D2, decoded third image data D3and decoded fourth image data D4, respectively.

Here, for example, encoded first to fourth image data E1 to E4 and decoded first to fourth image data D1 to D4 may each have the quality of a resolution of <NUM> mega pixels, a frame rate of <NUM> fps, and a bit rate of <NUM> mega bps.

The scaler <NUM> may scale the decoded first image data D1, the decoded second image data D2, the decoded third image data D3, and the decoded fourth image data D4 to output scaled first image data S1, scaled second image data S2, scaled third image data S3, and scaled fourth image data S4, respectively.

In this case, scaled first to fourth image data S1 to S4 may each have the quality of a resolution of <NUM> mega pixels and a frame rate of <NUM> fps, for example.

The first multiplexer <NUM> may generate multiple image data M by combining the scaled first image data S1, the scaled second image data S2, the scaled third image data S3, and the scaled fourth image data S4.

At this time, combined first image data C1, combined second image data C2, combined third image data C3, and combined fourth image data C4 constituting the multiple image data M may each have, for example, the quality of a resolution of <NUM> mega pixels and a frame rate of <NUM> fps.

The first encoder <NUM> may encode the multiple image data M and generate encoded multiple image data EM.

At this time, the encoded multiple image data EM may have, for example, the quality of a resolution of <NUM> mega pixels, a frame rate of <NUM> fps, and a bit rate of <NUM> mega bps.

As a result, the image quality of the encoded multiple image data EM may satisfy the image quality required by the client terminal <NUM>.

Referring back to <FIG>, when the host device <NUM> transmits a multiple image data response including multiple image data to the client terminal <NUM> through the second communication module <NUM> (operation S511), the client terminal <NUM> reproduces the multiple image data on one screen (operation S513).

Referring to <FIG>, when the client terminal <NUM> receives a user input (operation S701), the client terminal <NUM> transmits a multiple image data request to the host device <NUM> (operation S703).

Meanwhile, when an operation error of the host device <NUM> is detected, the host device <NUM> selects the first camera module <NUM> from among the plurality of camera modules <NUM> as a master camera module.

The host device <NUM> may detect an error when a problem is detected in image reception and/or processing.

For example, the host device <NUM> may detect an error when image data is not received from the camera module <NUM> for a predetermined time period.

The host device <NUM> may also detect an error when, for example, the host device <NUM> is unable to decode a plurality of pieces of image data, is unable to scale a plurality of pieces of decoded image data, is unable to combine a plurality of pieces of scaled image data, or is unable to encode multiple image data.

The host device <NUM> may select the first camera module <NUM> from among the first to fourth camera modules <NUM> to <NUM> as a master camera module. In this case, the first camera module <NUM> may be the camera module <NUM> of the predetermined highest priority or the camera module <NUM> with the least load from among the first to fourth camera modules <NUM> to <NUM>, but is not limited thereto.

Subsequently, the host device <NUM> transmits a network service provision request to the first camera module <NUM>, which is the master camera module (operation S709).

The first camera module <NUM>, which received the network service provision request, obtains first image data (operation S711) and receives second to fourth image data from the second to fourth camera modules <NUM> to <NUM> (operation S713).

When the client terminal <NUM> receives a user input again (operation S715), the client terminal <NUM> transmits a multiple image data request to the host device <NUM> (operation S717).

The host device <NUM> forwards the multiple image data request received from the client terminal <NUM> to the first camera module <NUM> (operation S719).

The first camera module <NUM> generates multiple image data by combining the first to fourth image data in response to the multiple image data request (operation S721).

The first camera module <NUM> transmits a multiple image data response including the multiple image data to the client terminal <NUM> (operation S723), and the client terminal <NUM> reproduces the multiple image data as one screen (operation S725).

Next, with reference to <FIG>, the overall operation of a surveillance system <NUM>" according to another embodiment will be described.

<FIG> is a diagram for explaining the surveillance system <NUM>" according to another embodiment.

Referring to <FIG>, the surveillance system <NUM>" according to another embodiment includes the camera module <NUM>, the network switch <NUM>, the network <NUM>, the client terminal <NUM>, the host device <NUM>, and a field programmable gate array (FPGA) device <NUM>.

The surveillance system <NUM>" according to another embodiment may provide a configuration that, when information regarding the camera module <NUM> collected by the FPGA device <NUM> is transmitted to the client terminal <NUM> through the host device <NUM> and the network <NUM>, a user may monitor information transmitted to the client terminal <NUM>.

The camera module <NUM> may transmit image data in a raw data state obtained by the image sensor to the FPGA device <NUM>. Hereinafter, image data in a raw data state will be referred to as raw image data.

The camera module <NUM> may be connected to the FPGA device <NUM> via a wire. In other words, the FPGA device <NUM> may receive image data of the camera module <NUM> quickly and efficiently by receiving raw image data from the camera module <NUM> connected via a wire.

Also, the FPGA device <NUM> may prevent a time difference between a plurality of pieces of image data by receiving a plurality of pieces of raw image data from the plurality of camera modules <NUM>.

Meanwhile, the camera module <NUM> encodes image data and transmits encoded image data to the network switch <NUM>. The network switch <NUM> forwards the encoded image data received from the camera module <NUM> to the host device <NUM>.

The FPGA device <NUM> may generate multiple image data by scaling a plurality of pieces of raw image data and combining a plurality of pieces of scaled raw image data.

In detail, the FPGA device <NUM> may scale the plurality of pieces of raw image data, such that the sum of the resolutions of the plurality of pieces of scaled raw image data is lower than or equal to the resolution required by the client terminal <NUM>. At this time, when the sum of the resolutions of the plurality of pieces of raw image data is lower than or equal to the resolution required by the client terminal <NUM>, the FPGA device <NUM> may omit scaling.

The FPGA device <NUM> may transmit the multiple image data to the host device <NUM>.

The host device <NUM> receives the multiple image data from the FPGA device <NUM>, receives a multiple image data request from the client terminal <NUM>, and transmits a multiple image data response including the multiple image data to the client terminal <NUM>.

The host device <NUM> may encode the multiple image data received from the FPGA device <NUM>. In other words, the host device <NUM> does not need to perform decoding or scaling.

Meanwhile, the host device <NUM> may transmit encoded image data received from the network switch <NUM> to the client terminal <NUM>.

For example, when the host device <NUM> receives a single image data request indicating second image data from the client terminal <NUM>, the host device <NUM> may transmit encoded second image data received from the network switch <NUM> to the client terminal <NUM>. At this time, the host device <NUM> does not need to perform encoding.

According to the present embodiment, since the load of the host device <NUM> is reduced, the surveillance system <NUM>') according to another embodiment with improved efficiency may be provided.

<FIG> is a block diagram showing the configuration of an image processing device according to another embodiment.

Referring to <FIG>, an image processing device <NUM> according to another embodiment includes a third communication module <NUM>, a third processor <NUM>, and a third memory <NUM>.

The image processing device <NUM> of <FIG> may be implemented as the host device <NUM> of <FIG>. In other words, the third communication module <NUM>, the third processor <NUM>, and the third memory <NUM> of <FIG> may be included in the host device <NUM>.

The third communication module <NUM> receives the multiple image data from the FPGA device <NUM>, receives a multiple image data request from the client terminal <NUM>, and transmits a multiple image data response including the multiple image data to the client terminal <NUM>.

In detail, the third communication module <NUM> may receive multiple image data generated by combining a plurality of pieces of raw image data obtained by the plurality of camera modules <NUM> from the FPGA device <NUM>.

In this case, since the multiple image data is image data that is decoded, scaled, and multiplexed by the FPGA device <NUM>, decoding, scaling, and multiplexing need not be performed by the image processing device <NUM>.

The third processor <NUM> encodes the multiple image data received from the FPGA device <NUM>. Therefore, the multiple image data included in a multiple image data response may be the multiple image data encoded by the third processor <NUM>.

Meanwhile, the third processor <NUM> does not need to encode the multiple image data received from the network switch <NUM>.

The third memory <NUM> may store data received through the third communication module <NUM> and/or data processed by the third processor <NUM>.

The third memory <NUM> may store an upgrade program downloaded from a server (not shown).

As described above, according to the present embodiment, as the function of the third processor <NUM> of the host device <NUM> is reduced, the image processing speed of the host device <NUM> is increased, and thus the performance of the surveillance system <NUM>" according to another embodiment may be improved.

<FIG> and <FIG> are flowcharts for describing a method by which an image processing device generates multiple image data according to another embodiment.

Referring to <FIG>, when the client terminal <NUM> receives a user input (operation S901), the client terminal <NUM> transmits a multiple image data request to the host device <NUM> (operation S903).

Meanwhile, the FPGA device <NUM> receives a plurality of pieces of raw image data from the plurality of camera modules <NUM>.

In detail, the FPGA device <NUM> receives first to fourth raw image data from the first to fourth camera modules <NUM> to <NUM> (operation S905) and generates multiple image data by combining the first to fourth raw image data (operation S907).

At this time, the FPGA device <NUM> may scale the first to fourth raw image data based on a resolution required by the client terminal <NUM> and generate the multiple image data by combining scaled first to fourth raw image data.

Next, the FPGA device <NUM> transmits the multiple image data to the host device <NUM> (operation S909).

The host device <NUM> encodes the multiple image data received from the FPGA device <NUM> (operation S911).

At this time, encoded multiple image data EM may have, for example, the quality of a resolution of <NUM> mega pixels, a frame rate of <NUM> fps, and a bit rate of <NUM> mega bps.

Referring to <FIG>, the FPGA device <NUM> may include a second multiplexer <NUM>, and the host device <NUM> may include a second encoder <NUM>.

The FPGA device <NUM> may receive first to fourth raw image data R1 to R4 having the quality of a resolution of <NUM> mega pixels and a frame rate of <NUM> fps from the first to fourth camera modules <NUM> to <NUM>.

The second multiplexer <NUM> of the FPGA device <NUM> may generate the multiple image data M by combining first raw image data R1, second raw image data R2, third raw image data R3, and fourth raw image data R4.

At this time, multiple image data M may have, for example, the quality of a resolution of <NUM> mega pixels, a frame rate of <NUM> fps, and a bit rate of <NUM> mega bps.

The second encoder <NUM> of the host device <NUM> may generate encoded multiple image data EM by encoding the multiple image data M received from the FPGA device <NUM>.

Referring back to <FIG>, when the host device <NUM> transmits a multiple image data response including multiple image data to the client terminal <NUM> (operation S913), the client terminal <NUM> reproduces the multiple image data on one screen (operation S915).

At this time, the client terminal <NUM> may reproduce multiple image data encoded by the host device <NUM>.

While the disclosure has been particularly shown and described with reference to preferred embodiments thereof. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the disclosure as defined by the appended claims.

The preferred embodiments should be considered in descriptive sense only and not for purposes of limitation. Therefore, the scope of the disclosure is defined not by the detailed description of the disclosure but by the appended claims, and all differences within the scope will be construed as being included in the disclosure.

Claim 1:
A surveillance system comprising:
a plurality of camera modules (<NUM>, <NUM>, <NUM>, <NUM>) configured to simultaneously obtain a plurality of pieces of image data of a surveillance region in a plurality of different directions and to share one internet protocol, IP, address while having separate real-time streaming protocol, RTSP, addresses;
a network switch (<NUM>);
a client terminal (<NUM>); and
a host device (<NUM>), wherein the host device (<NUM>) comprises:
a communication module (<NUM>) configured to receive the plurality of pieces of image data obtained by the plurality of camera modules, receive a multiple image data request from the client terminal (<NUM>), and transmit a multiple image data response comprising multiple image data to the client terminal (<NUM>); and
a processor (<NUM>) configured to generate the multiple image data by scaling the plurality of pieces of image data in response to the multiple image data request and combining a plurality of pieces of scaled image data,
when the processor (<NUM>) detects an overload of the host device (<NUM>), the processor (<NUM>) is configured to select one camera module from among the plurality of camera modules (<NUM>, <NUM>, <NUM>, <NUM>) as a master camera module,
when the master camera module is selected,
the communication module (<NUM>) is configured to transmit a network service provision request to the master camera module and to forward the multiple image data request received from the client terminal (<NUM>), and
the master camera module is configured to perform the function of the communication module (<NUM>) and/or the function of the processor in response to the network service provision request;
wherein the host device (<NUM>) is configured to reboot itself and provide a path to access the master camera module together with the network switch (<NUM>) after the network service provision request is transmitted to the master camera module.