Patent Description:
A technique that fixedly superimposes a mask on a specific region in an image captured by a monitoring camera or the like in order to secure the security or privacy of the specific region is known. In this specification, a mask to be fixedly superimposed on a specific region will be called a fixed mask hereinafter. Also, a technique that, when an image contains a moving object, superimposes a mask on a region (moving object region) including the moving object in the image by taking account of the privacy of the moving object is known. In this specification, a mask to be superimposed on a moving object region will be called a moving object mask hereinafter.

<CIT> discloses a method of superimposing a moving object mask on only a moving object outside a region set by the user, in order to prevent the moving object mask from being superimposed on a parcel conveyed in a distribution center.

In this technique disclosed in <CIT>, if a fixed region includes a region where a moving object mask may be rendered in an image and the moving object mask is superimposed on a fixed mask rendered in the fixed region, the existence of the moving object in the fixed region may be notified. In this case, an intended concealment of the fixed region by the fixed mask may become insufficient. The document <CIT> discloses a method and an apparatus for updating a background model used for background subtraction of an image, wherein image contents in a region of the image, which is classified as foreground, is replaced by image contents of a corresponding region in a background image. The document <CIT> relates to a camera and an image processor. More specifically, it discloses that a a video signal produced through a shooting operation of a camera is sent to a motion detecting unit which detects a motion included in the video signal to set an area of the motion as a space motion area and inputs information of the area to a distance determination circuit which calculates a distance between the motion area and the camera by use of a parallax signal produced from stereo cameras disposed in the camera to send information of the space motion area and the distance to a mask determination circuit. The mask determination circuit conducts a comparison between space motion area information and space mask area information and between distance information of the motion area and that of the space mask area, and resultantly compares three-dimensional positions between the detected motion and the mask to determine a position relationship therebetween. Further prior art is also known from the document <CIT> relating to a method and a device for video surveillance.

A technique for reliably implementing the concealment of a fixed region will be provided.

The invention in its first aspect provides an image processing apparatus as specified in claims <NUM> to <NUM>.

The invention in its second aspect provides an image processing apparatus control method as specified in claim <NUM>.

The invention in its third aspect provides a non-transitory computer-readable storage medium as specified in claim <NUM>.

An exemplary embodiment will be explained below with reference to the accompanying drawings. In a system corresponding to this embodiment, a moving object is detected in an image obtained by imaging a predetermined real space, and a moving object mask as a first mask is superimposed on the detected moving object. In addition, a fixed mask as a second mask is further superimposed on the image on which the moving object mask is superimposed, based on information of a fixed region that is preset in the captured image. An image obtained as described above on which the two types of masks, that is, the moving object mask and the fixed mask are superimposed is output (displayed).

In this embodiment, a moving object includes a person, a vehicle, or another arbitrary body. The moving object can include an arbitrary body as long as the body is an object detectable based on the difference between a background image created based on, for example, an image obtained by imaging a real space where the moving object does not exist, and the captured image. Also, an object can move in a real space as an imaging target, and can stay or can be standing still in a predetermined place in the real space.

<FIG> is a view showing an example of a system configuration corresponding to the exemplary embodiment. As shown in <FIG>, a system <NUM> of this embodiment includes an imaging apparatus <NUM>, a client apparatus <NUM>, an input apparatus <NUM>, a display apparatus <NUM>, and a network <NUM>. This configuration of the system <NUM> shown in <FIG> is merely an example, so it is also possible to further add other constituent elements other than these constituent elements. Each constituent element will be explained below.

The imaging apparatus <NUM> is an imaging apparatus such as a network camera. The imaging apparatus <NUM> generates a captured image by imaging a real space in accordance with imaging parameters such as a posture and a viewing angle set in the client apparatus <NUM>, and transmits the captured image to the client apparatus <NUM> across the network <NUM>. The imaging apparatus <NUM> can include a plurality of network cameras instead of a single network camera.

The client apparatus <NUM> receives a captured image from the imaging apparatus <NUM> across the network <NUM>. The client apparatus <NUM> detects a moving object from the received captured image, and sets a moving object region in the captured image based on the detected moving object. Also, the client apparatus <NUM> superimposes a moving object mask on the moving object region, and superimposes a fixed mask on a fixed region that is preset in the captured image. The client apparatus <NUM> outputs the captured image on which the moving object mask and the fixed mask are superimposed to the display apparatus <NUM>, thereby displaying the image.

The input apparatus <NUM> includes user interfaces such as a keyboard, a mouse, and a touch panel, and the user can input various instructions to the client apparatus <NUM> by operating the input apparatus <NUM>. The display apparatus <NUM> is a CRT, a liquid crystal display, or the like, and can display processing results from the client apparatus <NUM> by using images, characters, and the like.

Note that <FIG> shows the client apparatus <NUM>, the input apparatus <NUM>, and the display apparatus <NUM> as independent apparatuses, but the embodiment is not limited to this configuration. For example, it is also possible to integrate these apparatuses into a single apparatus, integrate the client apparatus <NUM> and the display apparatus <NUM>, or integrate the input apparatus <NUM> and the display apparatus <NUM>. The client apparatus <NUM> and the imaging apparatus <NUM> can also be integrated.

The network <NUM> is a network that connects the imaging apparatus <NUM> and the client apparatus <NUM> so that they can communicate with each other. The network <NUM> includes routers, switches, cables, and the like satisfying the communication standard such as Ethernet®. In this embodiment, the network <NUM> need only be able to perform communication between the imaging apparatus <NUM> and the client apparatus <NUM>, and the communication standard, scale, and configuration of the network <NUM> do not matter. For example, the network <NUM> can be the Internet, a wired LAN (Local Area Network), a wireless LAN, or a WAN (Wide Area Network).

The hardware configurations of the imaging apparatus <NUM> and the client apparatus <NUM> will be explained below with reference to <FIG>. First, a hardware configuration example of the imaging apparatus <NUM> will be explained by using a block diagram of <FIG>. As shown in <FIG>, the imaging apparatus <NUM> can include a CPU <NUM>, a main storage device <NUM>, an auxiliary storage device <NUM>, a driving unit <NUM>, and an imaging unit <NUM>. Note that this configuration shown in <FIG> is merely an example of a configuration applicable to the imaging apparatus <NUM>, and can be modified/changed as needed.

The CPU <NUM> executes processing by using computer programs and data stored in the main storage device <NUM>. Thus, the CPU <NUM> controls the operation of the whole imaging apparatus <NUM>, and executes or controls each processing (to be described later) to be executed by the imaging apparatus <NUM>. For example, the CPU <NUM> implements the function of each functional unit shown in <FIG> by executing processing by using the computer programs and data stored in the main storage device <NUM>. Details of these functional units will be described later.

The main storage device <NUM> is a storage device such as a RAM (Random Access Memory). The main storage device <NUM> has an area for storing computer programs and data loaded from the auxiliary storage device <NUM>, captured images obtained by the imaging unit <NUM>, and various kinds of data received from the client apparatus <NUM> via a network I/F <NUM>. In addition, the main storage device <NUM> has a work area which the CPU <NUM> uses when executing various kinds of processing. Thus, the main storage device <NUM> can provide various areas as needed.

The auxiliary storage device <NUM> is a large-capacity storage device such as a hard disk drive (HDD), a ROM (Read Only Memory), or an SSD (Solid State Drive). The auxiliary storage device <NUM> stores an OS (Operating System), and computer programs and data for causing the CPU <NUM> to execute or control each processing (to be described later) to be executed by the imaging apparatus <NUM>. The auxiliary storage device <NUM> also stores data (for example, the abovementioned imaging parameters) received from the client apparatus <NUM> via the network I/F <NUM>. The computer programs and data stored in the auxiliary storage device <NUM> are loaded into the main storage device <NUM> as needed under the control of the CPU <NUM>, and processed by the CPU <NUM>.

The driving unit <NUM> drives the imaging unit <NUM> based on imaging parameters received from the client apparatus <NUM>, and controls the posture (imaging direction), the viewing angle, and the like of the imaging unit <NUM>. Control targets of the driving unit <NUM> are not limited to specific targets, but can be one or both of the posture and viewing angle of the imaging unit <NUM>, and can also be another target (for example, the position of the imaging unit <NUM>). Furthermore, the position, posture, and viewing angle of the imaging unit <NUM> may also be fixed. In this case, the driving unit <NUM> is unnecessary.

The imaging unit <NUM> includes an imaging element and an optical system, and forms an image of an object on the imaging element by using, as an imaging center, the intersection of the optical axis of the optical system and the imaging element. Examples of the imaging element are a CMOS (Complementary Metal-Oxide Semiconductor) and a CCD (Charge Coupled Device). The network I/F <NUM> is an interface which the imaging apparatus <NUM> uses to perform data communication with the client apparatus <NUM> across the network <NUM>.

<FIG> is a view showing a hardware configuration example of the client apparatus <NUM>. As shown in <FIG>, the client apparatus <NUM> includes a CPU <NUM>, a main storage device <NUM>, an auxiliary storage device <NUM>, an input I/F <NUM>, an output I/F <NUM>, and a network I/F <NUM>.

The CPU <NUM> executes processing by using computer programs and data stored in the main storage device <NUM>, thereby controlling the operation of the whole client apparatus <NUM>, and executing or controlling each processing (to be described later) to be performed by the client apparatus <NUM>. For example, the CPU <NUM> implements the function of each functional unit shown in <FIG> by executing processing by using the computer programs and data stored in the main storage device <NUM>. Each functional unit shown in <FIG> will be described later.

The main storage device <NUM> is a storage device such as a RAM (Random Access Memory). The main storage device <NUM> has an area for storing computer programs and data loaded from the auxiliary storage device <NUM>, and various kinds of data (for example, captured images) received from the imaging apparatus <NUM> via the network I/F <NUM>. In addition, the main storage device <NUM> has a work area which the CPU <NUM> uses when executing various kinds of processing. Thus, the main storage device <NUM> can provide various areas as needed.

The auxiliary storage device <NUM> is a large-capacity storage device such as a hard disk drive (HDD), a ROM (Read Only Memory), or an SSD (Solid State Drive). The auxiliary storage device <NUM> stores an OS (Operating System), and computer programs and data for causing the CPU <NUM> to execute or control each processing (to be described later) to be executed by the client apparatus <NUM>. The auxiliary storage device <NUM> also stores various kinds of data (for example, captured images) received from the imaging apparatus <NUM> via the network I/F <NUM>, and background images for use in moving object detection. The computer programs and data stored in the auxiliary storage device <NUM> are loaded into the main storage device <NUM> as needed under the control of the CPU <NUM>, and processed by the CPU <NUM>.

The input I/F <NUM> is an interface for connecting the input apparatus <NUM> to the client apparatus <NUM>. The output I/F <NUM> is an interface for connecting the display apparatus <NUM> to the client apparatus <NUM>. The network I/F <NUM> is an interface which the client apparatus <NUM> uses to perform data communication with the imaging apparatus <NUM> across the network <NUM>.

The functional configurations of the imaging apparatus <NUM> and the client apparatus <NUM> will be explained below with reference to <FIG> is a view showing a functional configuration example of the imaging apparatus <NUM> corresponding to the exemplary embodiment. Each functional unit shown in <FIG> can be installed as hardware, and can also be installed as software (a computer program). In the latter case, the computer program is stored in the auxiliary storage device <NUM>. The CPU <NUM> loads the computer program into the main storage device <NUM> as needed and executes the program, thereby implementing the function of the corresponding functional unit.

As shown in <FIG>, the functional configuration of the imaging apparatus <NUM> can include an imaging control unit <NUM>, a signal processing unit <NUM>, a driving control unit <NUM>, and a communication control unit <NUM>. The imaging control unit <NUM> controls an imaging operation of the imaging unit <NUM>, and obtains an image captured by the imaging unit <NUM> as a captured image. The signal processing unit <NUM> performs predetermined image processing on the captured image obtained by the imaging control unit <NUM>. This image processing can include, for example, a developing process and an encoding process. Note that encoding for the captured image obtained by the imaging apparatus <NUM> is not essential, so the signal processing unit <NUM> need not perform any encoding process on the captured image.

The driving control unit <NUM> drives the imaging unit <NUM> based on imaging parameters received from the client apparatus <NUM>, thereby controlling the posture (imaging direction), the viewing angle, and the like of the imaging unit <NUM>. As described earlier, the posture (imaging direction), the viewing angle, and the like of the imaging unit <NUM> may also be fixed. The communication control unit <NUM> transmits the captured image processed by the signal processing unit <NUM> to the client apparatus <NUM> via the network I/F <NUM>. Also, the communication control unit <NUM> receives, via the network I/F <NUM>, control instructions (including imaging parameters) transmitted from the client apparatus <NUM> to the imaging apparatus <NUM>.

A functional configuration example of the client apparatus <NUM> corresponding to this embodiment will be explained below by using a block diagram of <FIG>. Each functional unit shown in <FIG> can be installed as hardware, and can also be installed as software (a computer program). In the latter case, the computer program is stored in the auxiliary storage device <NUM>. The CPU <NUM> loads the computer program into the main storage device <NUM> as needed and executes the program, thereby implementing the function of the corresponding functional unit. The functional configuration of the client apparatus <NUM> can include a setting determination unit <NUM>, a communication control unit <NUM>, an image obtaining unit <NUM>, a detection unit <NUM>, a rendering unit <NUM>, and a display unit <NUM>.

The setting determination unit <NUM> obtains setting information of a fixed region stored in the auxiliary storage device <NUM>, and performs various kinds of determination regarding the setting of the fixed region (or the rendering of a fixed mask). This determination can include determination of whether the fixed region is set (or whether the rendering of the fixed mask is set), and, if this setting is performed, can include determination of the position and size of the fixed region (or the position and size of the rendering of the fixed mask). The determination can also include determination of whether the set fixed region (or the fixed mask to be rendered) is changed, or determination of whether the setting of the fixed region (or the rendering of the fixed mask) is canceled. The fixed region setting method is not limited to a specific setting method. For example, the user can set a fixed region on a captured image by operating the input apparatus <NUM> and store setting information of the fixed region in the auxiliary storage device <NUM> in advance. As another embodiment, assume that an object to be concealed is predetermined in a captured image in a place to which this system is applied. In this case, it is possible to obtain a captured image by operating the system <NUM>, perform a process of detecting an object to be concealed in the captured image, determine a region where the detected object is concealed as a fixed region, and store the information in the auxiliary storage device <NUM>. The setting information of the fixed region can be various kinds of information. For example, the setting information can be information of the position of the upper left corner and the position of the lower right corner of the fixed region, and can also be information of the position of the upper left corner of the fixed region and the vertical and horizontal sizes of the fixed region.

The communication control unit <NUM> obtains an (encoded) image transmitted from the imaging apparatus <NUM> via the network I/F <NUM>. The communication control unit <NUM> also transmits control instructions for the imaging apparatus <NUM> to the imaging apparatus <NUM> via the network I/F <NUM>. The image obtaining unit <NUM> obtains a captured image received from the imaging apparatus <NUM> by the communication control unit <NUM>, and decodes the image. Note that if the imaging apparatus <NUM> transmits a captured image to the client apparatus <NUM> without encoding the image, the image obtaining unit <NUM> obtains the image received from the imaging apparatus <NUM> by the communication control unit <NUM> without decoding the image.

The detection unit <NUM> performs a moving object detection process on the captured image obtained by the image obtaining unit <NUM>, and sets a region associated with the detected moving object as a moving object region. In the moving object detection process of this embodiment, a background difference method can be used to detect a moving object (foreground object) from a captured image. More specifically, a preobtained background image and a captured image are compared, and the difference between the luminance values of pixels in corresponding positions is calculated. The background image can be stored in the auxiliary storage device <NUM>. It is determined that a region including a pixel for which the obtained difference exceeds a threshold is a moving object region. However, the method of implementing the moving object detection process is not limited to the background difference method, and can also be another well-known method. For example, this embodiment uses the difference between the luminance values, but it is also possible to handle RGB values as three-dimensional coordinates, and measure, as the difference, the distance between the RGB value of a background image and the RGB value of a captured image. In addition, the embodiment is not limited to the background difference, and a moving object may also be detected by calculating the moving direction of each coordinate in an image like an optical flow. Assume that no fixed mask is superimposed on a prepared background image in this embodiment.

The rendering unit <NUM> renders a moving object mask in a moving object region on a captured image, and renders a fixed mask in a fixed region on the captured image. Thus, the rendering unit <NUM> executes a process of anonymizing the moving object region and the fixed region. Note that the moving object mask corresponds to an image in which the moving object region is anonymized, and the fixed mask corresponds to an image in which the fixed region is anonymized. The display unit <NUM> causes the display apparatus <NUM> to display the captured image having undergone the rendering process performed by the rendering unit <NUM>. Note that the functional configuration of the client apparatus <NUM> shown in <FIG> will be explained in the second and third embodiments (to be described later).

An example of an image output process in the client apparatus <NUM> corresponding to the exemplary embodiment will be explained below with reference to <FIG>. In particular, a process of outputting an image obtained by superimposing a fixed mask on an image on which a moving object mask is superimposed by the client apparatus <NUM> will be explained. Processing complying with a flowchart shown in <FIG> is processing for one captured image obtained by the imaging apparatus <NUM>. Accordingly, when the imaging apparatus <NUM> transmits a plurality of captured images to the client apparatus <NUM>, the client apparatus <NUM> performs the processing complying with the flowchart shown in <FIG> on each captured image. To simplify the explanation, assume that the imaging apparatus <NUM> transmits an image without encoding it.

In step S400, the image obtaining unit <NUM> causes the main storage device <NUM> to store a captured image received by the communication control unit <NUM> from the imaging apparatus <NUM> via the network I/F <NUM>. An example of the obtained captured image will be explained with reference to <FIG>. An image <NUM> in <FIG> shows an example of a captured image containing workers <NUM>, <NUM>, and <NUM> as moving bodies. An image <NUM> in <FIG> shows an example of a captured image not containing a moving object such as the worker <NUM>.

In step S401, the detection unit <NUM> detects a moving object by the abovementioned background difference method from the captured image stored in the main storage device <NUM> in step S400, and sets a moving object region based on the detected moving object. For example, when the image <NUM> shown in <FIG> is obtained, the detection unit <NUM> detects the workers <NUM>, <NUM>, and <NUM>, so a moving object region is set for each of the detected workers <NUM>, <NUM>, and <NUM>. An image <NUM> shows examples of the set moving object regions. In the image <NUM>, moving object regions <NUM> to <NUM> are set for the positions of the workers <NUM> to <NUM> detected in the image <NUM>. On the other hand, when the captured image <NUM> is obtained, the detection unit <NUM> detects no moving object, so no moving object region is set. An image <NUM> in <FIG> is an example of a background image to be used in moving object detection. Note that the detection unit <NUM> may also detect a moving object in a region other than the fixed region in the captured image. That is, the detection unit <NUM> may also execute no moving object detecting process in the fixed region.

In step S402, the rendering unit <NUM> renders a moving object mask on the moving object region set in the captured image in step S401. If no moving object is detected in the captured image in step S401, no moving object mask is rendered. In step S403, the setting determination unit <NUM> performs determination on the setting of the fixed region. In this embodiment, it is determined that a region <NUM> in the image <NUM> shown in <FIG> is set as the fixed region. As shown in the image <NUM> of <FIG>, the fixed region <NUM> overlaps the moving object region <NUM> set when the worker <NUM> is detected as a moving object. In step S404, the rendering unit <NUM> renders a fixed mask on the fixed region in the captured image. Note that no fixed mask is rendered if it is determined in step S403 that no fixed region is set. In step S405, the captured image obtained by the above processing is output to the display apparatus <NUM> via the output I/F <NUM>.

The output image obtained as a result of the above processing will be explained with reference to <FIG> show examples of captured images to be displayed on the display apparatus <NUM> when the captured images are the images <NUM> and <NUM> shown in <FIG>. A displayed image <NUM> in <FIG> shows an example of a displayed image when the captured image is the image <NUM> shown in <FIG>. In the displayed image <NUM>, moving object masks <NUM> and <NUM> are rendered on the moving object regions <NUM> and <NUM> that are respectively set for the workers <NUM> and <NUM> in the image <NUM>. Also, a fixed mask <NUM> is rendered on the fixed region <NUM> in the image <NUM>.

In this case, the procedure shown in <FIG> first renders a moving object mask on the moving object region <NUM> in step S402, and superimposes the fixed mask <NUM> on the fixed region <NUM> containing the moving object region <NUM> in step S404 after that. Therefore, the moving object mask rendered earlier on the moving object region <NUM> is overwritten by the fixed mask <NUM>, so no moving object mask is visually recognizably rendered in the fixed region <NUM> on which the fixed mask <NUM> of the image <NUM> is rendered. Accordingly, the existence of the worker <NUM> in the fixed region <NUM> cannot be determined from the image <NUM> as the processing result, so the fixed region <NUM> can reliably be concealed.

The moving object mask can be an image corresponding to the shape of the moving object region, and can also be an image having an arbitrary shape including the moving object region. The fixed mask is an image having a shape including the fixed region. However, images applicable to these masks are not limited to specific images. For example, an image painted out by a predetermined pattern, a dot-pattern image, a stripe-pattern image, and an image painted out by a single color may also be used as the moving object mask and the fixed mask. In addition, a captured image obtained at a specific timing and an image pre-registered in the auxiliary storage device <NUM> may also be used as the moving object mask and the fixed mask. Furthermore, the moving object mask and the fixed mask can be rendered on the captured image by using alpha blending of the captured image and the moving object mask and fixed mask. Also, the moving object mask and the fixed mask need not be identical masks, and the rendering methods need not be the same. Note that when the moving object mask is enabled with the fixed mask being set, it is possible to notify the user to delete the setting of the fixed mask.

A displayed image <NUM> in <FIG> shows an example of a displayed image when the captured image is the image <NUM> shown in <FIG>. In the image <NUM>, no moving object mask is rendered because there is no moving object in the image <NUM>, and only a fixed mask <NUM> is rendered on the fixed region <NUM>.

In this embodiment as explained above, when a captured image contains a moving object and a fixed region is set in the image, a fixed mask is superimposed on a moving object mask that is rendered earlier. Consequently, the moving object mask does not visually recognizably remain in a fixed mask region of a displayed image, so the fixed region can reliably be concealed.

In the above-described first embodiment, the method of reliably concealing a fixed region when the fixed region is set has been described. In the first embodiment, no fixed mask is superimposed on a prepared background image, so the display result on the display apparatus <NUM> is unaffected even when the setting of a fixed region is canceled (deleted). However, if the setting of a fixed region is canceled or the number of set fixed regions decreases when a fixed mask is superimposed on a background image, a partial region of a moving object having the same color as that of the fixed mask may be exposed in an image displayed on the display apparatus <NUM>.

Assume that a captured image is an image <NUM> shown in <FIG>, and a background image is a background image <NUM> shown in <FIG>. The image <NUM> contains three workers <NUM> to <NUM> as moving bodies. On the other hand, a fixed mask <NUM> is set in the background image <NUM>. When the background difference method is applied based on these two images, the workers <NUM> to <NUM> are detected as moving bodies. Then, moving object masks are rendered on moving object regions, and a fixed mask is further superimposed after that. In this process, the worker <NUM> is detected as a moving object, but a clothing portion (a part of the moving object) is not detected as a moving object because the color of the clothing of the worker <NUM> is the same as or similar to the color of the fixed mask <NUM>. Consequently, no moving object mask is rendered on the clothing portion. Since, however, a fixed mask is rendered on a fixed region as long as the fixed region is set, the range of a moving object mask rendered on the worker <NUM> in a finally obtained image poses no problem.

When the setting of the fixed region is canceled, however, no fixed mask is superimposed on the moving object mask, so the moving object may not appropriately be concealed if the moving object mask is not appropriately rendered. As an example of a case like this, an image <NUM> in <FIG> shows an image as a result of rendering a moving object mask on the image <NUM>. In the image <NUM>, the color of the clothing of the worker <NUM> is gray, and is the same as the color of the fixed mask <NUM> in the background image <NUM>. In this case, if it is determined that the value of the background difference shows no significant value in the clothing portion, set moving object regions are as shown in an image <NUM>. In the image <NUM>, hatched regions <NUM>, <NUM>, and <NUM> are moving object regions. In the moving object region <NUM>, however, a rectangular region is not entirely hatched, and a clothing portion <NUM> of the worker <NUM> remains as the original clothing color.

A situation like this can occur after the setting of the fixed region is canceled and before the background image is recreated and updated. If the background image includes a fixed mask, the background image must be updated by a new background image from which the fixed mask is removed by canceling the setting of the fixed region. However, this update does not make it in time immediately after the setting is canceled, so the background image in which the fixed mask remains is used.

Accordingly, this embodiment provides a method of solving the problem that a detected moving object cannot appropriately be concealed until a background image is updated. In the following explanation, an explanation of the same contents as the first embodiment will be omitted. In this embodiment, the functional configuration of a client apparatus <NUM> is as shown in <FIG>. The client apparatus <NUM> of this embodiment can include a background creation unit <NUM> and a background determination unit <NUM>, in addition to the functional configuration shown in <FIG>. The background creation unit <NUM> creates and updates a background image. In accordance with the result of determination of whether a fixed region is set performed by a setting determination unit <NUM>, the background creation unit <NUM> creates a background image on which a fixed mask corresponding to a set fixed region is superimposed. Also, the background creation unit <NUM> updates the background image as needed in accordance with the determination result of the setting determination unit <NUM>. The created background image is stored in an auxiliary storage device <NUM>. The background determination unit <NUM> determines whether the background image used in moving object detection contains a fixed mask.

An example of an image output process in the client apparatus <NUM> corresponding to this embodiment will be explained below with reference to <FIG>. In particular, processing which the client apparatus <NUM> performs to output an image on which a moving object mask and a fixed mask are superimposed will be explained. Processing complying with flowcharts shown in <FIG> is processing for one captured image obtained by an imaging apparatus <NUM>. Accordingly, when the imaging apparatus <NUM> transmits a plurality of captured images to the client apparatus <NUM>, the client apparatus <NUM> performs the processing complying with the flowcharts shown in <FIG> on each captured image.

Processes from steps S700 to S702 in <FIG> are the same as the processes from steps S400 to S402 in <FIG>, so a detailed explanation thereof will be omitted. In step S703, the background determination unit <NUM> determines whether a background stored in the auxiliary storage device <NUM> and used in moving object detection contains an image of a fixed mask. This determination can be performed by, for example, determining whether an image of a fixed mask remains in a background image used in moving object detection by analyzing the background image. If the determination result indicates that the background used in moving object detection contains an image of a fixed mask, the process advances to step S704. On the other hand, if the background image does not contain an image of a fixed mask, the process advances to step S705. In step S704, a rendering unit <NUM> renders a fixed mask on a fixed region in a captured image obtained in step S700. In step S705, a display unit <NUM> outputs the captured image on which masks are superimposed by the mask rendering processes in steps S702 and S704 to a display apparatus <NUM> via an output I/F <NUM>. This makes it possible to continue the process of superimposing the fixed mask until the background image containing the fixed mask for which the setting is canceled is updated to a new background image not containing the fixed mask.

The background image updating process will be explained below with reference to <FIG>. This updating process can be executed in parallel with the processing corresponding to the flowchart shown in <FIG>. Assuming that only one fixed region is set, a process of determining whether the state in which one fixed region is set has changed to a state in which the setting is canceled will be explained below. Note that the embodiment is not limited to this, and can include a case in which a plurality of fixed regions are set and the number of fixed regions decreases.

In step S710, the setting determination unit <NUM> obtains setting information of a fixed region from the auxiliary storage device <NUM>. In step S711, the setting determination unit <NUM> determines whether the setting of the fixed region is canceled, based on the obtained setting information. If it is determined that the setting is canceled, the process advances to step S712. On the other hand, if it is determined that the setting is not canceled, the process returns to step S710. In step S712, the background creation unit <NUM> recreates a background image corresponding to the canceled fixed region and not containing a fixed mask. In step S713, the background image is updated by storing the background image recreated by the background creation unit <NUM> in the auxiliary storage device <NUM>.

The case in which the background image is automatically updated has been explained above, but the update may also be executed in accordance with an instruction from the user of the system. Also, in the above-described processing, the background determination unit <NUM> determines whether the background image contains a fixed mask image by analyzing the background image. However, the determination method is not limited to this. For example, the background determination unit <NUM> can also store, in a main storage device <NUM>, the date/time at which the fixed region is canceled, and determine that the background image contains a fixed mask if the period having elapsed from the stored date/time is less than a predetermined period, or determine that the background image contains no fixed mask if the elapsed period is equal to or longer than the predetermined period. The length of the predetermined period can be set based on an average time length during which the background creation unit <NUM> updates a background image on which a fixed mask is superimposed to a background image from which the fixed mask is removed.

Furthermore, the flowchart shown in <FIG> omits a process of determining whether a fixed region is set, which corresponds to step S403 in <FIG>, but this determination process may also be executed. In this case, the process in step S703 can be executed if the determination process determines that no fixed region is set, or determines that the number of set fixed regions has decreased. In addition, if it is determined that a fixed region is set or the number of set fixed regions has not decreased, it is also possible to execute the process in step S704 by skipping the process in step S703.

When the captured image obtained in step S700 is the image <NUM> shown in <FIG>, a captured image output to and displayed by the display apparatus <NUM> in step S705 in the above processing is, for example, an image <NUM> in <FIG>. Moving object masks <NUM> and <NUM> are rendered in the regions of workers <NUM> and <NUM>, and a fixed mask <NUM> is rendered in the fixed region of the image <NUM>. An image obtained by processing like this is the captured image <NUM> in <FIG>.

In this embodiment as explained above, when a fixed mask is rendered in a background image, a moving object detected in a captured image can reliably be concealed even if the setting of the fixed region is canceled or the number of set fixed regions has decreased.

In the second embodiment, the processing when a fixed mask is rendered in a background image and the setting of the fixed region is canceled or the number of set fixed regions has decreased has been explained. In the third embodiment, processing when a new fixed region is set or the number of fixed regions has increased will be explained.

In the following explanation, an explanation of the same contents as the first embodiment will be omitted. In this embodiment, the functional configuration of a client apparatus <NUM> is as shown in <FIG>. Note that the client apparatus <NUM> need not include a background determination unit <NUM> in this embodiment.

In this embodiment, a setting determination unit <NUM> determines whether the setting of a fixed region is updated and a new fixed region is set. A background creation unit <NUM> recreates a background image for use in moving object detection, in accordance with the determination result of the setting determination unit <NUM>. For example, the background creation unit <NUM> can create a background image by using one or a plurality of captured images.

An example of the image output process in the client apparatus <NUM> corresponding to this embodiment will be explained below with reference to <FIG>. In particular, processing which the client apparatus <NUM> performs to output an image on which a moving object mask and a fixed mask are superimposed, while setting a new fixed region or recreating a background image in accordance with an increase in fixed regions, will be explained. Processing complying with a flowchart shown in <FIG> is processing for one captured image obtained by an imaging apparatus <NUM>. Therefore, when the imaging apparatus <NUM> transmits a plurality of captured images to the client apparatus <NUM>, the client apparatus <NUM> performs the processing complying with the flowchart shown in <FIG> on each captured image.

Processes in steps S900 to S904 are the same as the processes in steps S400 to S404 shown in <FIG>, so a detailed explanation thereof will be omitted. In step S905, the setting determination unit <NUM> determines whether a fixed region is newly set, based on the setting of a fixed region obtained in step S903. The setting determination unit <NUM> can perform this determination by holding the fixed region setting information obtained by the last determination in a main storage device <NUM>, and comparing this fixed region setting information with the fixed region setting information obtained in step S903. If this determination result indicates that a fixed region is newly set, the process advances to step S906. On the other hand, if a fixed region is not newly set, the process advances to step S907. In step S906, the background creation unit <NUM> recreates a new background. The background creation unit <NUM> stores the new background image in an auxiliary storage device <NUM>, thereby updating the background image used in moving object detection in step S901. In step S907, a captured image obtained by the rendering processes in steps S902 and S904 is output to a display apparatus <NUM> via an output I/F <NUM>.

In this embodiment as described above, if it is determined that a new fixed region is set or the number of fixed regions has increased, the fixed region can reliably be concealed by discarding a background image used up to that point and recreating a new background image. Note that each exemplary embodiment has been described in detail above, but the embodiment is not limited to the individual embodiments according to the description, and it is also possible to freely combine the above-described embodiments.

Claim 1:
An image processing apparatus (<NUM>) comprising:
extraction means (<NUM>, <NUM>) configured to extract a region of a foreground object in a captured image obtained by an imaging unit;
superimposing means (<NUM>, <NUM>) configured to superimpose a first image (<NUM>, <NUM>), formed by anonymizing the region extracted by the extraction means, on a predetermined image, and to superimpose a second image (<NUM>, <NUM>), formed by anonymizing a region designated by a user on the captured image obtained by the imaging unit, on the predetermined image; and
display control means (<NUM>, <NUM>) configured to cause a display unit (<NUM>) to display an output image, wherein
the display control means (<NUM>, <NUM>) is configured to cause the display unit (<NUM>) to display an output image, in which the first image and the second image are superimposed on the predetermined image by the superimposing means (<NUM>, <NUM>), wherein an appearance of the second image is different from an appearance of the first image,
the extraction means (<NUM>, <NUM>) is configured to extract a region of a foreground object from at least a region other than the designated region in the captured image obtained by the imaging unit, and
even if the designated region contains the foreground object, the superimposing means (<NUM>, <NUM>) is configured to superimpose the second image on the predetermined image after the first image is superimposed on the predetermined image so that the first image formed by anonymizing the region of the foreground object in the designated region is not displayed on the output image displayed on the display unit by the display control means (<NUM>, <NUM>).