INFORMATION PROCESSING APPARATUS, IMAGE CAPTURING APPARATUS, INFORMATION PROCESSING METHOD, AND NON-TRANSITORY COMPUTER READABLE STORAGE MEDIUM

An obtaining unit is configured to obtain image capturing direction information indicating an image capturing direction of a first image capturing part that can change the image capturing direction. A determining unit is configured to determine, in accordance with the image capturing direction of the first image capturing part, a second image capturing part having an image capturing direction closest to the image capturing direction of the first image capturing part in an image capturing part set including a plurality of image capturing parts having a different image capturing direction. A generating unit is configured to generate a combined image such that an image captured by the second image capturing part is arranged at a predetermined position in the combined image by combining images captured by the respective image capturing parts in the image capturing part set.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an information processing apparatus, an image capturing apparatus, an information processing method, and a non-transitory computer readable storage medium.

Description of the Related Art

Network camera systems have been used for monitoring intruders or the like into restricted areas or the like in, for example, public buildings or places, banks, stores, such as supermarkets, dams, bases, or airfields. A multi-eye camera with an omnidirectional camera and a PTZ camera combined, for example, has been known as a camera used in the network camera systems. The omnidirectional camera refers to a camera that can capture a 360° omnidirectional image with a plurality of image sensors installed at a predetermined position of a casing of the camera. The omnidirectional camera refers to a camera in which an omnidirectional image is obtained by combining images in a predetermined image capturing range captured by each of the image sensors.

The PTZ camera has a pan, tilt, and zoom mechanisms, enabling its image capturing range to be changed. A multi-eye camera with the omnidirectional camera and the PTZ camera combined can output a video captured by the omnidirectional camera and a video in a specific range captured by the PTZ camera. For example, Patent Document 1 discloses a technique for facilitating correspondence between the position of each camera of the multi-eye camera and the position of a displayed image in accordance with the orientation (angle) of the multi-eye camera at the time of image capturing when the captured image is rotated (see Japanese Patent Laid-Open No. 2013-179397).

SUMMARY OF THE INVENTION

The present invention in its one aspect provides an information processing apparatus comprising an obtaining unit configured to obtain an image capturing direction information indicating an image capturing direction of a first image capturing part that can change the image capturing direction, a determining unit configured to determine, in accordance with the image capturing direction of the first image capturing part, a second image capturing part having an image capturing direction closest to the image capturing direction of the first image capturing part in an image capturing part set including a plurality of image capturing parts having a different image capturing direction, a generating unit configured to generate a combined image such that an image captured by the second image capturing part determined by the determining unit is arranged at a predetermined position in the combined image by combining images captured by the respective image capturing parts in the image capturing part set, and an outputting unit configured to output the combined image generated by the generating unit.

The present invention in its one aspect provides an information processing apparatus comprising an obtaining unit configured to obtain an image capturing position information indicating an image capturing position of a first image capturing part that can change the image capturing position, a determining unit configured to determine, in accordance with the image capturing position information of the first image capturing part, coordinates corresponding to the image capturing position of the first image capturing part on an omnidirectional image captured by an omnidirectional image capturing part that can perform omnidirectional image capturing, a generating unit configured to correct the omnidirectional image to generate an omnidirectional image such that the coordinates is positioned at specified coordinates prespecified on the omnidirectional image, and an outputting unit configured to output the omnidirectional image generated by the generating unit.

The present invention in its one aspect provides an information processing apparatus comprising an obtaining unit configured to obtain an image capturing position information indicating an image capturing position of a first image capturing part that can change the image capturing position, a generating unit configured to generate an omnidirectional image in accordance with an image capturing result from an omnidirectional image capturing part that can perform omnidirectional image capturing such that coordinates corresponding to the image capturing position of the first image capturing part on the omnidirectional image are positioned at specified coordinates prespecified on the omnidirectional image, and an outputting unit configured to output the omnidirectional image generated by the generating unit.

The present invention in its one aspect provides an information processing method comprising obtaining an image capturing direction information indicating an image capturing direction of a first image capturing part that can change the image capturing direction, determining, in accordance with the image capturing direction of the first image capturing part, a second image capturing part having an image capturing direction closest to the image capturing direction of the first image capturing part in an image capturing part set including a plurality of image capturing parts having a different image capturing direction, generating a combined image such that an image captured by the second image capturing part determined by the determining is arranged at a predetermined position in the combined image by combining images captured by the respective image capturing parts in the image capturing part set, and outputting the combined image generated by the generating.

The present invention in its one aspect provides a non-transitory computer readable storage medium storing a program that, when executed by a computer, causes the computer to perform an information processing method comprising obtaining an image capturing direction information indicating an image capturing direction of a first image capturing part that can change the image capturing direction, determining, in accordance with the image capturing direction of the first image capturing part, a second image capturing part having an image capturing direction closest to the image capturing direction of the first image capturing part in an image capturing part set including a plurality of image capturing parts having a different image capturing direction, generating a combined image such that an image captured by the second image capturing part determined by the determining is arranged at a predetermined position in the combined image by combining images captured by the respective image capturing parts in the image capturing part set, and outputting the combined image generated by the generating.

DESCRIPTION OF THE EMBODIMENTS

An embodiment of the present invention can provide a technique enabling easy recognition of a position of a captured image in an omnidirectional image.

First Embodiment

In a first embodiment, a network camera including an image capturing unit is described as an example. The present embodiment is also applicable to image capturing purposes using an image capturing unit other than the network camera. For example, the present embodiment is also applicable to an image capturing unit for capturing videos or a movies for broadcasting or videos for personal purposes.

FIG.1is a block diagram illustrating a configuration of an information processing system according to the first embodiment. InFIG.1, an information processing system10includes an image capturing apparatus100, an information processing apparatus110, and a network120. The image capturing apparatus100is connected to the information processing apparatus110through the network120. The image capturing apparatus100is an apparatus that captures an image of a monitored region, processes the obtained captured image, and transmits the processed captured image to the information processing apparatus110. The information processing apparatus110is an apparatus that receives the captured image from the image capturing apparatus100, processes, and displays the captured image. The captured image in the present embodiment is a video but may also be a still image. In addition, the image capturing apparatus100according to the present embodiment includes an information processing apparatus (not illustrated) for executing image processing on the obtained image.

The image capturing apparatus100includes a PTZ camera part101, fixed camera parts102, a combining unit103, a communication unit104, a CPU105, a ROM106, and a RAM107. Note that the number of fixed camera parts102provided in the image capturing apparatus100is not limited to four as illustrated inFIG.1but may be, for example, two or more. The PTZ camera part101is an image capturing apparatus that includes a zoom lens and an image sensor and can control pan, tilt, and zoom (hereinafter referred to as PTZ). Thus, the PTZ camera part101can change an image capturing range. The PTZ camera part101controls the zoom lens and a drive motor in accordance with a control instruction related to the image capturing range received from the information processing apparatus110. Here, the PTZ camera part101is defined as an image capturing part that can change the image capturing range.

On the other hand, the fixed camera parts102are image capturing apparatuses each including a fixed lens and an image sensor. To obtain an omnidirectional image with a plurality of images captured by the respective fixed camera parts102combined, the fixed camera parts102are each installed at a predetermined position of the image capturing apparatus100. In the present embodiment, the omnidirectional image is a panoramic image. Each of the fixed camera parts102captures an image of the corresponding image capturing range from the predetermined position. In the present embodiment, each of the fixed camera parts102is not limited to being fixed to the predetermined position, but its predetermined position can be changed to change the image capturing range. For example, each of the fixed camera parts102may control the PTZ, as with the case of the PTZ camera part101. The image sensors of the PTZ camera part101and the fixed camera parts102are elements that convert an image into an electrical signal in response to light and are, for example, CCD sensors and CMOS sensors.

The combining unit103combines a plurality of captured images captured by the respective fixed camera parts102into a single panoramic image. The combining unit103combines the plurality of captured images captured by the respective fixed camera parts102by arranging each of the plurality of captured images in a predetermined region in the panoramic image. The image capturing ranges of the respective fixed camera parts102include different image capturing ranges that do not overlap each other, but the present embodiment also includes a case where part of the image capturing ranges overlap part of other image capturing ranges.

The communication unit104transmits and receives captured images and information to and from the information processing apparatus110over the network120. For example, the communication unit104compresses the captured image and outputs the captured image to the information processing apparatus110over the network120. The CPU105implements various types of processing by reading a control program stored in the ROM106and executing the control program. The RAM107is used as a temporary storage area, such as a main memory for the CPU105or a work area. Functions and processing of the image capturing apparatus100described below are implemented with the CPU105reading a program stored in the ROM106and executing the program. The CPU105may read a program stored in a recording medium, such as a USB or an SD card, instead of the ROM106or the like.

The information processing apparatus110is an apparatus for processing various types of data and is an information processing apparatus such as a PC, a smartphone, a tablet, or the like. The information processing apparatus110includes a communication unit111, a display unit112, an input unit113, a CPU114, a ROM115, a RAM116, and a storage unit117. The communication unit111transmits and receives information to and from the image capturing apparatus100over the network120. The display unit112is a device that displays various types of information processed by the CPU114, and is, for example, a liquid crystal display (LCD) or an organic EL display (OLED). The display unit112can display the captured image captured by the PTZ camera part101and an omnidirectional image with the images captured by the respective fixed camera parts102unified. The input unit113includes a receiving unit for receiving various operations by the user, and is, for example, a keyboard, a mouse, and a joystick. The user can input an operation to the input unit113to perform PTZ control on the PTZ camera part101for example. The CPU114reads a control program stored in the ROM115and executes the control program to implement various types of processing.

The RAM116is used as a temporary storage area such as a main memory or a work area, by the CPU114. The storage unit117is a storage device that stores various types of data and program and is, for example, an HDD or an SSD. Functions and processing of the information processing apparatus110described below are implemented with the CPU114reading a program stored in the ROM115or the storage unit117and executing the program. The CPU114may read a program stored in a recording medium, such as a USB or an SD card, instead of the ROM115or the like.

FIG.2is a diagram illustrating a configuration of the information processing system according to the first embodiment. InFIG.2, a side view200illustrates the image capturing apparatus100as viewed from a side surface, and a bottom view210illustrates the image capturing apparatus100as viewed from a bottom surface. In the side view200ofFIG.2, the image capturing apparatus100is installed on a ceiling of a structure, for example. In the bottom view210ofFIG.2, the PTZ camera part101is installed at the center of the fixed camera parts102to be surrounded by the same. In the present embodiment, each of the fixed camera parts102is also referred to as an image capturing part set including a plurality of image capturing parts. The fixed camera parts102are arranged at a constant interval in a circumferential direction of the PTZ camera part101, for capturing an omnidirectional image using the fixed camera parts102. The constant interval is 90° in the present embodiment as illustrated inFIG.2but is not limited to this, and an appropriate interval may be set in accordance with the number of fixed camera parts102.

FIG.3is a block diagram illustrating a functional configuration of the image capturing apparatus according to the first embodiment. InFIG.3, the image capturing apparatus100includes an A/D conversion unit301, a development processing unit302, a data forming unit303, a communication processing unit304, a camera control unit305, and a position processing unit306. The A/D conversion unit301performs analog-to-digital conversion on a signal corresponding to light received by the image sensors of the PTZ camera part101and the fixed camera parts102, to obtain a captured image. The development processing unit302converts the captured image obtained by the A/D conversion unit301by using a predetermined method. The data forming unit303generates the captured image after the development processing and a panoramic image using a plurality of captured images and transmits these images to the communication processing unit304. The communication processing unit304transmits, over the network120, the images to the information processing apparatus110.

The camera control unit305receives a camera control instruction input through a user operation via the communication processing unit304. The camera control unit305controls the image capturing by the PTZ camera part101and the fixed camera parts102based on the control instruction. The camera control unit305also performs PTZ control on the PTZ camera part101. The position processing unit306obtains image capturing direction information of the PTZ camera part101and the fixed camera parts102. The image capturing direction information is assumed to be an angle in a pan direction (pan angle) and an angle in a tilt direction (tilt angle) of the PTZ camera part101and the fixed camera parts102. The position processing unit306also obtains image capturing direction information of each of the PTZ camera part101and the fixed camera parts102after the PTZ control is performed. The data forming unit303combines a panoramic image and an image captured by the fixed camera part102whose image capturing direction is closest to the image capturing direction of the PTZ camera part101obtained from the position processing unit306arranged in a center of a panoramic image into a panoramic image. Note that the position where the image captured by the fixed camera part102described above is arranged on the panoramic image is not limited to the center, and may be a position selected by the user.

FIG.4is a diagram illustrating an example of a display screen according to the first embodiment. InFIG.4, a display screen400is a screen displayed by the display unit112and displays a panoramic image401and a captured image402. The panoramic image401is a panoramic image with a plurality of captured images captured by the respective fixed camera parts102combined (combined image). The captured image402is a captured image captured by the PTZ camera part101. The captured image402is displayed directly below the center of the panoramic image401, but this is not limited. The display positions of the panoramic image401and the captured image402on the display screen400may be swapped.

FIG.5Ais a bottom view of a fixed camera part according to the first embodiment. InFIG.5A, a fixed camera part502A, a fixed camera part502B, a fixed camera part502C, and a fixed camera part502D represent the fixed camera parts102. Image capturing ranges510A to510D represent the image capturing range of the PTZ camera part101. The image capturing ranges510A to510D are respectively indicated by θAto θDinFIG.5A. Two crossing broken lines indicate are boundaries provided for describing each of the image capturing ranges510A to510D. Each of θAto θDin the present embodiment corresponds to 90° as a result of segmenting the movable range 360° of the PTZ camera part101in the pan direction in four, but this is not limited.

FIG.5Billustrates a table for determining a fixed camera part whose image capturing direction is closest to the image capturing direction of the PTZ camera part101. In the present embodiment, the table is also referred to as correspondence information. This table500includes an identification530and an image capturing range520. The identification530indicates each of the fixed camera parts502A to502D. The image capturing range520indicates OA to GD that are image capturing ranges of the PTZ camera part101respectively corresponding to the fixed camera parts502A to502D. In the present embodiment, the table is stored in advance in at least any of the ROM106of the image capturing apparatus100, or the ROM115or the storage unit117of the information processing apparatus. The position processing unit306obtains the image capturing direction (pan angle) of the PTZ camera part101and determines which one in the image capturing range520in the table500corresponds to the image capturing direction obtained. When determining that the image capturing direction (pan angle) obtained corresponds to GA in, for example, the image capturing range520, the position processing unit306determines that the fixed camera part whose image capturing direction is closest to the image capturing direction of the PTZ camera part101is the fixed camera part502A. The position processing unit306can determine the fixed camera part whose image capturing direction is closest to the image capturing direction of the PTZ camera part101by referring to the table500. Through calculation based on the image capturing direction information of the PTZ camera part101, the position processing unit306may determine the fixed camera part with the closest image capturing direction, without referring to the table500. The image capturing direction is specified by the pan direction of the PTZ camera part101in the present embodiment, but this is not limited. For example, when PTZ control of the fixed camera parts502A to502D can be performed, the range of the image capturing direction can be specified with the tilt direction, the pan direction, and a zoom ratio considered.

FIG.6is a flowchart illustrating panoramic image output processing according to the first embodiment. The present embodiment describes panoramic image output processing by using an example where the PTZ camera part101captures images while changing the image capturing direction in the pan direction, and the fixed camera parts102each capture an image at a fixed image capturing position. The panoramic image output processing according to the present embodiment will be described below with reference toFIGS.4,5A, and5B.

In S601inFIG.6, the position processing unit306obtains the image capturing direction of the PTZ camera part101controlled by the camera control unit305. The position processing unit306refers to the association between the physical position of each of the fixed camera parts102and the image capturing range520in the image capturing direction of the PTZ camera part101based on the table500inFIG.5Bstored in advance in the ROM106. Thus, the position processing unit306determines the fixed camera part102whose image capturing direction is closest to the image capturing direction of the PTZ camera part101. The position processing unit306refers to the table500to determine the fixed camera part102whose image capturing direction is closest to the image capturing direction of the PTZ camera part101, but this is not limited.

In S602, the data forming unit303generates a panoramic image such that the image captured by the fixed camera part102whose image capturing direction is closest to the image capturing direction of the PTZ camera part101is arranged in the center of the panoramic image. With the captured image captured by the PTZ camera part101displayed in the center of the panoramic image, the user can easily identify this captured image in the panoramic image. In S603, the communication processing unit304outputs the panoramic image generated by the data forming unit303to the display unit112. In the present embodiment, the panoramic image is also referred to as an output result. The display unit112displays the panoramic image and the captured image on the display screen400inFIG.4. Note that the display unit112may display at least one of the panoramic image and the captured image. In S604, the position processing unit306determines whether the PTZ camera part101is changing the image capturing direction. When the position processing unit306determines that the PTZ camera part101is not changing the image capturing direction (Yes in S604), the processing ends. When the position processing unit306determines that the PTZ camera part101is changing the image capturing direction (No in S604), the processing returns to S601, and the position processing unit306obtains the image capturing direction of the PTZ camera part101after in the image capturing direction is changed.

The present embodiment generates the panoramic image with the captured image captured by the fixed camera part102whose image capturing direction is closest to the image capturing direction of the PTZ camera part101positioned in the center of the panoramic image, but this is not limited. The captured image captured by the fixed camera part102described above may be displayed at a position in the panoramic image designated by the user in advance. With the present embodiment, the image capturing direction of the PTZ camera part101can be easily identified from the panoramic image, whereby a load on the user for monitoring an intruder or the like can be reduced.

As described above, the first embodiment enables the panoramic image to be generated such that the image captured by the fixed camera whose image capturing direction is closest to the image capturing direction of the PTZ camera that can control the image capturing direction is arranged at a predetermined position on the panoramic image. This enables the first embodiment to easily recognize where the captured image captured by the PTZ camera is displayed on the panoramic image.

Second Embodiment

The first embodiment generates a panoramic image with the image capturing direction of the PTZ camera positioned in the center of the panoramic image by using a multi-eye camera including the PTZ camera and the plurality of fixed cameras. On the other hand, a second embodiment uses a multi-eye camera including a PTZ camera and a fish-eye camera and thus does not require the processing of combining a plurality of images into a panoramic image. Further, the second embodiment generates a fish-eye image with coordinates corresponding to the image capturing direction of the PTZ camera positioned at coordinates in at upper center of the fish-eye image. The second embodiment describes fish-eye image output processing by using the multi-eye camera including the PTZ camera and the fish-eye camera. In the second embodiment, a difference from the first embodiment will be described.

FIG.7is a block diagram illustrating a configuration of an information processing system according to the second embodiment. InFIG.7, the information processing system10includes the image capturing apparatus100, the information processing apparatus110, and the network120. An image capturing apparatus100includes a PTZ camera part101, a fish-eye camera part108, and a communication unit104. The combining unit103can combine at least one of, for example, a panoramic image or a segment image obtained by dewarping a fish-eye image. Alternatively, the combining unit103can combine a fish-eye image by rotating a fish-eye image in a clockwise or counterclockwise direction, to position the coordinates corresponding to the image capturing direction of the PTZ camera part101at specified coordinates on the fish-eye image. The predetermined position is an upper center on the fish-eye image in the present embodiment, but this is not limited, and any coordinates on the fish-eye image may be set by a user selection, for example. The fish-eye camera part108is an image capturing part that includes an image sensor such as a CMOS sensor and a fish-eye lens, and can capture images in a 360° image capturing range. In the present embodiment, the data forming unit303inFIG.3executes the following processing based on the image capturing direction of the PTZ camera part101obtained from the position processing unit306and a result of image capturing by the fish-eye camera part108. The data forming unit303generates a fish-eye image with the image capturing direction of the PTZ camera part101positioned in the upper center on the fish-eye image.

FIGS.8A and8Bare diagrams each illustrating an example of a display screen according to the second embodiment. InFIG.8A, a display screen800includes a fish-eye image801and a captured image802. The fish-eye image801is a fish-eye image that is an omnidirectional image captured by the fish-eye camera part108and is displayed as a circular image. The fish-eye image801displays specified coordinates810and a target820. The target820is displayed in a lower center in the fish-eye image801, while being in an upside down standing state. On the other hand, the captured image802is a rectangular image captured by the PTZ camera part101and displays the target820in a standing straight state. As described above, the fish-eye image801is different from the captured image802in the displayed state and position of the target820. The user needs to search the captured image802for the target820in the fish-eye image801, and thus cannot quickly identify the target820. The present embodiment performs image processing described below on the fish-eye image801to easily identify the target820.

The specified coordinates810are coordinates specified to arrange the image capturing direction of the PTZ camera part101at a predetermined position on the fish-eye image801. Note that the specified coordinates810may be set on an omnidirectional image other than the fish-eye image. The predetermined position in the present embodiment is the upper center of the fish-eye image801to enable the user to easily identify the captured image802. The predetermined position is not limited to the upper center of the fish-eye image801, and any position on the fish-eye image801may be set in advance by the user selection. As illustrated inFIG.8A, the specified coordinates810are displayed with, for example, a star mark and is expressed by a two-dimensional coordinate system (X, Y). A screen840inFIG.8Bdisplays a fish-eye image811as a result of correcting the display screen800inFIG.8A. The data forming unit303executes image processing of rotating the fish-eye image801clockwise or counterclockwise to position the target820, on the fish-eye image801, at the specified coordinates810. This enables the display unit112to display a target830at the upper center of the fish-eye image811, enabling the user to easily identify the target830on the fish-eye image811.

FIG.9is a flowchart illustrating fish-eye image output processing according to the second embodiment. In the present embodiment, the PTZ camera part101captures an image of the target820inFIGS.8A and8B, and the fish-eye camera part108captures an image with the target820being within the3600image capturing range. In the present embodiment, the coordinates, on the fish-eye image, corresponding to the image capturing direction of the PTZ camera part101are set to correspond to the specified coordinates on the fish-eye image captured by the fish-eye camera part108. Specifically, in the present embodiment, the fish-eye image is rotated to position the image capturing direction of the PTZ camera part101at the upper center of the fish-eye image. The fish-eye image output processing according to the present embodiment will be described below with reference toFIG.9.

In S901inFIG.9, the position processing unit306uses a table (not illustrated) that is stored in advance in the ROM106and indicates association between the image capturing direction of the PTZ camera part101and coordinates on the fish-eye image acquired by the fish-eye camera part108. The position processing unit306determines which coordinates on the fish-eye image801in the table correspond to the image capturing direction of the PTZ camera part101received from the camera control unit305. Next, the position processing unit306determines whether to correct the fish-eye image801, based on whether the coordinates (hereinafter, referred to as detected coordinates) on the fish-eye image801corresponding to the image capturing direction of the PTZ camera part101correspond to the specified coordinates810corresponding to the upper center on the fish-eye image801. Here, the detected coordinates are defined as image capturing position information. The present embodiment includes obtaining results of the two determinations by the position processing unit306described above. Note that the coordinates on the fish-eye image801corresponding to the image capturing direction of the PTZ camera part101and the specified coordinates810are expressed by a two-dimensional coordinate system (X, Y), for example.

When it is determined that the detected coordinates indicating the image capturing direction of the PTZ camera part101do not match the specified coordinates810, the data forming unit303generates the fish-eye image811by correcting the fish-eye image801to position the detected coordinates at the specified coordinates810in S902. The data forming unit303may perform correction of rotating the fish-eye image based on a difference between the detected coordinates and the specified coordinates810. When it is determined that the detected coordinates match the specified coordinates810, the data forming unit303does not correct the fish-eye image801. In S903, the communication processing unit304outputs the fish-eye image811generated by the data forming unit303in S902or the fish-eye image801. In S904, the position processing unit306determines whether the PTZ camera part101is changing the image capturing direction. When the position processing unit306determines that the PTZ camera part101is not changing the image capturing direction (Yes in S904), the processing ends. When the position processing unit306determines that the PTZ camera part101is changing the image capturing direction (No in S904), the processing returns to S901, and the position processing unit306obtains the image capturing direction of the PTZ camera part101after the image capturing direction is changed.

The present embodiment generates a fish-eye image corresponding to an image capturing direction of a PTZ camera in a multi-eye camera including the PTZ camera and a fish-eye camera, but this is not limited. The present embodiment may generate any of, for example, a panoramic image or a four-segmented image obtained by dewarping a fish-eye image and output such images and the like.

As described above, according to the second embodiment, a fish-eye image corresponding to an image capturing direction of a PTZ camera can be generated by using a multi-eye camera including the PTZ camera and a fish-eye camera. This enables the second embodiment to easily recognize where the captured image captured by the PTZ camera is displayed on the fish-eye image captured by the fish-eye camera.

Other Embodiments

This application claims the benefit of Japanese Patent Application No. 2021-090589, filed May 28, 2021, which is hereby incorporated by reference herein in its entirety.