Imaging device, imaging method, display device, and display method that capture a particular capturing region corresponding to a selected cutout image

An imaging device includes a first imaging unit that captures a first capturing region, a second imaging unit that captures a second capturing region of the first capturing region, a holding unit that holds position information corresponding to the second capturing region captured by the second imaging unit, an output unit that outputs a cutout image of the second capturing region corresponding to the position information held by the holding unit out of a captured image captured by the first imaging unit, and a change unit that, when one of a plurality of cutout images output by the output unit is selected, controls the second imaging unit so as to capture the second capturing region corresponding to the selected cutout image.

This application claims the benefit of Japanese Patent Application No. 2017-217225 filed Nov. 10, 2017, which is hereby incorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an imaging device, an imaging method, a display device, and a display method.

Description of the Related Art

Conventionally, as one of schemes by which a user changes a capturing region of an imaging device, a scheme of using an image captured by the imaging device in the past is used. For example, Japanese Patent Application Laid-Open No. 2007-43505 discloses a technology in which a panoramic image created by using images captured in the past is displayed on a display unit, and once the user selects a particular region in the displayed panoramic image, the capturing region is changed to the selected region. Japanese Patent Application Laid-Open No. 2014-236313 discloses that, when the same region is captured by using the same camera, a pan angle, a tilt angle, a zoom position, or the like, is acquired from the past captured image, and capturing is performed at the same pan angle, or the like, as the past captured image.

In Japanese Patent Application Laid-Open No. 2007-43505 and Japanese Patent Application Laid-Open No. 2014-236313, since images captured in the past are used, these techniques are useful in terms of a reduction in load or power, or useful in an application of comparing the statuses of the same capturing region of a certain past time and the current time. When an object to be paid attention to is a moving object, however, there is a demand for always having the latest state of an image to be used in order to change a capturing region after confirming that the moving object appears in a particular region.

But, Japanese Patent Application Laid-Open No. 2007-43505 and Japanese Patent Application Laid-Open No. 2014-236313 do not support such a demand. Accordingly, the object of the present invention is to enable a change of a capturing region of an imaging device while allowing the user to confirm the current state of a region captured in the past.

SUMMARY OF THE INVENTION

An imaging device according to the present invention comprises a first imaging unit that captures a first capturing region, a second imaging unit that captures a second capturing region of the first capturing region, a holding unit that holds position information corresponding to the second capturing region captured by the second imaging unit, an output unit that outputs a cutout image of the second capturing region corresponding to the position information held by the holding unit out of a captured image captured by the first imaging unit, and a change unit that, when one of a plurality of cutout images output by the output unit is selected, controls the second imaging unit so as to capture the second capturing region corresponding to the selected cutout image.

DESCRIPTION OF THE EMBODIMENTS

Imaging System

An imaging system10according to a first embodiment of the present invention will be described with reference toFIG. 1. The imaging system10has an imaging device100and a UI display unit140connected to the imaging device100via an IP network130. The imaging device100is installed to a ceiling or a wall in a room. The UI display unit140is a display device (a client device), which is a general purpose computer having a CPU, a ROM, and a RAM. The imaging device100has a wide angle camera110, a pan-tilt-zoom camera (PTZ camera)120, a system control unit124, and a storage unit125. The imaging device100of the present embodiment is an imaging device in which the wide angle camera110and the PTZ camera120are integrated. The wide angle camera110and the PTZ camera120are accommodated in the same casing. The imaging device100is connected via a wire or wirelessly to the UI display unit140via the IP network130in a state where the imaging device100and the UI display unit140can communicate with each other. The term UI is an abbreviation for User Interface.

The wide angle camera110is formed of a capturing optical system111formed of a wide angle lens, or the like, for acquiring an image of a wide view angle, an image pickup device unit112, and an image processing unit113. The wide angle camera110of the present embodiment does not have means for changing a capturing direction (pan angle, tilt angle) and a zoom position (view angle). Further, the wide angle camera110of the present embodiment is formed of a single camera. The wide angle camera110has a view angle wider than that of the PTZ camera120. The wide angle camera110captures a predetermined capturing region.

The PTZ camera120is formed of a capturing optical system121formed of a zoom lens, a focus lens, an aperture mechanism, or the like, an image pickup device unit122, an image processing unit123, a drive control unit126, a zoom motor127, a pan motor128, and a tilt motor129. The PTZ camera120can adjust focus or aperture by driving the capturing optical system121by using the drive control unit126. Further, the PTZ camera120can change the capturing direction, the view angle, or the like, of the PTZ camera120by driving the zoom motor127, the pan motor128, and the tilt motor129. That is, the PTZ camera120can change the capturing region. The PTZ camera120stores, in the storage unit125, information on the pan angle, the tilt angle, and the zoom position (view angle) obtained before the capturing region is changed. While the view angle of the PTZ camera120is narrower than the view angle of the wide angle camera110, the PTZ camera120can change the capturing region over the entire view angle range of the wide angle camera110by driving the motors127,128, and129. The PTZ camera120captures a capturing region within a predetermined capturing region captured by the wide angle camera110.

The system control unit124performs a process such as camera control, encoding (coding) of an image, generation of a display image, storage to the storage unit125, a network delivery, cutout of an image (digital PTZ) of the wide angle camera110, or the like. The camera control performed by the system control unit124includes control of the wide angle camera110and control of the PTZ camera120. When an image from the wide angle camera110is cut out, a certain region is designated from a captured image from the wide angle camera110displayed on the UI display unit140, and thereby a cutout process can be performed. The system control unit124outputs the cut out image as a cutout image. The system control unit124is formed of a CPU, for example. The system control unit124transmits a captured image captured by the wide angle camera110to the UI display unit140.

The storage unit125is formed of a ROM and a RAM, for example. The ROM stores a program, or the like, required for the system control unit (CPU)124to perform a process, for example. The CPU performs a process based on the program stored in the ROM, and thereby a function or a process (FIG. 3), or the like, of the imaging device100is implemented. Note that this program, or the like, may be stored in an external memory or a removable storage medium (not illustrated). The RAM is a memory device used for expanding a program read out from the ROM by the CPU and performing the process thereof. Further, the RAM is also used as a storage region used for temporarily storing data to be subjected to various processing as a temporal storage memory. Further, the storage unit125can hold a plurality of pieces of pan-tilt-zoom (PTZ) position information viewed through the PTZ camera120in the past. For the position viewed by the PTZ camera120, the system control unit124extracts a PTZ position at which there has been no movement over a predetermined time and acquires position information (pan angle, tilt angle, zoom magnification) at this time from the drive control unit126.

The IP network130transmits and receives data based on an Internet protocol. The UI display unit140is a personal computer or a tablet terminal (or a display unit thereof), for example, and has an input/output function or a communication function. The UI display unit140transmits various commands such as an instruction of change of a capturing parameter or control of each of the motors127to129, start of a video streaming, or the like, to the wide angle camera110and the PTZ camera120. This command transmission enables the UI display unit140to control the wide angle camera110and the PTZ camera120. The imaging device100transmits a response or a video streaming to the UI display unit140in response to the command described above.

Respective function modules (the image processing unit113, the image processing unit123, the system control unit124, the drive control unit126) illustrated inFIG. 1are implemented by software or hardware. When the functions are implemented by software, a program for providing the functions of respective function modules described above is stored in the ROM of the storage unit125. The stored program is read to the RAM of the storage unit125as appropriate and executed by the CPU of the system control unit124, and thereby the functions of the above function modules are implemented.

On the other hand, when the functions are implemented by hardware, a predetermined compiler may be used to automatically generate a dedicated circuit on an FPGA from a program used for implementing the functions of respective function modules, for example. The term FPGA is an abbreviation for Field Programmable Gate Array.

Further, a gate array circuit may be configured in the same manner as an FPGA and implemented as hardware. Further, an Application Specific Integrated Circuit (ASIC) may be used for implementation. Note that the configuration of the function modules illustrated inFIG. 1is an example, and a plurality of function modules may form a single function module or any of the function modules may be divided into modules that perform a plurality of functions.

UI Display Unit

FIG. 2illustrates images201,202,204, and205generated by the system control unit124and displayed on the UI display unit140. A display screen200of the UI display unit140is a UI used by the user. The UI display unit140has a display unit that displays the images201,202,204, and205. The image201is a real-time image of a region captured by the wide angle camera110. The image202is a real-time image of a region (capturing region208) captured by the PTZ camera120.

In the present embodiment, the capturing region of the wide angle camera110and the pan angle, the tilt angle, and the zoom (telephoto, wide angle) position of the PTZ camera120are linked (related, associated) with each other. For example, once the user designates a region203within the image201from the UI display unit140, the system control unit124drives each of the motors127to129so that the PTZ camera120captures the region203based on the information on the pan angle, the tilt angle, and the zoom position linked with the region203. This motor driving enables a change of the capturing region of the PTZ camera120.

The image204is a thumbnail image in which the capturing region206captured at the immediately previous attitude of the PTZ camera120is cut out from an image currently captured by the wide angle camera110and is displayed. The image205is a thumbnail image in which the capturing region207captured at the second previous attitude of the PTZ camera120is cut out from an image currently captured by the wide angle camera110and is displayed. Information (capturing information) on the pan angle, the tilt angle, and the zoom position of the PTZ camera120obtained when the capturing region206is captured is linked with the thumbnail image204and stored in the storage unit125.

Information on the pan angle, the tilt angle, and the zoom position of the PTZ camera120, obtained when the capturing region207is captured, is linked with the thumbnail image205and stored in the storage unit125. Each of the capturing regions (capturing ranges)203and206to208indicates a region to be cut out from the image201and therefore may be referred to as a cutout region. Further, the thumbnail images204and205are cutout images. Note that the system control unit124may display a guide corresponding to each cutout image so as to overlap with the image201. In this case, the color or the display form of the line of the guide may differ for each position information held by the storage unit125to improve visibility.

Change of Capturing Region of PTZ Camera

A flow when the pan angle, the tilt angle, and the zoom position of the PTZ camera120are changed so as to capture a past capturing region will be described by usingFIG. 3,FIG. 4A, andFIG. 4B. As illustrated inFIG. 4A, first, the system control unit124(FIG. 1) detects that the user has selected the thumbnail image205by using a pointer401(S301). The system control unit124then reads out, from the storage unit125, information on the pan angle, the tilt angle, and the zoom position of the PTZ camera120linked with the thumbnail image205and stored (S302). The system control unit124transmits an instruction to the drive control unit126of the PTZ camera120so as to drive the zoom motor127, the pan motor128, and the tilt motor129, based on the read out information on the pan angle, the tilt angle, and the zoom position (S303). That is, the system control unit124transmits a drive instruction of each of the motors127to129to the drive control unit126so that the PTZ camera120captures the region207.

Next, on the image201captured by the wide angle camera110, the system control unit124changes the cutout region207of the selected thumbnail image205to the region208that has been captured by the PTZ camera120before image selection (S304). Subsequently, the system control unit124stores, in the storage unit125, information on the pan angle, the tilt angle, and the zoom position of the PTZ camera120obtained before the capturing region is changed in link with the image (the image202ofFIG. 2) (S305). Finally, the system control unit124updates the displayed thumbnail image (S306). Specifically, the image202ofFIG. 4Ais updated to the image204because the image202is “an image captured at the immediately previous attitude (the latest image)”, and the image204ofFIG. 4Ais updated to the image205because the image204is “an image captured at the second previous attitude”. The thumbnail image205ofFIG. 4Awill be deleted. This state is illustrated inFIG. 4B.

In such a way, after confirming the current status by viewing the real-time image201(FIG. 4A) captured by the wide angle camera110, the user can change the capturing region of the PTZ camera120to the capturing region207that has been set in the past. (The image202ofFIG. 4Ahas changed to an image of the region207inFIG. 4B.) Note that, while two thumbnail images (the thumbnail images204and205) of capturing regions captured in the past are displayed in the display screen200in the above description, the present embodiment is not limited to such a display form. For example, only a single thumbnail image of a capturing region captured in the past may be displayed in the display screen200, or three or more thumbnail images of capturing regions captured in the past may be displayed in the display screen200.

Further, inFIG. 4A, when the thumbnail images204and205are displayed on the display screen200, the thumbnail image204is displayed on the front layer of the thumbnail image205. That is, inFIG. 4A, when there are multiple thumbnail images, a thumbnail image temporally closer to the current time is displayed on the front layer. The present embodiment is not limited to such a display form (display order). For example, the display order when a plurality of thumbnail images are displayed on the display screen200may be such that a thumbnail image which has been set for a longer time as a capturing region may be displayed in the front layer.

Display of Cutout Image Position

When a single thumbnail image is selected, in order to indicate which region is to be captured, a frame line indicating the outer edge of the region in interest or a line connecting the thumbnail image in interest to a capturing region may be displayed and overlapped with the captured image201of the wide angle camera110. Further, when the pointer401of a user operation is pointed on a thumbnail image (that is, a single thumbnail image is selected), the frame line described above or the connecting line described above may be displayed and overlapped with the captured image201. A figure or an index such as a frame line or a connecting line is displayed on the captured image201, and thereby a thumbnail image and a capturing region are associated with each other, which facilitates confirmation of correspondence between the thumbnail image and the capturing region.

Update Timing of Thumbnail Image

Update of a thumbnail image may be performed when a change of the pan angle, the tilt angle, or the zoom position that is greater than or equal to a predetermined value occurs to the PTZ camera120. Further, when continuous update of thumbnail images is not allowed, the thumbnail image may be updated when no thumbnail image selection has been made by the pointer401for a predetermined time or longer after the capturing region is changed. Alternatively, update of a thumbnail image may be allowed when a predetermined time period has elapsed after a change of the capturing region. Thereby, for example, when there are a large number of thumbnail images, it is possible to avoid continuous update of thumbnail images due to a fine adjustment of a capturing region.

Priority of Image Delivery

In delivery of images captured by the wide angle camera110to the IP network130, an image of a region displaying a cutout image (for example, the region207ofFIG. 4A) may be delivered with a higher priority than images of other regions. In particular, when the wide angle camera110is formed of a plurality of cameras, by prioritizing the delivery of an image of a region in which a cutout image is displayed, the time lag between the cutout image captured by the wide angle camera110and an image captured by the PTZ camera120can be reduced.

As described above, according to the present embodiment, it is possible to change the capturing region of the PTZ camera120from the region208to the region207while displaying the current state of the region207captured in the past to the user by using the real-time images201and205.

Next, a second embodiment of the present invention will be described by usingFIG. 5toFIG. 7B. In the first embodiment, the capturing region of the PTZ camera120is changed to a capturing region of a thumbnail image selected by the user using the pointer401. In the second embodiment, a case when a plurality of capturing regions, in which capturing regions of the PTZ camera120are pre-set, are sequentially changed will be described. In the second embodiment, the capturing region of the PTZ camera120changes in predetermined order (circulation order). Such an image capturing may be referred to as circulation setting capturing.

A plurality of pre-set capturing regions may be referred to as a pre-set position in the following description. A change method of a capturing region and a cutout range of the imaging device101and a display method of a thumbnail image in the second embodiment will be described below. Note that the same reference numerals are used to the same components as those in the first embodiment, and the description thereof will be omitted. The configuration of the imaging system10of the second embodiment is the same as that of the first embodiment. In the following description, display on the UI display unit140having different form from that of the first embodiment and change of a capturing direction and a view angle (change of a pan angle, a tilt angle, and a zoom position) of the PTZ camera120will be described.

UI Display Unit

Display of an image on the UI display unit140will be described by usingFIG. 5. Images501,502,503, and504are each a thumbnail image in which a region that can be captured by a circulation setting capturing of the PTZ camera120(that is, a pre-set position) is cut out from the image201currently captured by the wide angle camera110and is displayed. That is, in the circulation setting capturing of the present embodiment, the four thumbnail images501to504are used. The user can select (set) which of the thumbnail images the circulation is started. For example, when the user selects the thumbnail image504, the capturing region of the PTZ camera120changes in the order of the thumbnail images504,501,502,503, and then504. This circulation order may be set by the UI display unit140based on the user instruction or may be set by the system control unit124of the imaging device100, for example. In the present embodiment, the thumbnail images501to504are prepared in advance. The thumbnail image501corresponds to a region505, the thumbnail image502corresponds to a region506, the thumbnail image503corresponds to a region507, and the thumbnail image504corresponds to a region508.

A pan angle, a tilt angle, and a zoom position of the PTZ camera120are linked with each of the thumbnail images501,502,503, and504and stored in the storage unit125. Such linking enables the PTZ camera120to capture the regions505to508based on the thumbnail images501to504.

Change of Capturing Region of PTZ Camera

A flow of changing the pan angle, the tilt angle, and the zoom position of the PTZ camera120to a pan angle, a tilt angle, and a zoom position linked with the pre-set position will be described by usingFIG. 6,FIG. 7A, andFIG. 7B. In the following description, a case when circulation setting capturing is started with the thumbnail image504will be described. The circulation setting capturing is set such that, after capturing is performed for a predetermined time at one capturing region, capturing is automatically moved to the next capturing region.

As illustrated inFIG. 7A, first, the system control unit124(FIG. 1) detects that the user has selected the thumbnail image504by using a pointer701(S601). The system control unit124then reads out, from the storage unit125, information on the pan angle, the tilt angle, and the zoom position linked with the thumbnail image504and stored (S602). The system control unit124transmits an instruction to the drive control unit126of the PTZ camera120so as to drive the zoom motor127, the pan motor128, and the tilt motor129based on the read out information on the pan angle, the tilt angle, and the zoom position (S603). That is, the system control unit124transmits a drive instruction of each of the motors127to129to the drive control unit126so that the PTZ camera120captures the region508. A state after the capturing region of the PTZ camera120has been changed is illustrated inFIG. 7B.

An image that is currently being captured by the wide angle camera110is displayed in the image201. Therefore, the user can change the pan angle, the tilt angle, and the zoom position of the PTZ camera120after confirming the status of a pre-set position in the image201when changing the pan angle, the tilt angle, and the zoom position of the PTZ camera120to the pre-set position (the region508). After the PTZ camera120captures the region508for a predetermined time period, the system control unit124reads out, from the storage unit125, information on the pan angle, the tilt angle, and the zoom position linked with the thumbnail image501(S604). That is, the system control unit124automatically, sequentially selects the thumbnail image501. The system control unit124then transmits an instruction to the drive control unit126of the PTZ camera120so as to drive each of the motors127to129based on the read out information on the pan angle, the tilt angle, and the zoom position (S603). In this way, in the circulation setting capturing, the process proceeds to S604after S603without updating a thumbnail image. Then, S603and S604are repeatedly performed. To stop the circulation setting capturing, for example, the user transmits a stop signal from the UI display unit140to the system control unit124.

Note that, in the above description, while which thumbnail image the circulation setting capturing is started is determined by the user selecting a thumbnail image, a thumbnail image with which circulation setting capturing is started may be pre-set. For example, circulation setting capturing may be set to start with the thumbnail image501. Further, while circulation setting capturing is started by the user selecting the thumbnail image504in the flowchart ofFIG. 6, the circulation setting capturing may be started in response to the UI display unit140being powered on, for example. Furthermore, the time period in which capturing is performed may be different for the four pre-set positions505to508. For example, the time period in which capturing is performed at the pre-set position505may be shorter than the time period in which capturing is performed at the pre-set position507. While the four thumbnail images501to504are used to perform circulation setting capturing in the example ofFIG. 7AandFIG. 7B, the number of thumbnail images used in circulation setting capturing is not limited to four.

According to the second embodiment, the user can change the pan angle, the tilt angle, and the zoom position of the PTZ camera120after confirming the status of a pre-set position in the image when changing the pan angle, the tilt angle, and the zoom position of the PTZ camera120to the pre-set position508. Further, also for the pre-set positions505,506, and507captured automatically after the pre-set position508, the current status of the present position (region) in interest can be confirmed. Furthermore, according to the second embodiment, a plurality of pre-set positions (capturing regions) can be captured with automatic circulation at predetermined time intervals.

While the wide angle camera110is formed of a single camera without means for changing a capturing direction and a zoom position in the first embodiment and the second embodiment, the wide angle camera110is not limited to such a configuration. The wide angle camera110may be formed of a plurality of cameras or may be formed of a camera capable of changing a capturing direction and a zoom position (for example, a PTZ camera). While described as a personal computer, for example, the UI display unit140may be a server. Further, while described as being accommodated in a single casing, the wide angle camera110and the PTZ camera120may be accommodated in separate casings. That is, the imaging device100may not be configured such that two cameras (wide angle cameras110) are integrated.

Setting Change by User

By transmitting a setting change instruction from the UI display unit140to the wide angle camera110, the user is able to switch which of the past capturing region or the circulation setting capturing region of the PTZ camera120is cut out and displayed as a thumbnail image. Similarly, the user is also able to transmit a drive control instruction to the drive control unit126via the UI display unit140to drive various motors (127,128,129) and to change the capturing region of the PTZ camera120. Further, it is possible to change the size of a thumbnail image or a display position (layout) of an image through the UI display unit140. Further, while the example in which the present invention is implemented in the imaging device100has been described above, the present invention may be implemented on the client device side. In the case of implementation on the client device side, the client device holds information indicating which portion of an image captured by the wide angle camera110is to be displayed on a viewer as a thumbnail and, when each thumbnail is selected, information indicating which position the PTZ camera120is to be PTZ-controlled. Further, when one of the thumbnails is selected, an instruction to drive the PTZ camera120to the corresponding PTZ position can be issued.

Advantageous Effect

According to the present invention, an image of the second capturing region is cut out from an image in which the first capturing region is captured, and the cut out image is displayed, which can facilitate the user to confirm a state of the second capturing region.

OTHER EMBODIMENTS