Patent Description:
Conventionally, an imaging system that is intended to promptly move the imaging direction of an imaging device in a desired direction has been known, as described in Patent Literature <NUM> below, for example.

<CIT> discloses that a picture photographed by a camera portion is sent to a video capturing portion of a computer. The picture is displayed in an operation area of a monitor. A panorama picture of which pictures in part or all moving range of a pan tiler are combined is displayed in a panorama operation area. A pan tilter portion sends positional information of pan and tilt to the computer through a mode controller. With a mouse, the operation area and the panorama operation area are operated so as to select an object. The computer obtains data for driving the pan tilter. Thus, the selected object is displayed at the center of the operation area.

<CIT> is concerned with enabling a user to perform the designation of a target local region easily in case of acquiring a detailed image by enabling the setting of the central direction of a panoramic image shown in order to grasp the state of the whole surrounding and its dimension. To this end, an image formation apparatus is disclosed that includes all direction image acquisition parts that are constituted including one or a plurality of cameras and acquires all direction images as an ambient image, a panoramic image edge setting part for setting the central direction and the angle of image to all the direction images acquired by the all direction image acquisition parts, and a panoramic image creation processor which makes the central direction set by the panoramic image edge setting part into the center position from all or the part of the all direction images acquired by the all direction image acquisition parts, and extracts the image of corresponding dimensions to the angle of the image to constitute the panoramic image.

<CIT> discloses a panoramic image management system. The system provides for improved calibration of a <NUM> degree panoramic camera, as well as improved means for manipulating the images, and correlating images from conventional video cameras with the images from the <NUM> degree panoramic camera. The system also provides means for archiving and retrieving stored images.

<CIT> is concerned with providing a camera apparatus capable of outputting a panoramic image and an enlarged image at the same time. To this end, an apparatus is disclosed that comprises a camera apparatus part and a universal head for camera. After panoramic image light received by the panoramic lens of the camera apparatus part is enabled to be rotated by <NUM>° by rotating the universal head for camera, the light received by the panoramic lens of the camera apparatus part is branched by a half-mirror and guided to a two different CCDs, and the zoom lens of a zoom mechanism is provided between the half-mirror and one of the CCDs to obtain image light corresponding to the panoramic image light and an enlarged image obtained by optically enlarging part of the image light at the same time.

The technique described in Patent Literature <NUM> is directed to control the imaging direction of an imaging device based on a rectangular panorama image. However, there is a problem in that control performed based on a panorama image can be intuitive only when the imaged object corresponding to the panorama image is rectangular in shape.

The present invention has been made in view of the foregoing problem, and it is an object of the present invention to provide a control device, a camera system, and a program that are capable of more intuitively controlling the imaging direction of an imaging device based on a panorama image even when the imaged object is not rectangular in shape.

In order to solve the aforementioned problem, according to one aspect of the present invention, there is provided a control device according to claim <NUM>.

In order to solve the aforementioned problem, according to still another aspect of the present invention, there is provided a camera system according to claim <NUM>.

In order to solve the aforementioned problem, according to yet another aspect of the present invention, there is provided a method for a camera control device according to claim <NUM>.

The appended dependent claims define further embodiments.

According to the present invention, a control device, a camera system, and a method can be provided that are capable of more intuitively controlling the imaging direction of an imaging device on a panorama image.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the appended drawings. Note that, in this specification and the drawings, elements that have substantially the same function and structure are denoted with the same reference signs, and repeated explanation is omitted.

Unless explicitly indicated as "embodiments according to the claimed invention", any embodiment in the description may include some but not all features as literally defined in the claims and are present for illustration purposes only.

Note that the description will be given in the following order.

<FIG> is a schematic diagram showing the overall configuration of a camera system according to the claimed invention. A camera system <NUM> includes camera terminal devices (IP cameras) <NUM>, a center server <NUM>, and clients <NUM>. The plurality of camera terminal devices <NUM>, the center server <NUM>, and the clients <NUM> are connected via a network <NUM>. In addition, a proxy server <NUM> is connected to the network <NUM>, and the plurality of camera terminal devices <NUM> is connected to the proxy server <NUM>. Further, a monitor <NUM> is connected to the center server <NUM>. In this embodiment, the camera terminal device <NUM> is, for example, a monitoring camera disposed inside or outside a building. When an image acquired with the camera terminal device <NUM> is displayed on the monitor <NUM> of the center server <NUM>, it becomes possible to monitor the area in which the camera terminal device <NUM> is disposed. The client <NUM> is connected to the camera terminal devices <NUM>, the center server <NUM>, or the proxy server <NUM> via the network <NUM>. In addition, the client <NUM> can have the function of the center server <NUM> and the function of the monitor <NUM> described below. A video captured with the camera terminal device <NUM> is sent to the client <NUM> via the network <NUM>. On a display screen of the client <NUM>, a first panorama image and a second panorama image are displayed together with a live video described below. Although an IP camera is given as an example of the camera terminal device <NUM>, the camera terminal device <NUM> is not limited thereto, and can be an analog camera.

A panorama image shows the range in which imaging can be performed with the camera terminal device <NUM>. Panorama images come in various shapes. Examples include a circular panorama image and a rectangular panorama image as shown in <FIG>, and a ringshaped panorama image like the one projected onto a cylindrical plane as shown in <FIG>. In the following embodiments according to the claimed invention, a circular panorama image, also described as a birds-eye view image, and a rectangular panorama image, also described as a wide-view image, will be described as examples of the first panorama image and the second panorama image, respectively.

<FIG> is a schematic diagram showing the details of the configurations of the camera terminal device <NUM> and the center server <NUM> in the system of <FIG>. Each component of the camera terminal device <NUM> or the center server <NUM> shown in <FIG> can be constructed from hardware (e.g., a circuit) or from an arithmetic processing unit such as a CPU and software (a program) for causing the CPU to function. A camera unit <NUM> of the camera terminal device <NUM> includes an imaging optical unit, and performs an imaging operation based on an imaging control signal CTa supplied from a terminal control unit <NUM> described below, and further generates a video signal Dcam. In addition, the camera unit <NUM> supplies the generated video signal Dcam to a signal processing unit <NUM> via a bus <NUM>. Note that a memory unit <NUM>, an imaging direction control unit <NUM>, a network interface unit <NUM>, a storage unit <NUM>, and the terminal control unit <NUM> are also connected to the bus <NUM>.

The signal processing unit <NUM> performs a compression process on the video signal Dcam and stores an obtained video encoded signal DV into the memory unit <NUM>. Further, according to the embodiments of the claimed invention, the signal processing unit <NUM> generates a video signal of a first panorama image (hereinafter referred to as a "first panorama image signal" DF and a video signal of a second panorama image (hereinafter referred to as a "second panorama image signal") DP using video signals Dcam obtained by sequentially moving the imaging direction of the camera unit <NUM>, and stores them into the storage unit <NUM>. Note that the compression process on the video signal Dcam and the generation of the first panorama image signal DF or the second panorama image signal DP are performed based on a signal processing control signal CTb supplied from the terminal control unit <NUM> described below.

The imaging direction control unit <NUM> calculates the speed and acceleration for moving the imaging direction of the camera unit <NUM> to a direction indicated by a direction control signal CTc supplied from the terminal control unit <NUM> described below. Based on the calculation result, the imaging direction control unit <NUM> further generates a drive signal MDp for performing a pan operation and a drive signal MDt for performing a tilt operation, and supplies them to an imaging direction adjustment unit <NUM>. Furthermore, the imaging direction control unit <NUM> generates a camera position information signal PM indicative of the imaging direction of the camera unit <NUM> and supplies it to the terminal control unit <NUM>.

The imaging direction adjustment unit <NUM> includes a pan operation motor for moving the camera unit <NUM> in the horizontal direction and a tilt operation motor for moving the camera unit <NUM> in the vertical direction. The imaging direction adjustment unit <NUM> drives the pan operation motor in accordance with the drive signal MDp and the tilt operation motor in accordance with the drive signal MDt to adjust the imaging direction of the camera unit <NUM> to the direction indicated by the direction control signal CTc. Further, the imaging direction adjustment unit <NUM> performs a pan operation in an endless manner without operating range limitations.

<FIG> is a schematic diagram for illustrating a pan operation and a tilt operation of the camera terminal device <NUM>, which shows a case in which the camera terminal device <NUM> is mounted on a ceiling or the like. Herein, <FIG> shows pan/tilt operations of the camera terminal device <NUM> in accordance with this embodiment. Meanwhile, <FIG> shows pan/tilt operations of a typical camera terminal device for comparison purposes. As shown in <FIG>, the typical camera terminal device performs a <NUM>° endless pan operation, and performs a tilt operation in the range of <NUM>° from the vertical direction to the horizontal direction. In contrast, the camera terminal device <NUM> in accordance with this embodiment can perform a <NUM>° endless pan operation, and perform a tilt operation in the range of <NUM>° with reference to the vertical direction as the center. Thus, it is possible to generate a video signal DF of a first panorama image with a method described below and store it into the storage unit <NUM>. The first panorama image described below is a circular panorama image. When operating the imaging direction by striding over an area around the center of the circle, the camera terminal device <NUM> in accordance with this embodiment is able to mechanically move the imaging direction to a target imaging direction via the shortest route. Meanwhile, the typical camera terminal device is unable to mechanically move the camera imaging direction in such a way that it strides over the center of the circle.

The network interface unit <NUM> is an interface for performing communication between the camera terminal device <NUM> and the center server <NUM> via the network <NUM>.

The terminal control unit <NUM> controls the camera unit <NUM> using the imaging control signal CTa, controls the signal processing unit <NUM> using the signal processing control signal CTb, controls the imaging direction control unit <NUM> using the direction control signal CTc, and performs imaging while sequentially moving the imaging direction. Then, the terminal control unit <NUM>, based on the obtained video signal Dcam, generates the first panorama image signal DF and the second panorama image signal DP, and stores them into the storage unit <NUM>.

The imaging system <NUM> in accordance with this embodiment can transmit a live video in accordance with the video signal Dcam, together with a first panorama image or a second panorama image in accordance with the first panorama image signal DF or the second panorama image signal DP, from the camera terminal device <NUM> to the center server <NUM>, and display them on the monitor <NUM>. Note that in this specification, the first panorama image and the second panorama image may be referred to as reference images. <FIG> is a schematic diagram showing the display states of the second panorama image and the first panorama image. In embodiments according to the claimed invention, the center server <NUM> can, in response to selection by a user, display one of the first panorama image or the second panorama image together with a live video on the monitor <NUM>. Alternatively, the center server <NUM> can be configured to display both the first panorama image and the second panorama image together with a live video. Although <FIG> shows as the first panorama image a circular panorama image in which an image around the center is missing, the first panorama image is not limited thereto, and a circular panorama image in which an image around the center is not missing can be used as the first panorama image.

The camera terminal device <NUM> is disposed on a ceiling or the like, and displays an image of an area in the downward direction of the camera terminal device <NUM>. Therefore, as shown in <FIG>, the first panorama image is acquired in a pan range of <NUM>° and a tilt range of <NUM>° with reference to a vertical line, which extends downward from the camera terminal device <NUM>, as the center, and the outline of the first panorama image is circular.

As described above, the imaging direction adjustment unit <NUM> is so constructed as to perform a pan operation in an endless manner. Accordingly, in order to avoid overlapping of images, the second panorama image stored in the storage unit <NUM> provides a "+<NUM>°" angle difference from a reference direction at one end and a "-<NUM>°" angle difference from the reference direction at the other end, as shown in <FIG>, for example. Further, since the pan operating range is not limited, it is not possible to determine the center position of the operating range as a reference direction unlike when the operating range is limited, for example. For this reason, a reference direction is preset, and a second panorama image signal DP is generated in such a way that the preset reference direction coincides with the center position of a second panorama image. The imaging direction control unit <NUM> generates a signal indicative of, for example, the angle difference of the imaging direction with respect to the preset reference direction, as a camera position information signal PM.

If a reference direction is thus preset, it is not necessary to determine which direction has been used as a reference of the imaging operation to generate a second panorama image. Assuming that the camera position information signal PM indicates the angle difference of the imaging direction with respect to the reference direction, it is possible to easily determine the position of the imaging direction on the second panorama image based on the camera position information signal PM.

For a tilt operation, a reference direction is also preset, and the camera position information signal PM indicates the angle difference of the imaging direction with respect to the reference direction. Accordingly, it is possible to easily identify the position of the imaging direction on each of the first panorama image and the second panorama image based on the camera position information signal PM. Note that for a tilt operation, if the camera terminal device <NUM> is disposed on a ceiling, the vertical direction can be set as the reference direction.

The terminal control unit <NUM> analyzes a command signal CM supplied from the center server <NUM> via the network interface unit <NUM> to generate an imaging control signal CTa, a signal processing control signal CTb, and a direction control signal CTc. Further, the terminal control unit <NUM> performs a process of sending out the video encoded signal DV stored in the memory unit <NUM>, the first panorama image signal DF and the second panorama image signal DP stored in the storage unit <NUM>, and the camera position information signal PM supplied from the imaging direction control unit <NUM> to the center server <NUM>.

A network interface unit <NUM> of the center server <NUM> is an interface for performing communication between the camera terminal device <NUM> and the center server <NUM> via the network <NUM>. The network interface unit <NUM> supplies the video encoded signal DV supplied from the camera terminal device <NUM> to a decompression processing unit <NUM>. Further, the network interface unit <NUM> supplies the camera position information signal PM to a position determination processing unit <NUM> and supplies the first panorama image signal DF and the second panorama image signal DP to the image processing unit <NUM>.

The decompression processing unit <NUM> performs a decompression process on the video encoded signal DV and supplies the obtained video signal Dcam to a display processing unit <NUM>.

The position determination processing unit <NUM> determines which position in the first panorama image or the second panorama image corresponds to the imaging direction indicated by the camera position information signal PM, and supplies a position determination result KP to the image processing unit <NUM>. In regard to the camera position information signal PM, if the first panorama image signal DF or the second panorama image signal DP is generated in such a way that the preset reference direction coincides with the center position of a pan operation and a tilt operation and the camera position information signal PM indicates the angle difference between the reference direction and the current imaging direction as described above, it is possible to easily determine which position in the bird's eye image or the second panorama image corresponds to the current imaging direction, based on the camera position information signal PM.

The image processing unit <NUM> performs image processing based on the position determination result KP, and generates a first panorama image signal DFC of the first panorama image and a second panorama image signal DPC of the second panorama image. In regard to the second panorama image signal DPC, it is possible to generate the second panorama image signal DPC in which the position determined from the position determination result KP is set as the center position. In this case, the image processing unit <NUM> determines the amount of difference between the determined position and the center position of the second panorama image, and processes the second panorama image in such a way that the determined position becomes the center position of the image, based on the determined amount of difference. That is, the image processing unit <NUM> sets a second panorama image display frame of "±<NUM>°" in which the determined position becomes the center position, pastes a second panorama image of the area for the amount of difference outside the second panorama image display frame to the area having no image, and thereby generates the second panorama image signal DPC of the second panorama image in which the determined position becomes the center position. In the case of the first panorama image signal DFC, in a "mode in which the first panorama image is rotated" described below, the amount of difference between the determined position and the reference position in the pan direction of the first panorama image is also determined as with the second panorama image signal DPC, and, based on the determined amount of difference, the first panorama image is processed in such a way that the determined position is located above the image. Further, the image processing unit <NUM> displays a position (a position that the optical axis of the imaging optical unit of the camera unit <NUM> faces) that is based on the position determination result KP on each of the first panorama image signal DFC and the second panorama image signal DPC. Further, the image processing unit <NUM> supplies the first panorama image signal DFC and the second panorama image signal DPC generated by performing image processing to the display processing unit <NUM>.

The display processing unit <NUM> generates a display drive signal HD using the video signal Dcam supplied from the decompression processing unit <NUM> and the first panorama image signal DFC and the second panorama image signal DPC supplied from the image processing unit <NUM>, and supplies the display drive signal HD to the monitor <NUM>. In addition, the display processing unit <NUM> generates a display drive signal HD using a video signal for GUI generated based on a GUI display control signal CTg supplied from the control unit <NUM> described below, and supplies the display drive signal HD to the monitor <NUM>.

The monitor <NUM>, based on the display drive signal HD, displays the first panorama image, the second panorama image in which the imaging direction is at the center position, an image (live video) shot by the camera unit <NUM>, a GUI image, and the like on the screen by driving display elements such as liquid crystal elements, plasma display elements, or a cathode ray tube.

For the user interface unit <NUM>, a GUI is used. The user interface unit <NUM> presents information to a user using the monitor <NUM>. When a user operation is performed using an operation input means such as a pointing device or a keyboard based on the presented information, the user interface unit <NUM> supplies an operation signal US in accordance with the user operation to the control unit <NUM> to cause it to perform a desired operation or the like.

Based on the operation signal US and the display state of the monitor <NUM>, the control unit <NUM> determines what kind of process the user has selected and what kind of process the user has requested for execution. Based on the determination result, the control unit <NUM> generates a command signal CTm and supplies it to each unit to control the operation of the center server <NUM>. Further, based on the determination result, the control unit <NUM> generates a command signal CM and supplies it to the camera terminal device <NUM> via the network interface unit <NUM> to control the operation of the camera terminal device <NUM>. Furthermore, the control unit <NUM> generates a GUI display control signal CTg and supplies it to the display processing unit <NUM>.

Although the camera terminal device <NUM> generates the video signal DF of the first panorama image and the video signal DP of the second panorama image from the video signal Dcam in the aforementioned example, the video signal DF of the first panorama image and the video signal DP of the second panorama image can be generated by the center server <NUM> or the client <NUM>. In such a case, the network interface unit <NUM> of the camera terminal device <NUM> sends the video signal Dcam output from the camera block <NUM> to the center server <NUM> or the client <NUM>. The center server <NUM> or the client <NUM> can, by having a similar signal processing unit to that of the camera terminal device <NUM>, generate the video signal DF of the first panorama image and the video signal DP of the second panorama image. The video signal DF of the first panorama image and the video signal DP of the second panorama image are stored into memory such as a hard disk of the center server <NUM> or the client <NUM>.

Next, the operation of displaying the first panorama image and the second panorama image will be described. At the start of the operation, the control unit <NUM> sends to the camera terminal device <NUM> the command signal CM requesting the first panorama image signal DF, the second panorama image signal DP, the camera position information signal PM, and the video encoded signal DV. If the storage unit <NUM> has stored therein the first panorama image signal DF and the second panorama image signal DP, the camera terminal device <NUM> reads the first panorama image signal DF and the second panorama image signal DP and sends them to the center server <NUM>. If the image storage unit <NUM> does not have stored therein the first panorama image signal DF or the second panorama image signal DP, the camera terminal device <NUM> controls the camera unit <NUM>, the signal processing unit <NUM>, and the imaging direction control unit <NUM> so as to perform an imaging operation while moving the imaging direction. Thus, the camera terminal device <NUM> generates the first panorama image signal DF and the second panorama image signal DP and sends them to the center server <NUM>. Note that the camera terminal device <NUM> stores the generated first panorama image signal DF and second panorama image signal DP into the storage unit <NUM>.

The camera terminal device <NUM> supplies the camera position information signal PM generated by the imaging direction control unit <NUM> to the center server <NUM>. Further, the camera terminal device <NUM> starts the imaging operation of the camera unit <NUM>, and compresses the obtained video signal Dcam with the signal processing unit <NUM> and then stores the video encoded signal DV into the memory unit <NUM>. Furthermore, the camera terminal device <NUM> supplies the video encoded signal DV stored in the memory unit <NUM> to the center server <NUM>. Note that when the camera terminal device <NUM> and the network <NUM> are connected via the proxy server <NUM> as shown in <FIG>, exchange of information between the camera terminal device <NUM> and the center server <NUM> is conducted via the proxy server <NUM>.

The position determination processing unit <NUM> of the center server <NUM> determines which position in the first panorama image or the second panorama image corresponds to the imaging direction indicated by the camera position information signal PM, and supplies a position determination result KP to the image processing unit <NUM>. In regard to the second panorama image, for example, when the angle difference indicated by the camera position information signal PM is "<NUM>°," for example, the imaging direction is located at the center position of the second panorama image because the imaging direction is set to the reference direction.

The image processing unit <NUM> performs image processing based on the position determination result KP. If the position determination result KP indicates that the imaging direction is located at the center position of the second panorama image, the image of the imaging direction is at the center of the panorama image. Thus, the image processing unit <NUM> does not perform second panorama image processing and supplies the second panorama image signal DP as a second panorama image signal DPC to the display processing unit <NUM>. Meanwhile, when the imaging direction is not located at the center position of the second panorama image, the image processing unit <NUM> determines the amount of difference between the determined position of the imaging direction and the center position of the second panorama image, and processes, based on the determined amount of difference, the second panorama image in such a way that the determined position becomes the center position of the image, and thus generates a second panorama image signal DPC.

Therefore, in regard to the second panorama image, when the angle difference indicated by the camera position information signal PM is "<NUM>°," for example, the image captured with the camera unit <NUM>, the second panorama image in which the center position coincides with the reference direction, and the image related to the GUI are displayed on the screen of the monitor <NUM> that is driven in accordance with the display drive signal HD from the display processing unit <NUM>.

In regard to the first panorama image, in the case of the "mode in which the first panorama image is rotated" described below, the amount of difference between the determined position and the reference position in the pan direction of the first panorama image is determined as with the second panorama image signal DPC, and, based on the determined amount of difference, the first panorama image is rotated in such a way that the determined position is located below the image (on a straight line L described above), so that a first panorama image signal DFC is generated. In addition, the image processing unit <NUM> displays, in each of the first panorama image signal DFC and the second panorama image signal DPC, the position of the imaging direction (a position that the optical axis of the imaging optical unit of the camera unit <NUM> faces) based on the position determination result KP.

The decompression processing unit <NUM> decompresses the video encoded signal DV and supplies the obtained video signal Dcam to the display processing unit <NUM>. The control unit <NUM> supplies the GUI display control signal CTg used for GUI display to the display processing unit <NUM>.

Next, a process of generating the first panorama image signal DF will be described with reference to <FIG>. When the first panorama image signal DF is generated, an image of each area of the first panorama image is acquired. First, as shown in <FIG>, with the zoom of the camera terminal device <NUM> fixed, pan and tilt are performed at given intervals to acquire still images. The number shown in <FIG> indicates the image acquisition order.

Next, coordinate transformation of a still image coordinate system is performed in accordance with the following procedures [<NUM>] to [<NUM>]. Herein, as shown in <FIG>, each point in a still image is transformed into a relative position (x,y,z) seen from the center of rotation of the camera terminal device <NUM>.

(x3,y3,z3) obtained herein is the value represented by a coordinate system obtained by viewing a given point (x,y,z) in a still image from the center of rotation of the camera.

Next, the relative position is transformed into the latitude and the longitude. Herein, as shown in <FIG>, (x3,y3,z3) obtained above is transformed into the latitude and the longitude. The angle made by the direction of (x3,y3,z3) with the horizontal direction for the -z direction is the longitude, and the angle made by the direction of (x3,y3,z3) with the x-z plane is the latitude, and can be determined from the following formulae.

Next, transformation into a first panorama image is performed. Herein, as shown in <FIG>, the obtained latitude and longitude (La,Lo) are represented by the coordinate system of the axis of the radius and the coordinate system of the circumferential direction, respectively.

The first panorama image signal DF can be generated in a manner described above.

Next, a GUI will be described. <FIG> is a schematic diagram showing a display screen of the monitor <NUM>. <FIG> shows both a normal image and a full screen display. In both cases, an operation panel (a control panel) is displayed on the left side of the screen, and a live video is displayed on the right side of the screen. When "Screen Mode" on the operation panel is set to "Full," a full screen is displayed.

<FIG> is a schematic diagram for illustrating an operation with the operation panel. The operation panel includes panels for individual functions, and each panel can be stored by being folded. The panels are classified as follows.

Each panel is configured to open when clicked. <FIG> shows a state in which "View" is clicked and a menu is displayed below the "View" field. Meanwhile, <FIG> shows a state in which "Camera control" is clicked and a menu is displayed below the "Camera control" field. In addition, <FIG> shows a state in which the panel "Camera control" is dragged. The panel "Camera control" can be turned into a floating window when dragged.

<FIG> shows a state in which menus such as "Screen Mode," "View Size," "Image Codec," and "Frame Rate" are displayed when the panel "View" is clicked. With the "Screen Mode" menu, the display mode can be switched between normal and a full screen. With the "View Size," <NUM>, <NUM>/<NUM>, <NUM>% magnification, or Fit mode of an image can be specified. In the Fit mode, an image is displayed in accordance with the size of the current display area. With the "Image Codec" menu, Codec can be changed. With the "Frame Rate" menu, the JPEG frame rate can be changed. A moving-image save button is used to save a moving image, and a still-image save button is used to save a still image. Besides, a microphone volume slider, a microphone mute on/off button, a sound volume slider, a sound mute on/off button, and the like are provided.

<FIG> shows a menu displayed when a panel "Camera" is clicked. With an "Operation Mode" menu, the operation mode can be switched on an image. With pan/tilt control buttons, it is possible to specify pan/tilt directions by operating the buttons of the arrow directions. A button located at the center of the pan/tilt control buttons is a home position button. A zoom button is a button for specifying the zoom of the camera terminal device <NUM>, and is constructed such that the boundary between the optical zoom and the digital zoom can be clearly seen. When a zoom button "W" or "T" is continuously pressed, a zoom operation is continuously performed. A focus button is used to set the focus to "long range" or "short range. " Besides, various buttons such as a ONE PUSH AUTO FOCUS button and a control right acquisition button for exclusive control are provided.

<FIG> shows each menu displayed when "Preset position," "Trigger," "Other," or "information" is clicked. "Preset position" is a thumbnail display of images shot in a plurality of predetermined shooting directions. When one of the thumbnail images is clicked and selected, the camera terminal device <NUM> can be made to face the direction of the selected thumbnail image.

Next, a display of the first panorama image will be described with reference to <FIG>. <FIG> is a schematic diagram showing a state in which "Preset position" on the operation panel is clicked on the display screen of the monitor <NUM> so that the first panorama image is displayed. Note that a live video is displayed to the right of the operation panel. As described above, it is possible to arrange a subpanel on which the first panorama image is displayed outside the operation panel as shown in <FIG>, by moving the "Preset position" field through dragging. In addition, the subpanel on which the first panorama image is displayed can be moved within the display screen.

<FIG> is a schematic diagram showing the details of the subpanel on which the first panorama image is displayed. On the first panorama image, the direction of an image displayed with the live video on the right side (the direction of the optical axis of the imaging optical unit of the camera terminal device <NUM>) is indicated by a + mark. In addition, as shown in <FIG>, when a panorama switch button on the upper right of the first panorama image is clicked, the first panorama image can be switched to the second panorama image.

<FIG> shows an operation of changing the direction of a live video using the first panorama image. When the pan/tilt directions of the camera terminal device <NUM> are changed, the change can be attained by using the aforementioned pan/tilt control buttons, or by specifying a given point in the first panorama image. In the state of <FIG>, the optical axis of the imaging optical unit of the camera terminal device <NUM> is made to face the + mark direction, and a live image in that direction is displayed. In such a state, when a • mark portion shown in the first panorama image of <FIG> is clicked, an operation signal US corresponding to the user operation is sent to the control unit <NUM>, and based on this, the imaging direction control unit <NUM> of the camera terminal device <NUM> is controlled. Accordingly, the optical axis of the imaging optical unit of the camera terminal device <NUM> is made to face the • mark direction. Thus, the live video displayed to the right of the operation panel is switched to a video in the • mark direction.

Accordingly, the user is able to specify the pan/tilt directions of the camera terminal device <NUM> by specifying a given point in the first panorama image. The first panorama image includes videos in all pan/tilt directions that can be shot with the camera terminal device <NUM>. Thus, the user is able to cause a live video in the desired direction to be displayed by specifying the direction of the camera terminal device <NUM> on the first panorama image. Note that it is also possible to control the optical axis by specifying an area with a frame surrounding a predetermined range, instead of using the + mark indicative of the clicked position.

When the direction of the camera terminal device <NUM> is specified on the first panorama image, there are the following modes: a mode in which the display state of the first panorama image is not rotated and only a live image is switched, and a mode in which the first panorama image is rotated. Only the mode in which the display state of the first panorama image is not rotated is an embodiment according to the claimed invention. The mode in which the first panorama is rotated is not according to the claimed invention. In the mode in which the first panorama image is rotated, as shown in <FIG>, the angular position of the first panorama image is set in such a way that the position (+ mark), for which a live video is currently displayed, is located on a straight line L that extends upward from the center of the first panorama image. Next, when the mark is clicked on the first panorama image, the live video displayed to the right of the operation panel switches to a video in the • mark direction, and at the same time, the first panorama image is rotated so that the • mark position is located on the straight line L above the first panorama image. Thus, by viewing the first panorama image, the user is able to immediately recognize that the direction of the live video is on the straight line L of the first panorama image.

Likewise, in regard to the second panorama image, the user is able to specify the pan direction of the camera terminal device <NUM> by specifying a given point in the second panorama image, and based on this, the direction of the live video can be switched.

<FIG> shows an example in which, instead of the panorama images such as those shown in <FIG>, a panorama image displayed on a cylindrical curved plane is used as a reference image. Like this, a rectangular panorama image displayed in the range of +<NUM>° to -<NUM>° can be displayed on a cylindrical plane. Accordingly, it is possible to enhance the viewability for the user and to display a video in the desired direction. In the case of <FIG>, the user is also able to specify the pan direction of the camera terminal device <NUM> by specifying a "display specified point" on the curved plane, and based on this, the direction of a live video can be switched. Note that the aforementioned process including the process of specifying the shooting direction is performed by the display processing unit <NUM> based on the GUI display control signal CTg.

Next, effects of displaying the first panorama image will be described with reference to <FIG>. <FIG> shows a view in which a "safe" and a "table" are arranged on the floor, and the camera terminal device <NUM> is disposed immediately above the intermediate position (point O) between the "safe" and the "table. " It is assumed that a suspicious person approaching the "safe" can be monitored using a live video obtained with the camera terminal device <NUM>.

<FIG> schematically shows a state in which, in the state of <FIG>, the first panorama image is displayed together with a live video on the monitor <NUM> (<FIG>) and a state in which the second panorama image is displayed together with a live video on the monitor <NUM> (<FIG>). As has been already described, when a given position on the first panorama image or the second panorama image is clicked, the direction of the optical axis of the camera terminal device <NUM> can be changed to the clicked position, and the direction in which a live video is shot can also be changed.

As shown in <FIG>, in the case of the second panorama image, an image is distorted at a position immediately below the camera terminal device <NUM>, that is, at the intermediate position between the "safe" and the "table. " Thus, even when the direction of the optical axis is changed to the clicked position, the direction of the optical axis will not be located at the actual intermediate position (point O shown in <FIG>) between the "safe" and the "table. " That is, when an area around the center between the "safe" and the "table" is to be viewed, if an operation is performed on the second panorama image (the rectangular panorama image), the video will not include the portion that the user originally wishes to view. Thus, intuitive control cannot be performed. For this reason, as shown in <FIG>, a circumstance may arise in which a video of the "safe" and the "table" will be outside of the display screen at the bottom side of the live video.

Meanwhile, as shown in <FIG>, when an image is displayed in a looked down state, it is possible to intuitively click the intermediate position (point O shown in <FIG>) between the "safe" and the "table. " In addition, as the first panorama image has relatively less distortion of video around the point O, the direction of the optical axis precisely faces the point O. Thus, even on a live video, there is no possibility that a video of the "safe" and the "table" will be outside of the display screen.

<FIG> shows a view in which, in the case of <FIG> and <FIG>, a suspicious person is moving around the "safe" in the live video, and the first panorama image and the second panorama image are shown. When a suspicious person is moving around the "safe" in the live video, in order to pursue the suspicious person, it is necessary to change the direction of the optical axis of the camera terminal device <NUM> in accordance with the movement of the suspicious person.

In such a case, in the case of the second panorama image, the image is distorted at the intermediate position between the "safe" and the "table. " Thus, it is necessary to intentionally operate the click position in such a way that the movement direction is changed at a position immediately below the image.

Meanwhile, in the case of the first panorama image, image distortion at the intermediate position between the "safe" and the "table" is small. Thus, it is possible to, by intuitively clicking in accordance with the movement of the suspicious person while viewing the live video, change the direction of the optical axis in accordance with the movement of the suspicious person.

Note that when an area not immediately below the camera terminal device <NUM> but around the camera terminal device <NUM> is to be viewed, the second panorama image, which has less distortion, would be able to be controlled more intuitively than the first panorama image. In this embodiment, it is possible to switch between the first panorama image and the second panorama image according to circumstances, which makes it possible to improve the usability for the user according to situations. Further, when there is space in the display screen, it is possible to, by displaying both the first panorama image and the second panorama image, eliminate the need for the switch operation and thus to improve the usability.

According to the aforementioned embodiments, the imaging direction is controlled in such a way that it coincides with the center position in the rectangular panorama image. However, the imaging direction is not limited thereto, and even when the imaging direction is changed, the panorama image can remain fixed without rotation.

Although the preferred embodiments of the present invention have been described in detail with reference to the appended drawings, the present invention is not limited thereto. It is obvious to those skilled in the art that various modifications or variations are possible insofar as they are within the technical scope of the appended claims.

Claim 1:
A control device, comprising:
a processing circuit configured to generate a birds-eye view image and a wide-view image from a plurality of captured images, each captured image of the plurality of captured images being in a different viewing direction; and
a control circuit configured to receive a selection input and to select one of a wide-view image and the birds-eye view image based on the selection input, and to control display in a monitoring process, based on the selection input, of the wide-view image, the birds-eye view image and a live video,
wherein a viewing angle of the wide-view image is larger than a viewing angle of the live video being displayed,
the birds-eye view image is not rotated in response to receiving the selection input to change a viewing direction of the live video, and wherein
when the birds-eye view image is simultaneously displayed with the live video and viewing direction information of the live video, the viewing direction of the live video is selected based on the selection input and the live video includes an image among the plurality of images that make up the birds-eye view image.