Client device for displaying images of a controllable camera, method, computer program and monitoring system comprising said client device

Embodiments provide to a client device for displaying camera images of a controllable camera. The client device includes a screen and a display device for displaying a first image representation on the screen. The first image representation shows an actual camera image in an actual visual range of the camera. The client device also includes a selection device designed to select a desired visual range of the camera, and has a communications device designed to request and receive a desired camera image in the desired camera visual range. The display device is designed to display, in a second image representation, at least some portion of the actual camera image correctly in terms of position and size in the desired camera visual range and, in an additional image representation, to display the desired camera image correctly in terms of position and size in the desired camera visual range.

BACKGROUND OF THE INVENTION

The invention relates to a client device for displaying camera images of a controllable camera, said client device having a screen, a display device for displaying a first image representation, said first image representation showing an actual camera image in an actual visual range of the camera. The client device also has a selection device which is designed to select a desired visual range of the camera, and has a communications device which is designed to request and receive a desired camera image in the desired camera visual range. The invention also relates to a method comprising the client device, a computer program for the client device as well as a monitoring system comprising the client device.

Camera-based monitoring systems are used for securing buildings or open public spaces. Such monitoring systems often have at least one monitoring camera which can be in the form of a fixed monitoring camera. However, PTZ cameras (Pan-Tilt-Zoom cameras) are also known which can be controlled by means of a servomotor with regard to the swivel angle, the angle of inclination and a magnification setting.

Such PTZ cameras are not only known in the field of monitoring systems. Hence, the WIPO patent publication WO 2011/046448 A1, which arguably represents the closest prior art, discloses a PTZ camera for a conference system. In the publication, the problem is discussed with regard to the fact that the request for setting the PTZ camera can be different for each conference, so that the PTZ camera has to be interactively controlled in order to accordingly be able to adapt the field of vision of the PTZ camera to the respective requests. In a possible embodiment, a touchscreen is provided, wherein the control commands for the PTZ camera are inputted via the touchscreen.

SUMMARY OF THE INVENTION

The invention relates to a client device for displaying camera images of a controllable camera. The client device can be in the form of a computer workstation, in particular in the form of a personal computer (PC). As an alternative thereto, the client device can however also be implemented as a mobile terminal, in particular as a laptop, smartphone or tablet.

The client device comprises a screen as well as a display device which is designed to actuate the screen. The display device can, for example, be designed as a data processing device, in particular as a digital data processing device. The screen is preferably an integral component of the client device. The display device is designed to display a first image representation on the screen, said first image representation being defined as the items of image information displayed on the screen. The first image representation is an actual camera image in an actual visual range of the camera. The actual camera image is preferably a current camera image of the controllable camera. The actual camera visual range is preferably defined by an orientation, a magnification and/or a viewing angle of the controllable camera.

The client device comprises a selection device, which, in particular, is designed as a human-machine interface, said selection device being designed to select a desired visual range of controllable camera. The desired visual range of the camera is achieved by means of an altered orientation and/or an altered magnification or, respectively, an altered viewing angle of the camera.

In addition, the client device comprises a communications device which is designed to request and receive a desired camera image in the desired camera visual range. Whereas the actual camera image is the camera image which is captured by the camera in the actual visual range of the camera, the desired camera image is the camera image that is captured by the camera in the desired camera visual range.

It is further proposed within the scope of the invention that the display device is designed to display, in a second image representation, at least some portion of the actual camera image correctly in terms of position and size in the desired camera visual range and, in an additional image representation, to display the desired camera image correctly in terms of position and size in the desired camera visual range.

After or when selecting the desired camera visual range, the client device switches the screen thereof into a second or additional image representation, wherein camera images or portions thereof of the desired camera visual range are displayed. These camera images are however, in fact, not or not completely present on the client device because the camera first has to orient itself accordingly or has to change the viewing angle/the magnification thereof. In this case, the actual camera image is displayed correctly in terms of position and size in the desired camera visual range. The actual camera image is thus displaced and/or scaled, i.e. magnified or reduced, on the screen in such a manner that at least portions of the actual camera image are displayed correctly in terms of position and size in the desired camera visual range now to be displayed. The remaining surface of the desired camera visual range can, for example, remain empty or be displayed in a covering color.

At a later point in time, the desired camera image is delivered via the communications device, said desired camera image filling the desired camera visual range to one hundred percent in the ideal case. In an additional image representation, the desired camera image is therefore displayed correctly in terms of position and size in the desired camera visual range. During the transition from the second image representation to the additional image representation, stationary, substantively identical image regions of the actual camera image and the desired camera image congruently overlap on the screen.

The advantage of the client device is particularly apparent by the fact that said device reacts very quickly to a selection with the selection device. Hence, an integration of the actual camera image into the desired camera visual range can be performed automatically by the client device without having to request and wait for further data from the controllable camera. Hence, the response time of the client device to generate the second image representation depends solely upon the computing power of said client device itself Due to the fact that the computing capacities of mobile terminals or other data processing devices are now very high, the second image representation can be performed almost or completely without delay in the perception of the user.

In contrast, a certain temporal delay or latency usually occurs as a result of requesting and receiving the desired camera image. This temporal delay particularly occurs because the controllable camera has to first adjust itself such that said camera takes up the desired camera visual range. For this reason, the additional image representation comprising the desired camera image first occurs at a later point in time. Nevertheless, the interactive process of the client device produces a direct link between the action of the operator via the selection device and the display via the display device and additionally a direct link for controlling the controllable camera and thus facilitates a very intuitive, exact and comfortable operation as well as actuation of the controllable camera.

Provision is made in a preferred modification to the invention for the display device to be designed to display, in at least one intermediate image representation upstream of the additional image representation, an intermediate camera image in an intermediate camera visual range correctly in terms of position and size in the desired camera visual range. This modification is based on the consideration that, when requesting the desired camera image in the desired camera visual range, the camera has to first be motorically moved. Depending on the actuator of the controllable camera, this process can take up a certain amount of time, wherein one or a plurality of camera images is captured during this period of time. These intermediate camera images are however not discarded but displayed by the display device on the screen, however once again only the partial sections of the intermediate camera images which can be displayed correctly in terms of position and size in the desired camera visual range. In this modification, the image representation on the screen is therefore continually updated; thus enabling camera images that are always current to be displayed. The desired camera visual range on the screen is in the majority of cases, starting from the actual image via one or a plurality of intermediate camera images, continuously further filled, and in fact always with current camera images; thus enabling the maximum available image information in the selected desired camera visual range to be displayed to the user.

In a preferred embodiment of the invention, the selection device is designed to implement the selection of the desired camera visual range by interactively moving a selected pixel of the actual camera image. In a first step, a pixel (or an image region comprising the pixel) of the actual camera image is thus defined by means of the selection device. In a second step, the selected pixel of the actual camera image is displaced. Provision is thereby made for the selection device to be designed such that the actual camera image is displaced jointly with the selected pixel. It is thereby ensured that at least some portion of the actual camera image is displayed in the desired camera visual range. The selected pixel can, for example, be selected and interactively displaced using a computer mouse or by operating a touchscreen. When interactively moving the selected pixel as well as the actual camera image, a transition is thus created from the first image representation to the second image representation. It is optionally possible for the actual camera image to be replaced by a succeeding intermediate camera image or even by a desired camera image already during the displacement of said selected pixel as well as said actual camera image. The replacement however takes place in such a manner that the currently displayed camera image is always disposed correctly in terms of position and size with respect to the current desired camera visual range and/or that the selected pixel is disposed at the position selected by the selection device.

In a modification or an alternative of the invention, the selection device is designed to select the desired camera visual range by interactively moving two selected pixels of the active camera image. In this way, the actual camera image is scaled as a function of the position of the two selected pixels in order to display at least some portion of the actual camera image in the desired camera visual range. By means of the so-called “pinching”, a zooming of the view is, for example implemented on smart phones. A transition from an actual camera visual range to a desired camera visual range likewise occurs when zooming, said transition—insofar that it relates to a symmetrical zooming—being implemented solely by means of a change in the viewing angle. When zooming, which likewise entails the interactive displacement of two selected pixels, the viewing angle as well as the orientation of the camera is changed in order to be able to produce a corresponding desired camera image.

It is also preferred in this change from the actual camera visual range to the desired camera visual range that the pixels selected by the selection device are disposed in the desired camera visual range at the positions selected by the selection device. It is thereby ensured that the user can always interactively operate the client device in a simple manner.

In a preferred implementation of the invention, the communications device is designed to receive the actual camera image, the intermediate camera image and the desired camera image as camera images together in each case with metadata, wherein the metadata facilitate an integration of the camera images into a common reference system. This implementation is based on the consideration that the selection of the desired camera visual range as well as the integration of the camera images into the desired camera visual range is dramatically simplified by the provision of metadata and the use of a common reference system. The metadata are particularly in the form of position specifications in the reference system and/or as setting parameters, in particular swivel angles, angles of inclination and viewing angles of the controllable camera.

In a possible embodiment of the invention, the reference system is designed as a flat 2D reference system. Two opposite corner coordinates of the camera images can, for example, be transmitted as metadata in the 2D reference system. In this embodiment, all possible camera visual ranges are therefore projected onto a two dimensional surface and the respectively current camera images are provided with the corresponding coordinates.

In a preferred embodiment of the invention, a 3D reference system or a polar coordinate system is however used, wherein the metadata comprise the actual pan values, the actual tilt values and the actual zoom values of the camera or values, respectively definitions, equivalent thereto. A spherical coordinate system can particularly be used which is limited to the angle specifications.

A further subject matter of the invention relates to a method for displaying camera images of a controllable camera on a screen. According to the method, an actual camera image is displayed, in a first image representation, in an actual camera visual range on the screen. In a succeeding step, a desired camera visual range is selected; and subsequently in a second image representation, at least some portion of the actual camera image is displayed correctly in terms of position and size in the desired camera visual range and, in an additional image representation, the desired camera image of the desired camera range is displayed correctly in terms of position and size in the desired camera visual range.

A further subject matter of the invention relates to a computer program comprising program code means.

A further subject matter of the invention relates to a monitoring system which has a client device, as said device was previously described or, respectively. In addition, the monitoring system comprises at least one controllable camera, as said camera was previously described.

DETAILED DESCRIPTION

FIG. 1shows a monitoring system1in a schematic block diagram, which can be designed to monitor any monitoring area. Said monitoring system can also relate to a monitoring of a conference. The monitoring system1is however particularly intended to monitor monitoring areas against unauthorized entry, theft, in particular shoplifting etc.

The monitoring system1comprises one or a plurality of client devices2and one or a plurality of PTZ cameras (Pan-Tilt-Zoom cameras)3as controllable cameras. The PTZ camera3is connected to the client device2via a network4. The network4can also relate to a cable connected network, such as a LAN. It is however also possible for the network4to be designed as a public network, in particular as an internet system.

The PTZ camera3has an actuator which allows said PTZ camera3to swivel (pan) about an axis perpendicular to an installation, base or reference surface and to tilt about an axis which is parallel to said surface and which is oriented perpendicularly to a viewing direction of the PTZ camera3. In addition, the PTZ camera3comprises a motorically driven zoom. In other exemplary embodiments, a PT camera can also be provided instead of the PTZ camera3, i.e. without the zoom operation mode, or a monitoring camera having electronic zoom.

The client device2can be designed as a personal computer, said device is however preferably in the form of a mobile terminal device, in particular in the form of a smartphone or tablet PC. In a particularly preferred manner, the connection from the client device to the network4is implemented without cables.

The client device2has a screen5for displaying camera images of the PTZ camera. The screen5is actuated by a display device6, wherein the display device6can at the same time constitute the control device of the client device2. The client device2further comprises a selection device7which forms a human-machine interface to said client device2and is designed, for example, as a touch sensitive field on the screen5; thus enabling said touch sensitive field to be implemented as a touchscreen. In addition, the client device2has a communications device8which is designed to communicate via a network4with the PTZ camera.

The monitoring system1is again objectively depicted inFIG. 2, wherein it can be seen that the PTZ camera3can be moved about a tilt axis T that extends horizontally in the depiction and about a vertically extending pan axis P. The PTZ camera3can also be mounted in other positions, so that the tilt axis T is then parallel to a mounting plane and the pan axis P is perpendicular to the mounting plane.

The client device2is designed as a smartphone, wherein a camera image K of the PTZ camera3is displayed on the screen5. The operation of the selection device7is represented graphically which—as described above—is jointly configured with the screen5as a touch screen, wherein two different pixels B are selected using two fingers and are displaced on the screen5in order to execute a user input.

With the aid ofFIG. 3, the functional principle of the monitoring system1is to be explained. The monitoring system1, in particular the client device2, is designed to control the PTZ camera by means of a user input in the selection device in order to change the camera image K on the screen5in an operator friendly and convenient manner. The interactive control of the PTZ camera3with respect to the pan axis P and the tilt axis T takes place by a pixel B being selected on the camera image K and being displaced by means of a sliding movement on the selection device, which is embodied as a touchscreen, to a position P. The PTZ camera is subsequently oriented such that the pixel B lies at the selected position P in a succeeding camera image K′, as shown in the camera image K′.

The processes taking place in the background are illustrated in detail with the aid ofFIG. 4. The image4shows a reference system9which is designed as a coordinate system and in which all of the pixels that can be captured by the PTZ camera3can be displayed. The reference system9is designed as a 2D reference system inFIG. 4, wherein the pixels which can be captured by the PTZ camera3are projected onto a 2D surface.

An actual camera visual range IS is depicted in the reference system9, said visual range being defined in the reference system9by specifying two corner points and thus the coordinates (min X; min Y); (max X; max Y). From a physical point of view, the actual camera visual range IS is defined by specifying the swivel angle p about the swivel axis P, the tilt angle t about the tilt axis T and a zoom axis Z as well as the intrinsic camera parameters, such as focal width of the PTZ camera3etc.

A spherical coordinate system or a 3D reference system can also be selected as the reference system9instead of a Cartesian coordinate system, the reference system9forming, for example, a hemisphere, wherein the corner points of the camera visual range (for example, an actual camera visual range IS, an intermediate camera visual range ZS, and a desired camera visual range SS) are defined by angular coordinates, in particular by specifying the swivel angle p and the tilt angle t. The selection of the reference system9as a 3D reference system is mathematically more complicated; however changes due to the viewing angle can also be displayed in a more realistic manner by projecting the camera images K into the reference system.

During an interaction, such as that depicted inFIG. 3, the actual camera visual range IS is displaced by a displacement vector V to a desired camera visual range SS. The displacement vector V is identical, however oppositely directed, to the vector between the pixel B and the desired position P. The desired camera visual range has other corner coordinates in the reference system9and is also associated with other camera parameters, so that particularly the swivel angle p and the angle of inclination t of the PTZ camera3are formed differently.

An actual camera image IK is displayed in the actual camera visual range IS. After swiveling and tilting of the PTZ camera3as well as transmitting a current image, a desired camera image SK is displayed in the actual camera visual range SS.

From a realistic point of view, the PTZ camera3however requires a certain amount of time in order to reset the swivel angle p, the tilt angle t as well as, if applicable, the zoom factor Z. For this reason, it is probable that intermediate camera images ZK are transmitted in the intervening time, said intermediate camera images ZK being disposed in an intermediate camera visual range ZS in the reference system9. If the three aforementioned camera images: actual camera image IK, intermediate camera image ZK and desired camera image SK are now consecutively displayed on the screen5, a display would then result such as that inFIG. 5. If the user moved the pixel B to the position P, there is initially no change in the actual camera image IK. In the next camera image, the PTZ camera has already been drawn along so that the intermediate image ZK results, wherein the pixel B and the position P are spaced apart at a smaller distance than in the actual camera image IK, are however not congruent. The pixel B and the position P are only congruent in the desired camera image SK. Due to the latency, this display feels “viscous” for a user of the client device2until the pixel B is drawn along to the desired position.

In order to avoid this latency, the actual camera image and, as the case may be, the intermediate camera image ZK are fit correctly in terms of position and size in the desired camera visual range SS. This behavior is depicted graphically inFIG. 6. By fitting the actual camera image into the desired camera visual range SS, the pixel B and the desired position P are immediately congruent. This congruency is particularly achieved by virtue of the fact that the actual camera image is drawn along with the pixel B. This displacement (and if applicable scaling) of the actual camera image can be performed automatically by the client device2and is therefore implemented without delay.

The entire desired camera visual range SS can however not be immediately filled, and thus pixel-free regions10remain for which the actual camera image IK does not have any items of image information. The succeeding intermediate camera image ZK is also inserted correctly in terms of position and size into the desired camera visual range SS, so that the pixel-free region10is reduced. As soon as the PTZ camera3has achieved the camera parameters defined by the desired camera visual range: swivel angle p, angle of inclination t and zoom factor Z, the desired camera visual range SS is completely filled by the desired camera image SK.

It is important to stress that the desired camera visual range SS is always to be displayed on the screen5. The client device2therefore appears to react without any or almost any delay when manipulated by the user via the selection device7, wherein current items of image information for regions in the desired camera visual range SS that are not congruent with the actual camera range IS are successively loaded.

IfFIG. 1is examined again, it can be seen that request data A are transmitted to the PTZ camera3when selecting a desired camera visual range. The request data A can particularly relate to the coordinates of the desired camera visual range SS, the vector between the pixel B and the position P or even to the specification of the desired swivel angle p, angle of inclination t and zoom factor Z. The PTZ camera3delivers the camera images, in particular the actual camera image IK, the intermediate camera image ZK and the desired camera image SK, wherein metadata M with respect to each camera image K are however jointly transmitted, said data facilitating an identification of the position of the camera images K in the desired camera visual range; thus enabling the camera images K to be inserted correctly in terms of position and size into the desired camera visual range.

FIG. 7schematically depicts the mode of operation of the client device2during a zoom operation by means of pulling apart two pixels B1, B2on the actual camera image onto the positions P1, P2(“pinching”). By pulling apart the pixels B1and B2onto the positions P1and P2, a zoom factor or magnification factor Z is determined.

If the reference system9inFIG. 8is examined again, a magnification of an image section represents a reduction of the desired camera visual range ZS in relation to the actual camera visual range IS, wherein the intermediate camera visual range ZS again assumes an intermediate size. When zooming as a user interaction, the actual camera image is again fit correctly in terms of position and size into the desired camera visual range SS; thus enabling an undelayed reaction of the client device2to take place.

The behavior of the client device2is particularly characterized in that a selected pixel B, B1, B2always stays under the “finger” of the user because said pixel always remains congruent with the desired position P as a result of the actual camera image IK being fit into the desired camera visual range SS.