Patent Description:
Conventionally, an imaging apparatus capable of performing pan/tilt control is equipped with a human figure detection function, and has an automatic tracking function of controlling pan/tilt to keep holding a subject detected from a captured image at a position close to the center of an imaging range.

Further, depending on the motion of a subject that is a tracking target, the automatic tracking function can lose track of the subject, for example, when failing to detect the subject, and the automatic tracking function has a function of searching for the subject again. <CIT> discusses a technique of searching for a tracking target by increasing an angle of view by zooming out in a case where the tracking target is lost.

However, in <CIT>, for example, in a case where the tracking target has moved away from a position at which track of the tracking target has been lost, the tracking target cannot be rediscovered just by zooming out.

<CIT> and <CIT> discuss an information processing device comprising: a presentation unit that executes processing for presenting, to a user, a list of objects recognized as a result of carrying out object recognition on a captured image; and a tracking unit that sets a tracking range for a subject corresponding to an object selected by the user from the list presented by the presentation unit, and that tracks the subject on the basis of the set tracking range.

According to a first aspect of the present invention, there is provided a control apparatus as specified in claims <NUM> to <NUM>. According to a second aspect of the present invention, there is provided a control method as specified in claim <NUM>. According to a third aspect of the present invention, there is provided a non-transitory computer readable storage medium storing a computer-executable program for causing a computer to perform an image processing method as specified in claim <NUM>.

Embodiments will be described below with reference to the attached drawings. Configurations to be described in the following embodiments are merely examples, and the embodiments are not limited to the illustrated configurations.

<FIG> is a diagram illustrating a system configuration according to a first embodiment. A system according to the present embodiment includes an imaging apparatus <NUM>, an information processing apparatus <NUM>, a display <NUM>, and a network <NUM>.

The imaging apparatus <NUM> and the information processing apparatus <NUM> are interconnected via the network <NUM>. The network <NUM> is implemented by, for example, a plurality of routers, switches, and cables compliant with a communication standard such as Ethernet (registered trademark).

The network <NUM> may be implemented by the Internet, a wired local area network (LAN), a wireless LAN, a wide area network (WAN), or the like.

The imaging apparatus <NUM> is an apparatus that captures an image, and also functions as a control apparatus that can change an imaging range by controlling at least one of pan, tilt, and zoom.

The imaging apparatus <NUM> transmits image data of a captured image, information about the date and time of capture of the image, identification information for identifying the imaging apparatus <NUM>, and information about an imaging range of the imaging apparatus <NUM> to an external apparatus such as the information processing apparatus <NUM>, via the network <NUM>. The information processing apparatus <NUM> is, for example, a client apparatus such as a personal computer on which a program for implementing a function of processing to be described below is installed. In the system according to the present embodiment, one imaging apparatus <NUM> is provided, but a plurality of imaging apparatuses <NUM> may be provided. In other words, the plurality of imaging apparatuses <NUM> may be connected to the information processing apparatus <NUM> via the network <NUM>. In this case, for example, using identification information associated with a transmitted image, the information processing apparatus <NUM> determines which one of the plurality of imaging apparatuses <NUM> has captured the transmitted image.

The display <NUM> includes a liquid crystal display (LCD) or the like, and displays an image captured by the imaging apparatus <NUM>, or the like. The display <NUM> is connected to the information processing apparatus <NUM> via a display cable compliant with a communication standard such as High-Definition Multimedia Interface (HDMI) (registered trademark). The display <NUM> and the information processing apparatus <NUM> may be disposed in a single housing.

Next, the imaging apparatus <NUM> according to the present embodiment will be described with reference to <FIG> and <FIG>. <FIG> illustrates an example of an external view of the imaging apparatus <NUM> according to the present embodiment. <FIG> illustrates an example of functional blocks of the imaging apparatus <NUM> and the information processing apparatus <NUM> according to the present embodiment. Among the functional blocks of the imaging apparatus <NUM> illustrated in <FIG>, functions such as an image processing unit <NUM>, an image analysis unit <NUM>, a system control unit <NUM>, a pan/tilt/zoom control unit <NUM>, and a communication unit <NUM> are implemented as follows. Specifically, these functions are implemented by a central processing unit (CPU) <NUM> of the imaging apparatus <NUM> executing a computer program stored in a read only memory (ROM) <NUM> of the imaging apparatus <NUM>. The ROM <NUM> and the CPU <NUM> will be described below with reference to <FIG>.

A direction in which the optical axis of a lens <NUM> points is an imaging direction of the imaging apparatus <NUM>, and a light beam that passes through the lens <NUM> is focused on an image sensor of an imaging unit <NUM> of the imaging apparatus <NUM>. A lens driving unit <NUM> includes a driving system for driving the lens <NUM>, and changes a focal length of the lens <NUM>. The lens driving unit <NUM> is controlled by the pan/tilt/zoom control unit <NUM>.

A pan driving unit <NUM> includes a mechanical drive system for performing pan operation and a motor as a drive source, and rotates the imaging direction of the imaging apparatus <NUM> in a pan direction <NUM>. The pan driving unit <NUM> is controlled by the pan/tilt/zoom control unit <NUM>.

A tilt driving unit <NUM> includes a mechanical drive system for performing tilt operation and a motor as a drive source, and rotates the imaging direction of the imaging apparatus <NUM> in a tilt direction <NUM>. The tilt driving unit <NUM> is controlled by the pan/tilt/zoom control unit <NUM>.

The imaging unit <NUM> includes an image sensor (not illustrated) such as a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. The imaging unit <NUM> generates an electrical signal by photoelectrically converting a subject image formed thereon upon passing through the lens <NUM>. The image processing unit <NUM> generates image data of a captured image by performing processing for converting the electrical signal, which is generated by a photoelectrical conversion by the imaging unit <NUM>, into a digital signal, and image processing such as compression coding processing.

The image analysis unit <NUM> performs processing for detecting a specific subject to be a tracking target from the image (i.e., the image captured by the imaging unit <NUM>) of the image data generated by the image processing unit <NUM>, using a known technology such as deep learning. For example, a human figure is detected as the specific subject. The image analysis unit <NUM> may detect a specific human figure registered beforehand as the tracking target by matching using feature data of images of facial parts. As the method for detecting the human figure that is the tracking target from the image, pattern matching may be used, or a trained model which is trained by machine learning may be used.

The pan/tilt/zoom control unit <NUM> controls pan, tilt, and zoom of the imaging apparatus <NUM> by controlling the pan driving unit <NUM>, the tilt driving unit <NUM>, and the lens driving unit <NUM> based on an instruction from the system control unit <NUM>.

The communication unit <NUM> communicates with the information processing apparatus <NUM> via an interface (I/F) <NUM> to be described below with reference to <FIG>. For example, the communication unit <NUM> transmits image data of an image captured by the imaging apparatus <NUM> to the information processing apparatus <NUM> via the network <NUM>. The communication unit <NUM> also transmits information indicating the current imaging range of the imaging apparatus <NUM>. Further, the communication unit <NUM> receives a control command, which is a command for controlling the imaging apparatus <NUM>, transmitted from the information processing apparatus <NUM>, and conveys the received control command to the system control unit <NUM>.

The system control unit <NUM> controls the entire imaging apparatus <NUM> based on processing executed by the CPU <NUM>, which will be described below with reference to <FIG>, and performs, for example, the following processing. Specifically, the system control unit <NUM> analyzes the control command for controlling the imaging apparatus <NUM> transmitted from the information processing apparatus <NUM>, and performs processing corresponding to the analyzed control command. Further, based on the position of the tracking target detected by the image analysis unit <NUM> in the image, the system control unit <NUM> calculates a change amount for each of a pan value, a tilt value, and a zoom value so that the tracking target appears in a predefined size at a central position of a captured image. It is not necessary to always calculate the change amounts for all of the pan value, the tilt value, and the zoom value. The change amount for one of these values can be calculated, or the change amount for each of two of these values can be calculated, depending on the situation. Subsequently, in the case of calculating the change amount for the pan value, the system control unit <NUM> instructs the pan/tilt/zoom control unit <NUM> to change the pan value of the imaging apparatus <NUM> by the calculated change amount for the pan value. In the case of calculating the change amount for the tilt value, the system control unit <NUM> instructs the pan/tilt/zoom control unit <NUM> to change the tilt value of the imaging apparatus <NUM> by the calculated change amount for the tilt value. In the case of calculating the change amount for the zoom value, the system control unit <NUM> instructs the pan/tilt/zoom control unit <NUM> to change the zoom value of the imaging apparatus <NUM> by the calculated change amount for the zoom value. This makes it possible for the imaging apparatus <NUM> to track the tracking target and capture an image of the tracking target.

However, depending on the motion of the tracking target, the image analysis unit <NUM> cannot detect the tracking target from the captured image, so that keeping track of the tracking target can fail. In a case where tracking of the tracking target is lost (i.e., in a case where keeping track of the tracking target has failed), the system control unit <NUM> performs a search by controlling pan, tilt, and zoom to sequentially capture images in a set search area, and resumes the tracking in a case where the tracking target is rediscovered.

The imaging range in the present embodiment is determined by the pan value, the tilt value, and the zoom value of the imaging apparatus <NUM>. The pan value is, for example, an angle of the imaging direction (optical axis) in the pan direction <NUM> of the imaging apparatus <NUM> when one of drive ends of the pan driving unit <NUM> is <NUM>°. The tilt value is, for example, an angle of the imaging direction (optical axis) in the tilt direction <NUM> of the imaging apparatus <NUM> when one of drive ends of the tilt driving unit <NUM> is <NUM>°. The zoom value of the imaging apparatus <NUM> when an image is captured by the imaging apparatus <NUM> is calculated from the focal length of the lens <NUM>.

Now, information processing by the information processing apparatus <NUM> according to the present embodiment will be described with reference to the functional blocks of the information processing apparatus <NUM> illustrated in <FIG>. Each function of the information processing apparatus <NUM> is implemented as follows, using the ROM <NUM> and the CPU <NUM> to be described with reference to <FIG>. In other words, each function illustrated in <FIG> is implemented by the CPU <NUM> of the information processing apparatus <NUM> executing a computer program stored in the ROM <NUM> of the information processing apparatus <NUM>.

A display control unit <NUM> displays an image captured by the imaging apparatus <NUM> and a graphical user interface (GUI) for setting a shot function, on the display <NUM>. An operation accepting unit <NUM> accepts information representing an operation by a user, via an input device (not illustrated) such as a keyboard, a mouse, or a touch panel. An input portion may be a button, a mouse, a joystick, or the like that receives various operations from the user. Here, for example, the display control unit <NUM> displays a GUI for setting a search area on the display <NUM>, and the operation accepting unit <NUM> accepts information representing a user operation on the GUI displayed on the display <NUM>. A system control unit <NUM> transmits a control command to the imaging apparatus <NUM> via a communication unit <NUM> in response to the operation by the user. Here, an example of the method of setting the search area will be described with reference to <FIG>. A wide-angle image <NUM> illustrated in <FIG> is an image captured in a state where the imaging apparatus <NUM> is in a wide-angle state (i.e., in a state where the zoom is in a wide end state). In place of the wide-angle image <NUM>, a panoramic image based on a plurality of images obtained by sequentially imaging the imaging range of the imaging apparatus <NUM> while changing the imaging range may be displayed. The wide-angle image <NUM> illustrated in <FIG> is displayed on the display <NUM> by the display control unit <NUM>. A search area <NUM> superimposed on the wide-angle image <NUM> indicates a search area currently set based on a user operation. An icon <NUM> is displayed at each of the four corners of the search area <NUM>, and the search area <NUM> can be changed based on a user operation of dragging the icon <NUM> with a mouse. Information about the search area <NUM> set based on the user operation of dragging the icon <NUM> is transmitted to the imaging apparatus <NUM>, and the system control unit <NUM> of the imaging apparatus <NUM> sets the search area <NUM> based on the transmitted information as a search area to be used for search processing to be described below.

The communication unit <NUM> transmits various setting commands and a control command for controlling the imaging apparatus <NUM> from the system control unit <NUM> to the imaging apparatus <NUM>, via the I/F <NUM> to be described below with reference to <FIG>. In addition, the communication unit <NUM> receives image data transmitted from the imaging apparatus <NUM>, and a response from the imaging apparatus <NUM> to a command transmitted from the information processing apparatus <NUM> to the imaging apparatus <NUM>, and transmits the received image data and the response to the system control unit <NUM>. The communication unit <NUM> also transmits information about the currently set search area to the imaging apparatus <NUM>. The imaging apparatus <NUM> acquires the information about the search area transmitted from the information processing apparatus <NUM>, and sets the search area as a search area to be used in the search processing to be described below. A storage unit <NUM> stores information about a search area, image data of an image acquired by the communication unit <NUM>, and the like.

The system control unit <NUM> generates various setting commands based on user operations accepted by the operation accepting unit <NUM> and control commands, and transmits the generated setting commands and control commands to the imaging apparatus <NUM> via the communication unit <NUM>.

Here, processing for calculating, based on the current imaging range and search area of the imaging apparatus <NUM>, a route (a search route) in the search processing to be executed in a case where keeping track of a subject has failed will be described with reference to <FIG>. An imaging range <NUM> illustrated in <FIG> is the current imaging range of the imaging apparatus <NUM>. In a case where keeping track of the subject has failed, the system control unit <NUM> of the imaging apparatus <NUM> calculates the first search route for controlling pan, tilt, and zoom to change the imaging range to an initial range <NUM> based on the search area <NUM> acquired from the information processing apparatus <NUM>. After zooming out in a range in which the tracking target can be detected, the system control unit <NUM> determines the initial range that touches a border that is the closest to the current imaging range <NUM> among the upper border, the lower border, the left border, and the right border of the search area <NUM> by changing the pan value or the tilt value. In the example illustrated in <FIG>, the upper border of the search area <NUM> is the closest to the current imaging range <NUM>, and thus the initial range <NUM> illustrated in <FIG> is selected. A route <NUM> is a route for moving from the current imaging range <NUM> to the initial range <NUM> thus determined. Next, the system control unit <NUM> identifies a border that is the farthest from the initial range <NUM> between the left border and the right border of the search area <NUM>, and changes the pan value to control the imaging range so that the identified border and the border of the imaging range touch each other. A route <NUM> is a route thus determined by the system control unit <NUM>. Subsequently, the system control unit <NUM> similarly controls the imaging range in a route <NUM> in which the right border of the imaging range touches the right border of the search area <NUM>, and the lower border of the imaging range arrives at the lower border of the search area <NUM>. In a case where the imaging range is controlled along the route <NUM>, the system control unit <NUM> changes the tilt value. Subsequently, the system control unit <NUM> controls the imaging range in a route <NUM> for a movement in which the lower border of the search area <NUM> and the lower border of the imaging range touch each other, and the left border of the imaging range arrives at the left border of the search area <NUM>. In the route <NUM>, the system control unit <NUM> changes the pan value. Subsequently, the system control unit <NUM> controls the imaging range in a route <NUM> in which the imaging range touches the left border of the search area <NUM>, and the upper border of the imaging range arrives at the upper border of the search area <NUM>. In this control, the system control unit <NUM> changes the tilt value. Subsequently, the system control unit <NUM> controls the imaging range in a route <NUM> in which the upper border of the imaging range touches the upper border of the search area <NUM>, and the imaging range arrives at the initial range <NUM>. In this control, the system control unit <NUM> issues an instruction to change the pan value to the pan/tilt/zoom control unit <NUM>. The system control unit <NUM> controls the imaging range using the search route consisting of the above-described routes <NUM> to <NUM>, and the image analysis unit <NUM> executes processing for detecting the tracking target from images sequentially captured in the meantime. The search operation ends when the tracking target is detected, and the tracking of the tracking target resumes. The image analysis unit <NUM> may detect a specific human figure registered as the tracking target using feature data of images of facial parts by matching, and the tracking operation may be resumed only when there is a match.

The above example of the search is an example in a case where the size of the imaging range in the longitudinal direction at a zoom magnification in searching for the tracking target is larger than or equal to a half of the size of the search area <NUM> in the longitudinal direction. In a case where the size of the search area <NUM> in the longitudinal direction is larger, the system control unit <NUM> repeats reciprocating from the left end to the right end of the search area <NUM> while changing the tilt value, thereby determining a search route to search the entire search area <NUM>.

Now, processing in a case where the imaging apparatus <NUM> tracks the tracking target and captures an image of the tracking target will be described with reference to a flowchart illustrated in <FIG>. None of the pan, tilt, and zoom of the imaging apparatus <NUM> is driven at the time when processing in the flowchart in <FIG> begins. The processing in the flowchart illustrated in <FIG> is executed by each of the functional blocks illustrated in <FIG> that are each implemented by the CPU <NUM> of the imaging apparatus <NUM> executing a computer program stored in the ROM <NUM> of the imaging apparatus <NUM>.

First, in step S601, the image analysis unit <NUM> detects a subject to be a tracking target from the image of the latest frame. Next, in step S602, the image analysis unit <NUM> determines whether the subject as the tracking target is included in the image. In a case where the subject as the tracking target is included (YES in step S602), the processing proceeds to step S603. In a case where the subject is not included (NO in step S602), the processing proceeds to step S606. The image analysis unit <NUM> may determine only a subject appearing in a predetermined area in the image as the tracking target. The image analysis unit <NUM> may perform matching to find a specific human figure registered as a tracking target using feature data of images of facial parts, and determine only a subject determined as the specific human figure as the tracking target.

In step S603, based on at least one of the current pan value and tilt value, and the position of the tracking target detected by the image analysis unit <NUM> in the image, the system control unit <NUM> calculates a change amount for the current pan value and/or tilt value so that the tracking target appears at a central position of the image.

Subsequently, the system control unit <NUM> instructs the pan/tilt/zoom control unit <NUM> to change the current pan value and/or tilt value by the calculated change amount for the pan value and/or tilt value. The pan/tilt/zoom control unit <NUM> thereby controls the pan driving unit <NUM> and/or the tilt driving unit <NUM> to change the current pan value and/or tilt value by the change amount as instructed by the system control unit <NUM>.

Then, in step S604, the system control unit <NUM> determines a change amount for the zoom value so that the tracking target in a predefined size appears in the image. For example, at first, the system control unit <NUM> determines whether the size of the tracking target in the image is appropriate with respect to the size of the image. For example, the system control unit <NUM> determines whether the ratio of the number of pixels in the longitudinal direction of the circumscribed rectangle of the tracking target to the number of pixels in the longitudinal direction of the captured image exceeds a first ratio (e.g., <NUM>%). As a result, in a case where the ratio of the number of pixels in the longitudinal direction of the circumscribed rectangle of the tracking target to the number of pixels in the longitudinal direction of the captured image exceeds the first ratio, the system control unit <NUM> determines that it is desirable to zoom out so that the ratio becomes less than a second ratio. Further, the system control unit <NUM> determines whether the ratio of the number of pixels in the longitudinal direction of the circumscribed rectangle of the tracking target to the number of pixels in the longitudinal direction of the captured image is less than the second ratio (e.g., <NUM>%). Here, in a case where the ratio of the number of pixels in the longitudinal direction of the circumscribed rectangle of the tracking target to the number of pixels in the longitudinal direction of the captured image is less than the second ratio, the system control unit <NUM> determines that it is desirable to zoom in so that the ratio becomes more than or equal to the second ratio. Here, the system control unit <NUM> may allow the user to set the first ratio and the second ratio beforehand.

Subsequently, the system control unit <NUM> instructs the pan/tilt/zoom control unit <NUM> to change the current zoom value of the imaging apparatus <NUM> by the determined change amount for the zoom value. The pan/tilt/zoom control unit <NUM> controls the lens driving unit <NUM> to change the zoom value by the change amount instructed by the system control unit <NUM>.

Then, in step S605, the system control unit <NUM> determines the size of the tracking target in a captured video image after the zoom control based on the size of the tracking target detected in step S601 and the change amount for the zoom value in step S604. For example, suppose zoom-in is performed in step S604, and the change amount for the zoom value at this moment is <NUM> times. In this case, if the size of the tracking target in the longitudinal direction before the zoom-in is <NUM>% of the size of the captured image in the longitudinal direction, the height of the tracking target in the captured image after the zoom-in is <NUM>% of the size of the image in the longitudinal direction. Further, suppose zoom-out is performed in step S604, and the change amount for the zoom value at this moment is a value for enlarging an angle of view to <NUM>% in length and breadth. In this case, if the size of the tracking target in the longitudinal direction before the zoom-out is <NUM>% of the size of the captured image in the longitudinal direction, the size of the tracking target in the longitudinal direction in the captured image after the zoom-out is <NUM>% of the size of the captured image in the longitudinal direction.

Subsequently, the system control unit <NUM> stores the thus determined size of the tracking target in the captured image after the zoom control into a memory of the system control unit <NUM> or an external memory (not illustrated). After step S605, the processing returns to step S601.

In step S606, in a case where the state of detecting the tracking target is changed to the state of not detecting the tracking target (in a case where keeping track of the tracking target has failed), the system control unit <NUM> executes processing (search processing) of searching for the lost tracking target. The processing in step S606 will be described in detail below with reference to a flow in <FIG>.

In step S607, the system control unit <NUM> determines whether the tracking target is detected (found) by the image analysis unit <NUM> from the image by the search processing, and in a case where the tracking target is detected (YES in step S607), the processing proceeds to step S603. On the other hand, in a case where the tracking target is not detected by the image analysis unit <NUM> from the image by the search processing (NO in step S607), the processing proceeds to step S608.

In step S608, the system control unit <NUM> erases the size of the tracking target recorded in the memory. In step S609, the system control unit <NUM> instructs the pan/tilt/zoom control unit <NUM> to change the current values to the pan value, the tilt value, and the zoom value of the initial imaging range set beforehand.

Here, the search processing in step S606 will be described in detail with reference to <FIG>. Processing in a flowchart illustrated in <FIG> is executed by each of the functional blocks illustrated in <FIG> that are each implemented by the CPU <NUM> of the imaging apparatus <NUM> executing a computer program stored in the ROM <NUM> of the imaging apparatus <NUM>.

In step S701, the image analysis unit <NUM> executes detection processing for detecting the tracking target from the captured image for a predetermined time. The predetermined time may be the number of seconds set beforehand by the user.

Next, in step S702, the image analysis unit <NUM> determines whether the tracking target is detected (found) by the image analysis unit <NUM>, and in a case where the tracking target is detected (YES in step <NUM>), the subject search processing illustrated in <FIG> ends, and the processing proceeds to step S607. On the other hand, in a case where the tracking target is not detected by the image analysis unit <NUM> (NO in step S702), the processing proceeds to step S703.

In step S703, the image analysis unit <NUM> reads the size recorded in the memory in step S605, i.e., the size of the tracking target on the image immediately before track of the tracking target is lost. Then, in step S704, the image analysis unit <NUM> performs the following processing. Suppose the tracking target having the size read in step S703 appears on the image of the current frame. The image analysis unit <NUM> calculates to what degree zoom-out can be performed in a range of zoom values by which the tracking target can be detected in this state. For example, the image analysis unit <NUM> determines that highly accurate detection is possible if the size of the tracking target in the longitudinal direction detected from the image is larger than or equal to <NUM>% of the size of the image in the longitudinal direction, and this is determined as a minimum detection size. In this process, if zoom-out is performed so that the tracking target having the height of the same size as the minimum detection size appears, detecting the tracking target fails when the tracking target becomes smaller than the minimum detection size in a case where the tracking target moves away from the imaging apparatus <NUM>. Thus, the system control unit <NUM> calculates a zoom-out amount so that the tracking target appears in a size equal to <NUM>%, which is <NUM> times the minimum detection size. Further, the system control unit <NUM> instructs the pan/tilt/zoom control unit <NUM> to zoom out by the calculated zoom-out amount, so that the lens driving unit <NUM> performs the zoom-out.

In step S705, the system control unit <NUM> reads position information about the search area transmitted from the information processing apparatus <NUM> from a memory or the like. The position information about the search area has been transmitted from the information processing apparatus <NUM> to the imaging apparatus <NUM> and stored in a memory of the imaging apparatus <NUM> beforehand.

In step S706, the system control unit <NUM> determines a search route for controlling the pan/tilt to sequentially capture images in the search area in the shortest path based on the current imaging range and the set search area.

In step S707, the system control unit <NUM> instructs the pan/tilt/zoom control unit <NUM> to control the imaging range along the search route determined in step S706. The pan/tilt/zoom control unit <NUM> thereby controls the pan driving unit <NUM> and/or the tilt driving unit <NUM> to control the imaging range to be along the search route. In the example illustrated in <FIG>, the imaging range is controlled starting from the initial range <NUM> along the search route consisting of the routes <NUM> to <NUM>.

In step S708, whether the tracking target (the subject) is detected by the image analysis unit <NUM> during the movement is determined. In a case where the tracking target is detected by the image analysis unit <NUM> from the captured image in the process of controlling the imaging range along the search route in step S707 (YES in step S708), the processing proceeds to step S709. On the other hand, in a case where the tracking target is not detected by the image analysis unit <NUM> from the captured image in the process of controlling the imaging range along the search route (NO in step S708), the processing proceeds to step S710.

In step S709, the system control unit <NUM> instructs the pan/tilt/zoom control unit <NUM> to stop the control of the imaging range along the search route in step S707. After step S709, the processing proceeds to step S607 illustrated in <FIG>. In step S710, the system control unit <NUM> determines whether the number of times the processing (the processing of controlling the imaging range along the search route) in step S707 has been performed is more than or equal to a first threshold (a threshold for giving up the search processing and stopping the search processing). In a case where the number of times the processing in step S707 has been performed is more than or equal to the first threshold (YES in step S710), the search processing ends, and the processing proceeds to step S607. In a case where the number of times the processing (the processing of controlling the imaging range along the search route) in step S707 has been performed is less than the first threshold (NO in step S710), the processing proceeds to step S711.

In step S711, the system control unit <NUM> determines whether the number of times the processing (the processing of controlling the imaging range along the search route) in step S707 has been performed is more than or equal to a second threshold (a threshold for suspending the search and returning to a home position). In a case where the number of times is determined to be more than or equal to the second threshold (YES in step S711), the processing proceeds to step S712 to temporarily return the imaging range to the home position. On the other hand, in a case where the number of times is determined to be less than the second threshold (NO in step S711), the processing returns to step S707, so that the processing of controlling the imaging range along the search route is executed again. At this moment, a count value of the number of times the processing in step S707 has been executed is incremented by <NUM>.

In step S712, the system control unit <NUM> instructs the pan/tilt/zoom control unit <NUM> to move to the home position (the initial imaging range) determined by the pan value, the tilt value, and the zoom value set beforehand by the user. Here, the home position may be set beforehand by the user, or a range in which the tracking target is likely to be present may be calculated from a movement history up to this point of the tracking target, and an angle of view for capturing the calculated range may be determined as the home position.

In step S713, the image analysis unit <NUM> performs the detection processing for detecting the tracking target on images sequentially captured for a predetermined time. In step S714, whether the tracking target is detected by the image analysis unit <NUM> is determined, and if the tracking target is detected (YES in step S714), the subject search processing ends, and the processing proceeds to step S607 illustrated in <FIG>. On the other hand, in a case where the tracking target is not detected by the image analysis unit <NUM> (NO in step S714), the processing proceeds to step S707, so that the processing of controlling the imaging range along the search route is executed again to search for the tracking target again. At this moment, the count value of the number of times the processing in step S707 has been executed is incremented by <NUM>.

As described above, the imaging apparatus <NUM> according to the present embodiment executes the processing of searching for the tracking target while controlling the imaging range for the entire search area set by the user as the range in which the tracking target can be present in a case where detection of the tracking target has failed. This can increase the possibility of discovering the tracking target again even in a case where keeping track of the tracking target has failed.

In a second embodiment, in a case where keeping track of a tracking target has failed, the tracking target is searched for by sequentially capturing images by pan/tilt/zoom control in a set search area, in a manner similar to the first embodiment. After that, the tracking target found by the search is tracked and imaged. In addition, in the present embodiment, the search area is updated to include a position of the tracking target detected outside the search area in a case where the tracking target has been tracked in a place outside the set search area during tracking of the tracking target. This makes it possible to include all the places where the tracking target has been present in the past in the search area, thereby making it easy to search a place where the tracking target is likely to be present during a subject search. Moreover, the search area can be appropriately updated as a range where the subject can be present even in a case where a user cannot accurately set the search area. A part different from the first embodiment will be mainly described below. Components and processes identical or equivalent to those of the first embodiments are assigned the same reference numerals as those of the first embodiment, and the description thereof will be omitted.

Here, tracking imaging operation for the tracking target by an imaging apparatus <NUM> will be described with reference to a flowchart in <FIG>. Processing in a flowchart illustrated in <FIG> is executed by each of functional blocks illustrated in <FIG> that are each implemented by a CPU <NUM> of the imaging apparatus <NUM> executing a computer program stored in a ROM <NUM> of the imaging apparatus <NUM>. Step S601 to step S609 in <FIG> are similar to those described with reference to <FIG> in the first embodiment, and thus the description thereof will be omitted.

In step S810, a system control unit <NUM> determines whether the position of the tracking target detected by an image analysis unit <NUM> is out of the currently set search area. In a case where it is determined that the position of the detected tracking target is out of the search area (YES in step S810), the operation proceeds to step S811 to enlarge the search area. On the other hand, in a case where the position of the detected tracking target is within the currently set search area (NO in step S810), the operation returns to step S601.

In step S811, the system control unit <NUM> updates the setting of the search area to include the current position of the tracking target. Here, processing of updating the search area will be described with reference to <FIG>. An imaging range <NUM> is the current imaging range of the imaging apparatus <NUM>, and a search area <NUM> is the currently set search area. A circumscribed rectangle <NUM> is the circumscribed rectangle of a human figure detected by the image analysis unit <NUM> as the tracking target, and indicates the position of the human figure. In the example illustrated in <FIG>, the circumscribed rectangle <NUM> of the tracking target is present outside the search area <NUM>. In this case, the system control unit <NUM> updates the search area <NUM> to include the circumscribed rectangle <NUM> located outside the search area <NUM>. Specifically, a border <NUM>, which is the right border of the circumscribed rectangle <NUM>, is located to the right of a border <NUM>, which is the right border of the search area <NUM>, and thus the system control unit <NUM> determines that it is necessary to enlarge the search area <NUM> in the right direction. In this case, the system control unit <NUM> enlarges the search area <NUM> by moving the position of the right border <NUM> of the search area <NUM> to the position of the right border <NUM> of the circumscribed rectangle <NUM> so that the search area <NUM> contains the circumscribed rectangle <NUM>. A range of enlargement in this process corresponds to a direction and a distance indicated by an arrow <NUM>. While the case where the range is enlarged in the right direction is described as an example, the system control unit <NUM> enlarges the search area <NUM> by moving the left border of the search area <NUM> in the left direction in a case where the left border of the circumscribed rectangle <NUM> is located to the left of the left border of the search area <NUM>. Similarly, the system control unit <NUM> enlarges the search area in the upper direction based on a comparison between the upper borders, and in the lower direction based on a comparison between the lower borders.

The above-described operation makes it possible to enlarge the search area to contain the current position of the tracking target and thereby update the setting. The system control unit <NUM> may transmit information about the updated search area to an information processing apparatus <NUM>, and a display control unit <NUM> of the information processing apparatus <NUM> may superimpose the updated search area on a wide-angle image <NUM> and display the resultant image on a display <NUM>.

In this way, in the present embodiment, it is possible to include all the areas where the tracking target has been present in the past in the search area when performing a search using pan/tilt/zoom driving for the entire range where the target can be present, in a case where detection of the tracking target has failed. This increases the possibility that an area where the tracking target is present can be searched.

In a third embodiment, a search area is automatically set from a trail of pan/tilt/zoom control manually operated by a user. The search area can be thereby set in a simplified manner. A part different from the first embodiment will be mainly described. Components and processes identical or equivalent to those of the first embodiments are assigned the same reference numerals as those of the first embodiment, and the description thereof will be omitted. In particular, processing of controlling an imaging range while tracking a tracking target and search processing in a case where keeping track of the tracking target has failed are similar to those in the first embodiment, and thus the description thereof will be omitted.

In the present embodiment, search area setting processing is executed before start of processing similar to the processing of the flowchart illustrated in <FIG> in the first embodiment. A system control unit <NUM> starts the search area setting processing upon receiving a request command for setting the search area from an information processing apparatus <NUM> based on a user operation. Here, the search area setting processing will be described with reference to a flowchart illustrated in <FIG>. The processing in the flowchart illustrated in <FIG> is executed by each of functional blocks illustrated in <FIG> that are each implemented by a CPU <NUM> of the imaging apparatus <NUM> executing a computer program stored in a ROM <NUM> of an imaging apparatus <NUM>.

First, in step S1001, the system control unit <NUM> acquires a control command for controlling at least one of a pan value, a tilt value, and a zoom value (i.e., a control command for controlling an imaging range) transmitted from the information processing apparatus <NUM>. An operation accepting unit <NUM> accepts a user operation for changing the pan value, the tilt value, and the zoom value, and in response to the acceptance, a system control unit <NUM> of the information processing apparatus <NUM> executes the following processing. Specifically, the system control unit <NUM> generates a control command for changing the current values to a pan value, a tilt value, and/or a zoom value requested by an instruction from the user, and the generated control command is transmitted to the imaging apparatus <NUM>. In step S1001, the system control unit <NUM> acquires the control command thus transmitted. The system control unit <NUM> instructs a pan/tilt/zoom control unit <NUM> to change the current values to the pan value, the tilt value, and/or the zoom value requested by the instruction from the user based on the acquired control command. The pan/tilt/zoom control unit <NUM> controls a pan driving unit <NUM>, a tilt driving unit <NUM>, and/or a lens driving unit <NUM>, based on the instruction.

In step S1002, the system control unit <NUM> generates history information indicating time-series transition of the pan value, the tilt value, and the zoom value of the imaging apparatus <NUM> by user operations, and stores the generated history information in a memory.

In step S1003, the system control unit <NUM> determines whether an instruction command for ending the search area setting processing is received from the information processing apparatus <NUM>. In a case where the command is received (YES in step S1003), the processing proceeds to step S1004. In a case where the command is not received (NO in step S1003), the processing returns to step S1001.

In step S1004, based on the history information indicating the transition of the pan value, the tilt value, and the zoom value stored in step S1002, the system control unit <NUM> sets a range that has been included at least once in the imaging range of the imaging apparatus <NUM> controlled by the user operation as the search area. Here, the search area setting processing will be described with reference to <FIG>. An area <NUM> indicates an imaging location, and a range <NUM> indicates a range that has been included at least once in the imaging range based on the pan value, the tilt value, and the zoom value requested by the user. A trail <NUM> is a trail along which the imaging range has moved by user operations.

Here, if the range <NUM> is set as it is as the search area, the search area has a complicated shape, so that a search route in searching for the tracking target is complicated. Thus, a range <NUM> (hereinafter referred to as a search area <NUM>), which is the circumscribed rectangle of the range <NUM>, is set as the search area. The left border of the search area <NUM> is determined to pass through a point <NUM>, which is the left end of the range <NUM>. Similarly, the right border of the search area <NUM> is determined to pass through a point <NUM>, which is the right end of the range <NUM>. Further, the upper border of the search area <NUM> is determined to pass through a point <NUM>, which is the upper end of the range <NUM>, and the lower border of the search area <NUM> is determined to pass through a point <NUM>, which is the lower end of the range <NUM>. The example in which the search area is rectangular is described above, but a search area having a complicated shape may be set, and the search area may be searched. The search route may be determined from the history information about the transition of the pan value, the tilt value, and the zoom value stored in step S1002, not by using the method illustrated in <FIG> of the first embodiment. In this case, the search processing is executed based on a search route along the trail <NUM>.

There is described the example in which the pan/tilt/zoom operation for setting is performed by the user after the imaging apparatus <NUM> has accepted the request for setting the search area from the information processing apparatus <NUM>. Alternatively, history information about the transition of the pan value, the tilt value, and the zoom value by the user at the time of normal imaging may be generated, and the search area may be determined based on the history information.

As described above, the imaging apparatus <NUM> according to the present embodiment sets the search area for executing the search processing based on the pan/tilt/zoom operation by the user. The search area can be thereby set in a simplified manner.

In a fourth embodiment, in a case where keeping track of the tracking target has failed, images are sequentially captured in a set search area by pan/tilt/zoom control, and a tracking target is searched for, in a manner similar to the first embodiment. After that, the tracking target found by the search is tracked and imaged. Further, in addition to the first embodiment, information about a history (movement history) of the past positions of the tracking target is held in the present embodiment. In a case where detection of the tracking target has failed and a search is performed, a position where the tracking target has been frequently present in the past is searched first, thereby making it possible to increase the possibility that the tracking target is found quickly. A part different from the first embodiment will be mainly described. Components and processes identical or equivalent to those of the first embodiments are assigned the same reference numerals as those of the first embodiment, and the description thereof will be omitted.

A system control unit <NUM> divides the set search area evenly into a plurality of blocks.

The system control unit <NUM> counts a tracking-target presence time for a block including the central position of the detected tracking target based on a result of tracking target detection processing for an image by an image analysis unit <NUM>. The system control unit <NUM> can thereby hold a total tracking-target presence time for each of the blocks as the movement history of the tracking target.

When determining a search route for the search processing to be executed in a case where keeping track of the tracking target has failed, the system control unit <NUM> determines a search route that first searches for a block where the tracking target has been present for a long time in the past based on the tracking-target presence time for each block in the search area. Here, operation for determining the search route will be described with reference to <FIG>. An area <NUM> indicates an environment that is an imaging target, and a search area <NUM> is the currently set search area. An imaging range <NUM> is an imaging range at the time of starting the search processing due to a failure in keeping track of the tracking target. The search area <NUM> includes blocks 1204a to 1204e obtained by dividing the search area <NUM> into five blocks laterally aligned as illustrated in <FIG>. The system control unit <NUM> holds the total tracking-target presence time, which is the time during which the tracking target has been present, for each of the blocks 1204a to 1204e. Here, for example, in a case where the tracking target has been present for the longest time in the leftmost block 1204a among the blocks 1204a to 1204e, the system control unit <NUM> determines a search route that captures an image in the block 1204a first. A movement route in controlling the imaging range in this case is a route <NUM>. After that, in a case where the tracking target cannot be detected in the leftmost block 1204a, the system control unit <NUM> performs a search while controlling the imaging range in a movement route indicated as a route <NUM> to also search the blocks on the right. In the above-described example, it is described that the block division is laterally performed, but is not limited thereto. The unit of block can be changed. For example, vertical division or division into smaller units may be adopted.

Further, in the above-described example, a search order is determined based on the tracking-target presence time in the past, but there are other methods such as a method of determining the order starting from the last position at which track of the tracking target has been lost. In this case, if the last detected position of the tracking target is on the left side of the angle of view of an imaging apparatus, the left side is searched first. The search route may be determined by setting a search priority by another method using the history of the past positions of the tracking target.

In this way, in the present embodiment, an area where the tracking target is likely to be present now is estimated from the movement history of the tracking target, and the estimated area is searched first, so that the possibility that the tracking target is found in a short time can be increased.

Now, a hardware configuration of the imaging apparatus <NUM> for implementing each function in each of the embodiments will be described with reference to <FIG>. While the hardware configuration of the imaging apparatus <NUM> will be described below, the information processing apparatus <NUM> will have a similar hardware configuration.

The imaging apparatus <NUM> according to the present embodiment includes the CPU <NUM>, a random access memory (RAM) <NUM>, the ROM <NUM>, a hard disk drive (HDD) <NUM>, and the I/F <NUM>.

The CPU <NUM> controls the entire imaging apparatus <NUM>. The RAM <NUM> temporarily stores a computer program to be executed by the CPU <NUM>. In addition, the RAM <NUM> provides a work area to be used when the CPU <NUM> executes processing. The RAM <NUM> also functions as, for example, a frame memory, and as a buffer memory.

The ROM <NUM> stores a program and the like for the CPU <NUM> to control the imaging apparatus <NUM>. The HDD <NUM> is a storage device for recording image data and the like. The I/F <NUM> communicates with an external apparatus using Transmission Control Protocol/Internet Protocol (TCP/IP), Hypertext Transfer Protocol (HTTP), or the like via the network <NUM>.

In each of the above-described embodiments, the CPU <NUM> is described as executing the processing, but at least part of the processing executed by the CPU <NUM> may be performed by a dedicated hardware. For example, processing of reading a program code from the ROM <NUM> and loading the program code into the RAM <NUM> may be executed by direct memory access (DMA) that functions as a transfer apparatus.

The present invention can also be implemented by processing in which one or more processors read a program for implementing one or more functions of the above-described embodiments and execute the program. The program may be supplied to a system or apparatus having a processor via a network or storage medium.

The present invention can also be implemented by a circuit (e.g., an application specific integrated circuit (ASIC)) for implementing the one or more functions of the above-described embodiments. Each component of the imaging apparatus <NUM> may be implemented by the hardware illustrated in <FIG>, or may be implemented by software.

Another apparatus may have one or more functions of the imaging apparatus <NUM> according to each of the above-described embodiments. For example, the information processing apparatus <NUM> may have one or more functions of the imaging apparatus <NUM> according to each of the embodiments. The above-described embodiments may be freely combined.

According to the above-described embodiments, the possibility of rediscovering the tracking target can be increased even if keeping track of the subject as the tracking target has failed.

Claim 1:
A control apparatus (<NUM>) comprising:
control means (<NUM>) for controlling an imaging range of imaging means (<NUM>, <NUM>) to track a tracking target; and
setting means (<NUM>) for setting a search area based on a range set by a user, a pan/tilt/zoom operation by a user, or a past position of the tracking target,
wherein the control means (<NUM>) controls the imaging range to search for the tracking target in the search area in a case where tracking of the tracking target has failed,
characterized in that:
the control means (<NUM>) determines a route for controlling the imaging range in the search area, and controls the imaging range along the determined route,
wherein the search area is divided into a plurality of blocks, and
wherein the control means (<NUM>) determines the route based on a tracking-target presence time during which the tracking target has been present for each of the blocks.