CAPTURE CONTROL APPARATUS, CAPTURE CONTROL METHOD, AND IMAGE CAPTURE SYSTEM

Disclosed is a capture control apparatus that determines a specific subject in a captured area as a tracking target subject and controls a capture direction of a camera so as to track and capture the tracking target subject. While a video captured by the camera is selected as a specific video by an external selection apparatus, capture control apparatus does not change the tracking target subject to be tracked by the camera.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to a capture control apparatus, a capture control method, and an image capture system, and in particular to a technique to control a plurality of image capture apparatuses.

Description of the Related Art

In live distribution and recording, a multi-camera image capture system is used that dynamically selects or switches to a video to be used in broadcasting and recording (a main video), from among videos that have been captured by a plurality of cameras in parallel, with use of a video switcher, and outputs the video (Japanese Patent Laid-Open No. 2022-45529).

In such an image capture system, labor can be saved by introducing a capture control apparatus that performs centralized control on camera operations. For example, when the capture control apparatus automatically controls the operations of other cameras based on information obtained from one specific camera, diverse types of image capture can be performed even with a small number of people.

Meanwhile, it is necessary to devise a way to prevent the execution of automatic control that is against the intention of a user who dynamically selects a main video (an operator of the video switcher).

SUMMARY OF THE INVENTION

Accordingly, one aspect of the present invention provides a capture control apparatus and a capture control method that can suppress camera operations that are not intended by a user in a capture control apparatus that automatically controls the operations of a plurality of cameras.

According to an aspect of the present invention, there is provided a capture control apparatus, comprising: one or more processors that execute a program stored in a memory and thereby function as: a determination unit configured to determine a specific subject in a captured area as a tracking target subject; and a control unit configured to control a capture direction of a camera so as to track and capture the tracking target subject, wherein, while a video captured by the camera is selected as a specific video by an external selection apparatus, the determination unit does not change the tracking target subject to be tracked by the camera.

According to another aspect of the present invention, there is provided a capture control method executed by a capture control apparatus, the capture control method comprising: determining a specific subject in a captured area as a tracking target subject; and controlling a capture direction of a camera so as to track and capture the tracking target subject, wherein, while a video captured by the camera is selected as a specific video by an external selection apparatus, the determination includes making no change to the tracking target subject to be tracked by the camera.

According to a further aspect of the present invention, there is provided an image capture system, comprising: a capture control apparatus; one or more cameras whose image capture is controlled by the capture control apparatus; the selection apparatus; and a communication network via which the capture control apparatus, the cameras, and the selection apparatus are connected in a communication-enabled manner, wherein the capture control apparatus comprises: one or more processors that execute a program stored in a memory and thereby function as: a determination unit configured to determine a specific subject in a captured area as a tracking target subject; and a control unit configured to control a capture direction of each of the one or more cameras so as to track and capture the tracking target subject, wherein, while a video captured by a camera of the one or more cameras is selected as a specific video by an external selection apparatus, the determination unit does not change the tracking target subject to be tracked by the camera.

According to another aspect of the present invention, there is provided a non-transitory computer-readable medium storing a program for causing a computer to perform a capture control method comprising: determining a specific subject in a captured area as a tracking target subject; and controlling a capture direction of a camera so as to track and capture the tracking target subject, wherein, while a video captured by the camera is selected as a specific video by an external selection apparatus, the determination includes making no change to the tracking target subject to be tracked by the camera.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments will be described in detail with reference to the attached drawings. Note, the following embodiments are not intended to limit the scope of the claimed invention. Multiple features are described in the embodiments, but limitation is not made to an invention that requires all such features, and multiple such features may be combined as appropriate.

Furthermore, in the attached drawings, the same reference numerals are given to the same or similar configurations, and redundant description thereof is omitted.

First Embodiment

FIG.1is a schematic diagram showing an exemplary configuration of a multi-camera image capture system10(hereinafter simply referred to as an image capture system) according to the present embodiment. The image capture system10includes a plurality of cameras300and400ato400c, a capture control apparatus100, and a video switcher (hereinafter simply referred to as a switcher)1000. The plurality of cameras300and400ato400c, the capture control apparatus100, and the switcher1000are connected in such a manner that they can perform communication via a communication network600.

Here, an operator is present for the switcher1000. Furthermore, an operator (user) may be present also for the capture control apparatus100, but this operator is not indispensable. The operator of the switcher1000may be the user of the capture control apparatus100. The sub cameras400ato400cdo not require a photographer because image capture thereof is controlled by the capture control apparatus100. The overhead camera300does not require a photographer, either.

The communication network600conforms with a known wired or wireless communication standard, such as the IEEE 802.3 series and the IEEE 802.11 series. Furthermore, each of the plurality of cameras300and400ato400c, the capture control apparatus100, and the switcher1000include a communication interface that conforms with the standard of the communication network600.

Although the illustration ofFIG.1indicates that all signals are communicated via the communication network600, for example, video signals and control signals may be communicated using different methods. For example, each of the plurality of cameras300and400ato400cmay supply video signals directly to the capture control apparatus100or the switcher1000via a cable.

Among the plurality of cameras300and400ato400c, the camera300captures an entirety of a predetermined captured area20. The captured area20is set as, for example, an area in which subjects to be captured can exist in a studio. Therefore, all of the subjects in the captured area are captured in a video of the camera300.

The purpose of the camera300is to capture images for detecting subjects to be captured that exist in the captured area. Therefore, the capture direction and the angle of view of the camera300are determined in accordance with the position and the captured area20of the camera300, and are basically fixed during image capture. Furthermore, it is desirable that the entirety of the captured area20be captured by the camera300without being hidden by an object outside the captured area20. For this reason, here, the camera300is placed at a position where an overhead view of the entirety of the captured area20is attained. Hereinafter, the camera300will be referred to as an overhead camera for distinction from other cameras400ato400cwhose capture directions and angles of view are not basically fixed during image capture. However, the position of placement of the camera300is not limited to a position where an overhead view of the captured area is attained. The operations of the overhead camera300can be controlled from the capture control apparatus100.

The cameras400ato400care, for example, PTZ cameras, and the operations thereof, including the capture directions (pan and tilt angles) and angles of view (zooming), can be controlled by the capture control apparatus100. Also, the cameras400ato400cmay be configured in such a manner that the capture directions (pan and tilt angles) thereof can be controlled by attaching camera bodies to camera platforms. Furthermore, the cameras400ato400cmay be configured in such a manner that exchangeable lenses capable of zooming are mounted on the camera bodies. Hereinafter, the cameras400ato400cwill be referred to as sub cameras. Also, hereinafter, the sub cameras400ato400cwill be collectively referred to as sub cameras400. The number of the sub cameras400is not limited to three; it may be equal to or smaller than two, or may be equal to or larger than four.

The switcher1000is a selection apparatus that receives video signals from each of the sub cameras400ato400c, selects a video from one sub camera designated by the user, and outputs the video as a main video.

The capture control apparatus100detects subjects from video signals received from the overhead camera300. Based on the detection result, the capture control apparatus100determines subjects to be tracked by the respective sub cameras400ato400c. Then, the capture control apparatus100determines target capture directions (and also target angles of view, if necessary) of the respective sub cameras400ato400cso that the sub cameras400ato400crespectively capture the determined subjects. The capture control apparatus100transmits control commands including the determined target capture directions to the sub cameras400ato400c. The capture control apparatus100continuously executes the subject detection, the decision on target capture directions, and the transmission of control commands; in this way, each of the sub cameras400ato400ccan be controlled so that it tracks and captures a specific subject.

(Exemplary Functional Configuration of Each Apparatus)

FIG.2is a block diagram showing an exemplary functional configuration of each device that composes a multi-camera image capture system shown inFIG.1. Note that the constituents represented as functional blocks in the figure can be realized by an integrated circuit, such as an ASIC and an FPGA, a discrete circuit, or a combination of a memory and a processor that executes a program stored in the memory. Also, one functional block may be realized by a plurality of integrated circuit packages, or a plurality of functional blocks may be realized by one integrated circuit package. Furthermore, the same functional block may be implemented as different constituents depending on the operating environment, required capabilities, and so on.

First, an exemplary functional configuration of the capture control apparatus100will be described. The capture control apparatus100may be, for example, a general-purpose computer device, such as a personal computer and a workstation. The capture control apparatus100is configured in such a manner that a CPU101, a RAM102, a ROM103, an inference unit104, a network interface (I/F)105, a user input I/F106, a video input unit107, and a display unit108are connected to one another via an internal bus110.

The CPU101is a microprocessor capable of executing programmed instructions. The CPU101realizes the functions of the capture control apparatus100, which will be described later, by reading a program stored in the ROM103into the RAM102and executing the program, for example. The CPU101can realize the functions of the capture control apparatus100by, for example, executing a capture control application that operates on base software (OS).

The RAM102is used to load the program executed by the CPU101, and to temporarily store data that is processed by the CPU101, data that is currently processed, and so forth. Also, a part of the RAM102may be used as a video memory for the display unit108.

The ROM103is a nonvolatile rewritable memory, and stores the program (OS and application) executed by the CPU101, user data, and so forth.

The inference unit104executes processing for detecting subject regions using a machine learning model with respect to a video of the overhead camera300. The inference unit104can be implemented using a hardware circuit that can execute computation of the machine learning model at high speed, such as a graphics processing unit (GPU) and a neural network processing unit (NPU), for example. Alternatively, the inference unit104may be implemented using a reconfigurable logic circuit, such as a field-programmable gate array (FPGA). The CPU101may realize the functions of the inference unit104by executing the program.

The machine learning model may be a convolutional neural network (CNN) that has been trained in accordance with the type of subjects to be detected. It is assumed here that the inference unit104detects human body regions or human face regions as subject regions from an input image. Also, it is assumed that the inference unit104outputs, for each detected subject region, the position and the size of a rectangular region in which the subject region is inscribed, and a detection reliability degree. Note that processing for detecting different types of subject regions may be executed with respect to the same input image using a plurality of types of machine learning models. Note that the inference unit104may execute processing for detecting subject regions using a known method that does not use a machine learning model. The inference unit104can detect subject regions using, for example, a method that uses local feature amounts, such as SIFT and SURF, a method that uses pattern matching, or the like.

The network I/F105is an interface for connecting the capture control apparatus100to the communication network600. The capture control apparatus100(CPU101) can communicate with external apparatuses in the communication network600, such as the overhead camera300, the sub cameras400ato400c, and the switcher1000, via the network I/F105. Note that the capture control apparatus100may communicate with external apparatuses via another non-illustrated communication interface (a USB, Bluetooth®, or the like).

In order to communicate with each apparatus (the overhead camera300, the sub cameras400ato400c, and the switcher1000) in the communication network600, the CPU101obtains a network address of each apparatus at an arbitrary timing, and stores the same into the RAM102. Furthermore, the CPU101also obtains information of each apparatus (the type, model name, and the like of the apparatus) at an arbitrary timing (e.g., at the time of first communication), and stores the same into the RAM102. It is assumed that, as described above, the CPU101is aware of at least identification information and the types of apparatuses with respect to the overhead camera300, the sub cameras400ato400c, and the switcher1000. Note that the user may be allowed to give any name to each individual apparatus.

The user input I/F106is an interface for connecting an input device (not shown), such as a mouse, a keyboard, and a touch panel. The capture control apparatus100accepts a user instruction via the user input I/F106.

The video input unit107is an input interface for video signals. The video input unit107may be a receiver that conforms with, for example, the serial digital interface (SDI) standard or the high-definition multimedia interface (HDMI®) standard. The video input unit107receives video signals from the overhead camera300.

The display unit108is a display apparatus, such as a liquid crystal display (LCD). The display unit108displays a GUI screen provided by the OS, the capture control application, or the like.

Next, an exemplary functional configuration of the switcher1000will be described.

A CPU1001is a microprocessor capable of executing programmed instructions. The CPU1001controls the operations of each functional block and realizes the functions of the switcher1000, which will be described later, by reading a program stored in a ROM1008into a RAM1002and executing the program, for example.

The RAM1002is used to load the program executed by the CPU1001, and to temporarily store data that is processed by the CPU1001, data that is currently processed, and so forth. Also, the RAM1002may be used as a buffer for video signals received from the sub cameras400.

The ROM1008is a nonvolatile rewritable memory, and stores the program executed by the CPU1001, setting values of the switcher1000, user data, and so forth.

Video input units1003ato1003care input interfaces for video signals. The video input units1003ato1003cmay be receivers that conform with, for example, the SDI standard or the HDMI standard. The video input units1003a,1003b, and1003creceive video signals from the sub cameras400a,400b, and400c, respectively. Note, it is sufficient that the number of the video input units1003be equal to or larger than the number of the sub cameras400.

FIG.2shows a configuration in which the video input units1003and the sub cameras400are connected in one-to-one correspondence. However, video signals from one or more of the sub cameras400ato400cmay be supplied to the switcher1000via the communication network600. In this case, the switcher1000receives the video signals via a network I/F1007. Note that a sub camera400that supplies video signals to the video input unit1003and a sub camera400that supplies video signals via the communication network600may coexist.

A video switch control unit1004selects video signals selected via a user input I/F1006as a specific video from among a plurality of video signals that have been input to the video input units1003ato1003c. Here, the video switch control unit1004can select a main video and a preview video as the specific video. Also, the video switch control unit1004outputs the video signals selected as the main video to a video output unit1005. The preview video is a video that is scheduled to be used as a main video next.

The video output unit1005is an interface that outputs the video signals selected by the video switch control unit1004to the outside (e.g., a live distribution device or a program recording apparatus). The video output unit1005may be a transmitter that conforms with, for example, the SDI standard or the HDMI standard.

A monitor output unit1009generates a video for a screen on which the video signals from the sub cameras400ato400c, which have been received via the video input unit1003or the network I/F1007, are displayed in an aligned manner, and outputs the video to a non-illustrated external display apparatus. Furthermore, the monitor output unit1009also outputs a video for a setting screen of the switcher1000to the external display apparatus.

The user input I/F1006is, for example, an interface for connecting an input device (not shown), such as a button, a dial, a joystick, and a touch panel. The switcher1000accepts a user instruction related to selections of a main video and a preview video and settings of the switcher1000via the user input I/F1006. The preview video is a video that is scheduled to be used as a main video next.

Based on the user instruction via the user input I/F106, the CPU1001categorizes each of the states of the sub cameras400ato400cinto one of the following: distributing, previewing, and standing by. Then, the CPU1001stores identification information pieces and the determined states of the respective sub cameras400ato400cinto the RAM1002in association with each other. Hereinafter, information indicating the state of each sub camera will be referred to as camera state information STREAMING. The CPU1001updates camera state information pieces STREAMING, for example, each time the selected video signals have changed.

Specifically, the CPU1001determines that the state of a sub camera that is capturing a video currently output from the video output unit1005(a main video) is “distributing”. Also, the CPU1001determines that the state of a sub camera that is capturing a preview video is “previewing”. Furthermore, the CPU1001determines that the state of a sub camera that is capturing another video is “standing by”. Identification information of a sub camera400may be any information that allows each individual sub camera to be identified, such as a unique name given by the user, a manufacturing number, and a network address.

Based on the camera state information pieces STREAMING, the CPU1001transmits control signals (tally information) for performing tally display to sub cameras that are distributing and previewing via the network I/F1007.

The network I/F1007is an interface for connecting the switcher1000to the communication network600. The switcher1000(CPU1001) can communicate with external apparatuses in the communication network600, such as the overhead camera300, the sub cameras400ato400c, and the capture control apparatus100, via the network I/F1007. Note that the switcher1000may communicate with the external apparatuses via another non-illustrated communication interface (a USB, Bluetooth, or the like).

Next, an exemplary functional configuration of the overhead camera300will be described.

A CPU301is a microprocessor capable of executing programmed instructions. The CPU301controls the operations of each functional block and realizes the functions of the overhead camera300, which will be described later, by reading a program stored in a ROM303into a RAM302and executing the program, for example.

The RAM302is used to load the program executed by the CPU301, and to temporarily store data that is processed by the CPU301, data that is currently processed, and so forth. Furthermore, the RAM302may be used as a buffer for video signals obtained through image capture.

The ROM308is a nonvolatile rewritable memory. The ROM308stores the program executed by the CPU301, setting values of the overhead camera300, user data, and so forth. Note that the ROM308can also be used as a recording destination of video signals. The ROM308may include a built-in memory and an attachable/removable memory card.

An image sensor307includes an image capture optical system and an image sensor. The image sensor may be a known CCD or CMOS color image sensor that includes, for example, color filters based on the primary-color Bayer arrangement. The image sensor includes a pixel array in which a plurality of pixels are two-dimensionally arrayed, and peripheral circuits for reading out signals from each pixel. Each pixel accumulates charges corresponding to the amount of incident light by way of photoelectric conversion. Signals with a voltage corresponding to the amount of charges accumulated in an exposure period are read out from each pixel; as a result, a group of pixel signals (analog image signals) representing a subject image formed on an image plane is obtained.

An image processing unit306applies predetermined signal processing and image processing to the analog image signals output from the image sensor307, thereby generating signals and image data that suit an intended use, and obtaining and/or generating various types of information.

The processing applied by the image processing unit306can include, for example, preprocessing, color interpolation processing, correction processing, detection processing, data editing processing, evaluation value calculation processing, special effects processing, and so forth.

The preprocessing can include A/D conversion, signal amplification, reference level adjustment, defective pixel correction, and so forth.

The color interpolation processing is processing which is executed in a case where the image sensor307includes color filters, and in which the values of color components that are not included in the individual pieces of pixel data that compose image data are interpolated. The color interpolation processing is also called demosaicing processing.

The correction processing can include such processing as white balance adjustment, tone correction, correction of image deterioration caused by optical aberration of the image capture optical system (image recovery), correction of the influence of vignetting of the image capture optical system, and color correction.

The data editing processing can include such processing as cutout of a region (cropping), composition, scaling, encoding and decoding, and generation of header information (generation of a data file). The data editing processing also includes generation of video signals to be output to the switcher1000, and video data to be recorded into the ROM308.

The evaluation value calculation processing can include such processing as generation of signals and evaluation values used in automatic focus detection (AF), and generation of evaluation values used in automatic exposure control (AE). The CPU301executes AF and AE.

The special effects processing can include such processing as addition of blur effects, alteration of shades of colors, relighting, and so forth.

Note that these are examples of processing that can be applied by the image processing unit306, and are not intended to limit processing applied by the image processing unit306.

The image processing unit306outputs information and data that have been obtained or generated to the CPU301, the RAM302, a video output unit311, and the like in accordance with an intended use.

Note that the types and settings of processing applied by the image processing unit306can be controlled by transmitting commands from the capture control apparatus100to the overhead camera300.

A network I/F305is an interface for connecting the overhead camera300to the communication network600. The overhead camera300(CPU301) can communicate with external apparatuses in the communication network600, such as the capture control apparatus100, the switcher1000, and the sub cameras400ato400c, via the network I/F305. Note that the overhead camera300may communicate with external apparatuses via another non-illustrated communication interface (a USB, Bluetooth, or the like).

The video output unit311is an interface for outputting video signals that have been captured using the image sensor307to an external apparatus (here, the capture control apparatus100). The video output unit311may be a transmitter that conforms with, for example, the SDI standard or the HDMI standard.

In a case where monitor output has been set to be enabled and a non-illustrated external display apparatus is connected to the overhead camera300, a monitor output unit312generates a video for a screen that displays video signals obtained through image capture, and outputs the video to the external display apparatus.

Next, an exemplary functional configuration of the sub cameras400ato400cwill be described. Note, it is assumed here that the sub cameras400ato400chave the same functional configuration, and the following describes an exemplary functional configuration of the sub camera400aas a representative.

It is assumed that the functional blocks of the sub camera400aand the overhead camera300with the same name have the same functions, and a description thereof is omitted. Note that a video output unit411outputs video signals to the switcher1000.

As described above, the sub camera400is a PTZ camera, and its capture direction and angle of view can be controlled from outside. For this reason, the sub camera400includes a driving unit409capable of performing pan and tilt operations and a zoom operation, and a driving I/F408. The driving I/F408is a communication interface between the driving unit409and a CPU401.

The driving unit409includes a pan/tilt mechanism by which the sub camera400is supported so that it can be panned and tilted, a zoom mechanism that changes the angle of view of the image capture optical system, a motor that drives these mechanisms, and the like. Enlargement and reduction of images performed by an image processing unit406may be used in the zoom mechanism. The driving unit409drives the motor in accordance with instructions received from the CPU401via the driving I/F408, and adjusts the optical axis direction and the angle of view of the image capture optical system.

Also, a state display unit412is a so-called tally lamp. It is assumed here that the state display unit412includes red and green LEDs. Regarding the two LEDs of the state display unit412, none of them is lit, or only one of them is lit; both of them are not lit together. The CPU401controls display of the state display unit412based on tally information received from the capture control apparatus100. Here, the CPU401controls the state display unit412so that the red LED is lit in a case where the received tally information indicates “distributing”, and the green LED is lit in a case where the received tally information indicates “previewing”. Furthermore, the CPU401controls the state display unit412so that the red and green LEDs are not lit in a case where the received tally information indicates “standing by”.

Note that the number of LEDs included in the state display unit412may be one, or may be three or more. Also, regarding the colors thereof, any colors may be used as long as two or more LEDs do not have the same color. Note that the relationship between tally information and the display state of the state display unit412should be unified in all of the sub cameras400.

<Description of Operations of Each Apparatus>

Next, usingFIG.3andFIGS.4A to4D, the following describes the operations of each apparatus in a case where the capture control apparatus100automatically controls capture operations of the sub cameras400ato400cbased on a video of the overhead camera300in the multi-camera image capture system.

FIG.3is a diagram showing a processing sequence that is performed when the capture control apparatus100controls the operations of the sub cameras400ato400c, with a focus on main operations and flows of signals. The functional blocks shown in the capture control apparatus100schematically indicate the main operations, and are equivalent to the main functions provided by the capture control application. Each functional block ofFIG.3is realized by a combination of the CPU101, which executes the capture control application, and one or more of the functional blocks of the capture control apparatus100shown inFIG.2.

Also,FIG.4AtoFIG.4Dare flowcharts related to the operations of the capture control apparatus100, the overhead camera300, the sub cameras400, and the switcher1000, respectively. As the operations of the sub cameras400ato400care basically the same, they will be described collectively.

In the following description, it is assumed that the capture control apparatus100is aware of the three-dimensional coordinate value of the viewpoint position of the overhead camera300and the capture direction (the optical axis direction) thereof. Furthermore, it is assumed that known position information, such as the three-dimensional coordinate values of the viewpoint positions of the sub cameras400ato400c, and the coordinate values of markers placed in the captured area20, is stored in advance as predetermined position information REF_POSI in the ROM103. Note, it is assumed that the coordinate system of a position is determined in advance in accordance with the type of the position.

(Operations of Capture Control Apparatus100)

First, the operations of the capture control apparatus100will be described with reference toFIG.3andFIG.4A. The operations described below are realized by the CPU101executing the capture control application.

In step S101, the CPU101transmits a capture instruction command to the overhead camera300via the network I/F105using a predetermined protocol. In response to this command, the overhead camera300starts to supply video signals (moving image data) IMG to the video input unit107. The CPU101starts to store the video signals received by the video input unit107into the RAM102, and then executes step S102.

In step S102, the recognition unit121executes the following processing.

(1) Apply processing for detecting subject regions to an input frame image, and store the detection results.
(2) For each of the detected subject regions, apply coordinate transformation to position information (image coordinates).
(3) For each of the detected subject regions, apply identification processing and specify identification information (in the case of a new subject, add information for identification processing).
(4) For each of the detected subject regions, store identification information ID[n] and position information POSITION[n] in association with each other.

The recognition unit121is realized mainly by the CPU101and the inference unit104. The CPU101reads out, from the RAM102, one frame of the video received from the overhead camera300, and inputs the frame to the inference unit104.

The following describes the operations of the recognition unit121in order.

(1) First, the inference unit104inputs the frame image to the machine learning model, and detects subject regions. The inference unit104stores the positions and the sizes of the respective subject regions that have been detected and the detection reliability degrees thereof, which have been output by the machine learning model as detection results, into the RAM102. A position and a size of a subject region may be any information that allows a position and a size of a rectangular region in which the subject region is inscribed to be specified. Here, the central coordinates of the lower edge of the rectangular region, and the width and the height of the rectangular region, are used as the position and the size of the subject region.

Also, the inference unit104stores the detection results for the first frame image into the RAM102in association with identification information pieces ID[n] of subjects. Here, n is a subject number, and is an integer that takes a value from one to the total number of detected subject regions. Furthermore, the inference unit104stores the subject regions detected from the first frame image as templates for identifying the individual subjects into the RAM102in association with the identification information pieces ID[n] of the subjects. In a case where template matching is not used in identification of subjects, the templates may not be stored.

FIG.6Ashows examples of the results of subject detection processing that has been executed by the inference unit104with respect to a video of the overhead camera300shown inFIG.5A. Here, the regions of human subjects A to C, who are present in a captured area20, are detected, and the coordinates of the centers of the lower edges of rectangular regions in which the subject regions are inscribed (foot coordinates) are output as positions.

Note, for example, in a case where markers (Marks) are placed at known positions in the captured area20as shown inFIG.5Bfor the purpose of later-described coordinate transformation, the CPU101detects images of the markers included in the frame image (FIG.5A), and stores the positions thereof into the RAM102. The inference unit104may be configured to execute the detection of marker images as well. The detection of marker images can be carried out using any known method, such as pattern matching that uses marker templates. Marker images may be detected using a pre-stored machine learning model intended for marker detection.

(2) Next, the coordinate transformation executed by the inference unit104will be described.FIG.5Aschematically shows a video of the overhead camera300, andFIG.5Bschematically shows a state where the captured area20is viewed from directly above the center thereof. The inference unit104transforms the coordinates of the positions of the subject regions in a coordinate system of the overhead camera into the values of a coordinate system (planar coordinate system) of a case where the captured area20is viewed from directly above the center thereof.

Here, the reason why the coordinates are transformed into the values of the planar coordinate system is because the coordinate transformation is convenient for calculation of pan values for causing the sub cameras400ato400cto capture a specific subject (angles of movement on a horizontal plane). Note that the present description is provided on the premise that each sub camera400is placed so that the driving unit409performs a pan operation on a horizontal plane parallel to the floor of the captured area20.

The coordinate transformation can be executed using a variety of methods; here, markers are placed at a plurality of known positions on the floor of the captured area20, and the coordinates of the overhead camera coordinate system are transformed into the coordinates of the planar coordinate system based on the marker positions in the video obtained from the overhead camera300. Note that the coordinate transformation may be performed with use of, for example, the viewpoint position and the capture direction of the overhead camera300, without using markers.

The coordinate transformation can be executed using a homography transformation matrix H in accordance with the following formula 1.

In formula 1, x and y on the right side are the horizontal coordinate and the vertical coordinate in the overhead camera coordinate system, whereas X and Y on the left side are the horizontal coordinate and the vertical coordinate in the planar coordinate system.

The homography transformation matrix can be calculated by solving simultaneous equations by assigning the coordinates of the four markers detected from the video and the (known) coordinates of the four markers placed in the captured area20in formula 1. In a case where the positional relationship between the captured area20and the overhead camera300is fixed, the homography transformation matrix H can be calculated in advance at the time of test image capture and stored into the ROM103, for example.

The CPU101sequentially reads out the positions of the subject regions from the RAM102, and transforms the coordinates thereof into the values of the planar coordinate system.FIG.6Bschematically shows a state where the foot coordinates (x, y) of each subject region detected from the video of the overhead camera300shown inFIG.6Ahave been transformed into the coordinate values (X, Y) of the planar coordinate system with use of formula 1 and the homography transformation matrix H stored in the ROM103. The CPU101stores the foot coordinates obtained through the coordinate transformation as POSITION[n] into the RAM102.

(3) Next, a description is given of an operation of the inference unit104to specify the identification information pieces ID[n] of the subjects. It is assumed here that the subjects are identified using template matching. Subject identification is carried out with respect to the results of subject detection processing that has been executed for the second time onward. With respect to the results of processing that has been executed for the first time, it is sufficient to newly assign identification information pieces ID[n] to the subject regions.

The inference unit104specifies identification information pieces ID[n] of the detected subject regions by way of template matching that uses templates stored in the RAM102. As a result, the subjects in the captured area are identified. For example, for each of the detected subject regions, the inference unit104calculates evaluation values indicating correlations with the individual templates. Then, the inference unit104specifies identification information ID[n] corresponding to a template with which it has a correlation equal to or larger than a certain level and it has the highest correlation as identification information ID[n] of the subject region. For example, a known value, such as the sum of absolute differences between pixel values, can be used as an evaluation value.

Note, with respect to a subject region that does not have a correlation equal to or larger than the certain level with any template, the inference unit104assigns new identification information ID[n] and adds an image of the subject region as a template.

Also, the inference unit104may update existing templates using the subject regions that have been detected in the latest frame image, and may delete a template if a subject region that has a correlation equal to or larger than the certain level therewith does not exist for a certain period. Furthermore, the inference unit104may store templates corresponding to identification information pieces ID[n] that frequently appear into the ROM103.

Note that the subjects may be identified using a method other than template matching. For example, the same identification information ID[n] may be specified for a subject region that is closest, in terms of at least one of the detected position and the size, to an immediately-preceding subject region. Also, a position in the current frame image may be predicted using a Kalman filter or the like based on positional transitions in the plurality of past detection results associated with the same identification information, and the same identification information ID may be specified for a subject region that is closest to the predicted position. Furthermore, these methods may be combined. When template matching is not used, the accuracy of identification of different subjects with similar appearances can be increased.

(4) The inference unit104stores the specified identification information pieces ID[n] and the positions (planar coordinate system) POSITION[n] of the corresponding subject regions into the RAM102in association with each other.

Note that among the processing of (1) to (4), processing other than the subject detection may be executed by the CPU101in place of the inference unit104.

Here, the identification information pieces ID[n] and the positions POSITION[n] related to the subjects in the captured area20are obtained using the video of the overhead camera300. However, videos of the sub cameras400may be used. In this case, the CPU101executes the operations shown in the flowchart ofFIG.4Afor each sub camera. The positions of subject regions are output as values of a coordinate system of each sub camera400. In this way, the overhead camera300is not indispensable, but it is considered that the accuracy of subject detection is higher when the overhead camera300is used.

Returning to the description ofFIG.4A, in step S103, the CPU101as a subject of interest determination unit122ofFIG.3determines a subject of interest. The CPU101can determine a subject designated by the user, or a subject selected based on a predetermined condition, as the subject of interest.

In a case where the user selects the subject of interest, the CPU101causes the display unit108or an external display apparatus to display the frame image to which the subject detection processing has been applied in step S102, together with indicators that indicate detected subject regions. The indicators may be, for example, rectangular frames indicating the outer edges of the subject regions shown inFIG.6A, or may be other indicators. Furthermore, the CPU101may cause the display unit108to also display, for example, a message for encouraging a selection of a subject of interest in the image.

The user can select a subject region corresponding to a desired subject of interest by operating an input device connected to the user input I/F106. The selection method is not limited in particular, but may be an operation of designating a desired subject region by operating a mouse or a keyboard.

Upon detecting a user operation for designating one of the subject regions, the CPU101stores the identification information ID[n] corresponding to the designated subject region as identification information MAIN_SUBJECT of the subject of interest into the RAM102. Alternatively, the CPU101automatically stores identification information corresponding to one subject region that matches a predetermined condition as identification information of the subject of interest into the RAM102. For example, the CPU101stores identification information ID[n] that corresponds to a subject region closest to the center of the screen, or a subject region that is largest in size among subject regions that are located at a distance of a threshold or smaller from the center of the screen, as identification information MAIN_SUBJECT into the RAM102.

Alternatively, the CPU101may determine a subject captured by one predetermined camera among the sub cameras400ato400cas the subject of interest. In this case, a video from one predetermined sub camera is supplied to the capture control apparatus100via the communication network600or via the video input unit107.

Also, the CPU101stores position information POSITION_OH corresponding to the identification information MAIN_SUBJECT of the tracking target subject into the RAM102.

Thereafter, the CPU101executes step S104. In step S104, the CPU101as a tracking target subject determination unit123ofFIG.3determines the subjects to be tracked and captured by the respective sub cameras400ato400c.

It is assumed here that every sub camera is to track and capture the subject of interest. Therefore, the CPU101sets the identification information MAIN_SUBJECT of the subject of interest, which has been determined in step S103, as identification information SUBJECT_ID of the tracking target subject.

Note that there may be a sub camera that captures a subject other than the subject of interest. For example, in a case where three subjects are present in the captured area20as shown inFIG.6B, different tracking target subjects may be determined for the sub cameras400ato400c, respectively.

The CPU101writes the identification information SUBJECT_ID of the determined tracking target subject to the RAM102. In a case where the tracking target subject can differ among the sub cameras, the CPU101stores the identification information pieces SUBJECT_ID of the tracking target subjects in association with the identification information pieces of the sub cameras. Note that in a case where the tracking target subject has changed, the CPU101keeps holding information of the previous tracking target subject in the RAM102without deleting the same.

Next, in step S105, the CPU101as a pan/tilt calculation unit124calculates the amounts of changes in the pan angle and the tilt angle that are necessary for the sub cameras to track and capture the tracking target subject that has been determined in step S104.

It is assumed here that the following information is stored in advance as predetermined position information REF_POSI in the ROM103for each of the sub cameras400ato400c.Three-dimensional coordinates of the position of placement (a value in the planar coordinate system)A capture direction corresponding to the initial values of the pan angle and the tilt angle of the driving unitRange in which the pan and tilt angles can be controlled

The CPU101reads out the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracking target subject from the RAM102. Then, the CPU101first determines the pan angles from the position information POSITION_OH and the positions of placement of the sub cameras.

FIG.7is a diagram showing an example of a positional relationship between one sub camera and the tracking target subject in the planar coordinate system. It is assumed here that a pan angle θ for pointing the optical axis direction of the sub camera at the subject position is determined. The CPU101calculates the pan angle θ using the following formula 2.

In formula 2, px and py are the horizontal coordinate and the vertical coordinate of the position information POSITION_OH corresponding to the identification information SUBJECT_ID of the tracking target subject. Also, subx and suby are the horizontal coordinate and the vertical coordinate of the position of placement of the sub camera. It is assumed here that the current pan angle is the initial value 0°, and the optical axis direction is the vertical direction (Y-axis direction). In a case where the current optical axis direction is not the vertical direction, it is sufficient to reflect the angle difference between the current optical axis direction and the vertical direction in the angle obtained from formula 2. Furthermore, the pan direction is the counterclockwise direction if subx>px, and the clockwise direction if subx<px.

Next, the method of determining on the tilt angle will be described usingFIG.8.FIG.8shows a state where a sub camera and the tracking target subject are viewed from the side. It is assumed that the current optical axis of the sub camera400extends in the horizontal direction, the height thereof is h1, and the face of the tracking target subject at which the optical axis is to be pointed is at a height of h2. It is assumed that the angle difference in the height direction between the current optical axis direction and a target optical axis direction (the tilt angle) is ρ. The CPU101calculates the tilt angle ρ using the following formula 3 and formula 4.

The coordinate values used in formula 4 are the same as the coordinate values used in formula 2. It is assumed that h1and h2are input to the capture control application and stored into the RAM102in advance. In this case, identification numbers that are associated with h2of the respective subjects are set to be the same as identification numbers assigned in the subject detection processing. Alternatively, a value that has been measured in real time using a non-illustrated sensor may be used as h2.

It is assumed here that the current tilt angle is the initial value 0°, and the optical axis direction is the horizontal direction (the heights are constant). In a case where the current optical axis direction is not the horizontal direction, it is sufficient to reflect the angle difference between the current optical axis direction and the horizontal direction in the angle obtained from formula 4. Furthermore, the tilt direction is a downward direction if h1>h2, and an upward direction if h1<h2.

The CPU101cyclically communicates with each sub camera400via the communication network600, obtains the current optical axis directions (the pan angles and the tilt angles of the driving units), and stores them into the RAM102. Note that the communication cycle can be, for example, equal to or smaller than the reciprocal of the frame rate. Alternatively, for each sub camera400, the CPU101may hold the value of the sum total of the pan angles and the tilt angles that have been controlled from the initial state in the RAM102, and use this value as the current optical axis direction.

The CPU101calculates the amounts of changes in the pan angle and the tilt angle of each sub camera400in the foregoing manner, and stores them into the RAM102.

The amounts of changes in the pan angle and the tilt angle may be an angular velocity for causing a sub camera400to turn to the direction of the tracking target subject. For example, the CPU101obtains the current pan angle and tilt angle from each sub camera400via the communication network600. Then, the CPU101obtains a pan angular velocity proportional to the difference between the pan angle θ that has been read out from the RAM102and the current pan angle. Also, the CPU101obtains a tilt angular velocity proportional to the difference between the tilt angle ρ that has been read out from the RAM102and the current tilt angle. The CPU101stores the angular velocities calculated in the foregoing manner into the RAM102.

Note that the amounts of changes in the pan angle and the tilt angle may be calculated using a video of a sub camera400instead of a video of the overhead camera300. In this case, the CPU101may calculate the amount of change in the pan angle from the difference in the horizontal direction between the current optical axis direction and the direction of the tracking target subject in the coordinate system of the sub camera400, and calculate the amount of change in the tilt angle from the difference in the vertical direction therebetween. Furthermore, in the image capture system, changing of the capture direction for tracking and capturing the tracking target subject may be performed only in one of the pan direction and the tilt direction; in such an image capture system, only the amount of change in one of the pan angle and the tilt angle may be calculated.

Next, in step S106, the CPU101reads out, from the RAM102, the amounts of changes in the pan and tilt angles that have been calculated in step S105. Then, with respect to each individual sub camera, the CPU101generates a control command PT_VALUE for instructing the sub camera to make changes equivalent to these amounts of changes to the pan angle and the tilt angle. It is assumed that the format of the control commands has been determined in advance. The CPU101stores the generated control commands PT_VALUE into the RAM102.

Next, in step S107, the CPU101reads out the control commands PT_VALUE generated in step S106from the RAM102, and transmits them to the communication network600via the network I/F105. The sub cameras400ato400creceive the control commands PT_VALUE addressed to themselves via network I/Fs405.

The CPU101executes the processing from step S101with respect to the next frame image in the video of the overhead camera300. Note that the processing shown inFIG.4Aneed not necessary be executed on a per-frame basis.

Next, the operations of the overhead camera300will be described with reference toFIG.4B. The operations described below are realized by the CPU301executing the program.

When the power of the overhead camera300has been turned on, the CPU301initializes each functional block, and then a capture standby state begins. In the capture standby state, the CPU301starts moving image capture processing for live-view display. Image data for display generated by the image processing unit306is output via the monitor output unit312.

In the capture standby state, the CPU301waits for reception of a control command via the network I/F305. Upon receiving a control command, the CPU301executes operations corresponding to the control command. The following describes operations for a case where a capture command has been received as the control command from the capture control apparatus100.

In step S201, the CPU301receives a capture command from the capture control apparatus100via the network I/F305.

Note that in the capture command, such capture parameters as the frame rate and the resolution may be designated. Furthermore, the capture command may include settings related to processing applied by the image processing unit306.

In S202, in response to the reception of the capture command, the CPU301starts processing for capturing moving images to be supplied to the capture control apparatus100. In this moving image capture processing, moving images that have higher image quality than those of the moving image capture processing for live-view display are captured. For example, the captured moving images are higher in at least one of the moving image resolution and the capture frame rate than moving images for live-view display. The image processing unit306applies processing to the images based on settings for the moving images to be supplied to the capture control apparatus100. The image processing unit306sequentially stores the generated pieces of moving image data into the RAM302.

In step S203, the CPU101reads out the pieces of moving image data from the RAM302, and supplies them as video signals from the video output unit311to the video input unit107of the capture control apparatus100.

From then on, processing from the image capture to the supply of video signals is continued until a control command for stopping the image capture is received.

(Operations of Sub Cameras400)

Next, the operations of a sub camera400will be described with reference toFIG.4C. The operations described below are realized by the CPU401executing the program. Note that as the sub cameras400ato400cexecute the same operations, the following will be described as the operations of the sub cameras400.

When the power of the sub camera400has been turned on, the CPU401initializes each functional block, and then starts processing for capturing moving images to the supplied to the switcher1000. The image processing unit406applies, to analog image signals obtained from an image sensor407, processing based on settings for the moving images to be supplied to the switcher1000. The image processing unit406sequentially stores the generated pieces of moving image data into the RAM402. The CPU401reads out the pieces of moving image data from the RAM402, and supplies them as video signals from the video output unit411to the video input unit1003of the switcher1000.

While supplying the video signals to the switcher1000, the CPU401waits for reception of a control command via the network I/F305. Upon receiving a control command, the CPU401executes operations corresponding to the control command. The following describes operations for a case where a pan/tilt operation command from the capture control apparatus100, or a state notification command from the switcher1000, has been received as a control command.

In step S301, the CPU401receives the control command from the capture control apparatus100or the switcher1000via the network I/F405.

In step S302, the CPU401determines whether the received control command is a pan/tilt operation command or a state notification command. The CPU401executes step S303if it has been determined that the control command is the pan/tilt operation command, and executes step S306if it has been determined that the control command is the state notification command. A description of operations corresponding to other control commands is omitted.

In step S303, the CPU401reads out an operation direction (the direction(s) of pan and/or tilt) and a corresponding operation amount (angle) from the control command, and stores them into the RAM302.

In step S304, the CPU401generates driving parameters for the driving unit409based on the operation direction and the operation amount that have been read out in step S303. The CPU401may obtain, for example, driving parameters corresponding to the combination of the operation direction and the operation amount with use of a table that has been held in the ROM403in advance. Note that in a case where the operation amount is provided in the form of a target value (e.g., a target angle), the CPU410obtains driving parameters from the difference from the current value.

In step S305, the CPU401controls the driving unit409via the driving I/F408based on the driving parameters obtained in step S304. As a result, the driving unit409is driven in the operation direction designated through the control command by the designated operation amount.

In step S306, the CPU401reads out state information included in the state notification command, and stores the same into the RAM402.

In step S307, the CPU401distinguishes the state information stored in the RAM402. It is assumed here that the state information is one of “distributing”, “previewing”, and “standing by”. Then, the CPU401controls the state display unit412in accordance with the distinguished state information.

Here, as one example, in a case where the state information is “distributing”, the CPU401lights the red LED and turns off the green LED. Also, in a case where the state information is “previewing”, the CPU401lights the green LED and turns off the red LED. Furthermore, in a case where the state information is “standing by”, the CPU401turns off the red and green LEDs.

Note that the state notification command is not limited to being received via the network I/F405, and may be received via another communication interface.

Next, the operations of the switcher1000will be described with reference toFIG.4D. The operations described below are realized by the CPU1001executing the program. The following describes operations for a case where the switcher1000has detected an instruction for changing (switching) a main video or a preview video via the user input I/F1006, or a case where it has received a state information request command via the network I/F1007.

In step S401, the CPU1001detects a user instruction or a control command via the user input I/F1006or the network I/F1007.

In step S402, the CPU1001distinguishes the content detected in step S401. The CPU401executes step S403in a case where it has been determined that the video switching instruction has been detected via the user input I/F1006, and step S406in a case where it has been determined that the state information request command has been received via the network I/F1007.

The video switching instruction is an instruction for changing a main video (a video to be output from the video output unit1005) or a preview video (a video that is scheduled to be used as a main video next) among the videos that are supplied to the video input units1003ato1003cfrom the sub cameras400ato400c. Note that the main video and the preview video may not be selected. For example, an instruction for changing the main video or the preview video is input as an operation on a display screen for an input video via a mouse or a keyboard connected to the user input I/F1006.

In step S403, in the case of the instruction for changing the main video, the CPU1001switches a video to be supplied to the video output unit1005as the main video by controlling the video switch control unit1004.

Then, in step S404, the CPU1001notifies at least a sub camera whose state has been changed of the changed state information. A sub camera whose state has not been changed may not be notified of the state information. The state information is transmitted to the communication network600via the network I/F1007.

Note that instead of giving notice of the state information, the states of the state display units412of the sub cameras400may be controlled directly from the switcher1000. In this case, it is necessary for the state display units412of the respective sub cameras400to be connected to the switcher1000. As long as the sub cameras400can be notified of the state information, a means for this notification is not limited.

In step S405, the CPU1001stores, into the RAM1002, state information STREAMING in which identification information and the state are associated with each other for each sub camera.

In step S406, the CPU1001reads out the state information STREAMING stored in the RAM1002.

In step S407, the CPU1001transmits the state information STREAMING to the capture control apparatus100via the network I/F1007.

Note that in a case where an instruction for checking the state information has been detected via the user input I/F1006, the CPU401reads out the state information STREAMING stored in the RAM1002, and outputs a state information list screen to an external display apparatus via the monitor output unit1009.

(Operations of Determining on Tracking Target Subject in Consideration of State of Sub Camera)

Next, using a flowchart shown inFIG.9, a description is given of operations of the capture control apparatus100to determine a tracking target subject in consideration of a state of a sub camera400. The operations shown inFIG.9can be executed in place of, for example, step S104ofFIG.4A.

In step S501, the CPU101determines whether the subject of interest has been changed as a result of processing of step S103; step S502is executed if it has been determined that the subject of interest has been changed, and the processing of the flowchart shown inFIG.9is ended if it has not been determined that the subject of interest has been changed.

The CPU101stores, into the RAM102, at least identification information MAIN_SUBJECT of the subject of interest that has been determined most recently. Then, in a case where the subject of interest determined in step S103is different from the subject of interest that has been determined most recently, the CPU101determines that the subject of interest has been changed.

Steps S502to S506are executed for each sub camera with which the capture control apparatus100can communicate. Information of sub cameras400with which the capture control apparatus100can communicate can be obtained by the CPU101from each of the sub cameras400ato400cvia the network I/F105upon, for example, activation of the capture control apparatus100. Alternatively, the CPU101may obtain information of each of the sub cameras400ato400cfrom the switcher1000. The CPU101stores the obtained information of the sub cameras into the RAM102.

Instead of obtainment via the network I/F105, the user may register identification information pieces (e.g., network addresses, serial numbers, and the like) as sub cameras to be controlled with the capture control application. Either way, the CPU101is aware of identification information pieces of sub cameras to be controlled, and executes processing of steps S502to S506while sequentially switching a target sub camera.

In step S502, the CPU101transmits a state information request command to the switcher1000via the network I/F105. The CPU101receives the state information STREAMING from the switcher1000as a response to the state information request command, and stores the same into the RAM102. Here, the states of the sub cameras400ato400care collectively obtained from the switcher1000; however, the states may be obtained individually from the sub cameras400ato400c. In this case, the volume of communication between the switcher1000and the capture control apparatus100can be reduced.

In step S503, the CPU101refers to the state information STREAMING stored in the RAM102, and obtains the state of the sub camera400targeted for the processing. Then, the CPU101determines whether the state is “distributing”. The CPU101executes step S504if the state has been determined to be “distributing”, and step S505if the state has not been determined to be “distributing”.

Note that in a case where the state has not been determined to be “distributing”, the CPU101may further determine whether the state is “previewing”, and execute step S505in a case where the state has not been determined to be “previewing”. This can prevent unintended changing of a subject in a video that is scheduled by the operator of the switcher1000to be used as a main video next.

In step S504, the CPU101provides a notification indicating a subject of interest change refusal (or indicating that the subject of interest is not to be changed), together with the identification information of the sub camera targeted for the processing. The notification may be, for example, a message displayed on the display unit108.

Also, the CPU101may transmit the notification to the outside. For example, the CPU101can transmit the notification to the overhead camera300via the network I/F105. Upon receiving the notification via the network I/F305, the overhead camera300displays the subject of interest change refusal on an external display apparatus connected to the monitor output unit312.

Furthermore, the CPU101may transmit, to the overhead camera300, the result of subject detection performed by the recognition unit121, the identification information MAIN_SUBJECT of the subject of interest, the identification information SUBJECT_ID of the tracking target subject, and the identification information of the sub camera targeted for the processing, together with the notification.

Then, based on the result of subject detection, the CPU301of the overhead camera300can output an image obtained by superimposing the identification information of the sub camera targeted for the processing and an indicator (e.g., a rectangular frame) indicating the detected subject on a live-view display image from the monitor output unit312. Also, the CPU301can cause the indicators of the subject of interest and the tracking target subject, among the detected subjects, to be visually different from those of other subjects.

For example, in a case where the subject of interest and the tracking target subject are the same, the indicator is given the color blue, whereas in a case where they are different, the indicators of the subject of interest and the tracking target subject are given the colors green and red, respectively. Furthermore, the indicators of other subjects are not displayed, or given another color, such as white. Note that the colors are merely examples, and visual elements other than colors may vary. In a case where a blue indicator is not displayed but green and red indicators are displayed, it is apparent that changing of the tracking target subject is refused with respect to the sub camera targeted for the processing.

In a case where the subject of interest and the tracking target subject are both subject A with regard to a certain sub camera, the subject A is displayed with a blue indicator (FIG.10A). It is assumed here that a subject that is neither the subject of interest nor the tracking target subject is not displayed with an indicator. In a case where the subject of interest has been changed to the subject B in this state, if the state of the sub camera is not “distributing” (or is not “previewing”), the new subject of interest becomes the tracking target subject. Therefore, the indicator of the subject A disappears, and the subject B is newly displayed with a blue indicator (FIG.10B). However, in a case where the state of the sub camera is “distributing” (or “previewing”), the tracking target subject is not changed. Therefore, the indicator of the subject A is changed to red, and the subject B is newly displayed with a green indicator (FIG.10C).

Note that the CPU101may transmit the result of subject detection, the identification information MAIN_SUBJECT of the subject of interest, and the identification information pieces SUBJECT_ID of the tracking target subjects of the respective sub cameras to the overhead camera300, regardless of a change in the subject of interest. In this way, an external display apparatus connected to the overhead camera300can perform the aforementioned display on a continual basis. Also, the sub camera targeted for display may be changed in sequence, for example, at a constant frequency. In this way, in a case where there are a plurality of sub cameras, the tracking target subjects of the respective sub cameras and the subject of interest can be grasped. Furthermore, the display unit108may also perform display similar to that of the overhead camera300.

Note that the notification indicating the tracking target subject change refusal need not necessarily be provided, and the user may be able to set whether the notification is required via the user input I/F106. In a case where there are a plurality of sub cameras400, this may be settable on a per-camera basis.

In step S505, the CPU101changes the identification information SUBJECT_ID of the tracking target subject of the sub camera targeted for the processing, which is stored in the RAM102, to the identification information MAIN_SUBJECT of the changed subject of interest. That is to say, the CPU101changes the tracking target subject of the sub camera targeted for the processing.

In step S506, the CPU101stores the identification information MAIN_SUBJECT of the subject of interest into the RAM102.

Once processing of steps S502to S506has been executed with respect to each of the sub cameras400ato400cto be controlled, the CPU101ends the processing shown in the flowchart ofFIG.9. The CPU101subsequently executes step S105ofFIG.4A.

Note, regarding a sub camera for which changing of the tracking target subject has been refused, a change may be made so that the subject of interest is set as the tracking target subject if the sub camera has changed to a state where the tracking target subject can be changed when step S104ofFIG.4Ais executed from the next time onward. Specifically, regarding a sub camera whose tracking target subject is different from the subject of interest in a state where the tracking target subject can be changed, the CPU101can make a change so that the subject of interest is set as the tracking target subject.

As described above, in a case where a sub camera to be automatically controlled so as to track and capture a specific subject is in the state “distributing” (i.e., in a case where the selection has been made by the switcher1000), the capture control apparatus100of the present embodiment does not change the tracking target subject even if the specific subject has been changed. This can prevent a subject in a main video from being changed to a subject that is against the intention of the operator of the switcher1000. Furthermore, with regard to a sub camera in the state “previewing” as well, the tracking target subject is not changed even if the specific subject has been changed; this can also prevent a subject that is scheduled to be in a main video next from being changed to a subject that is against the intention of the operator of the switcher1000.

The capture control apparatus100of the present embodiment can realize the advantageous effect of preventing an unintentional change in a subject in a main video while allowing image capture to be easily performed using a plurality of sub cameras with a small number of staff.

Modification Example

Note that in step S104ofFIG.4A, the tracking target subject determination unit123(CPU101) may determine a tracking target subject of a target sub camera400based on a role set on the sub camera400.

How to use information obtained from a main camera in controlling the operations of a sub camera400is determined in advance in relation to a role that can be set on the sub camera400. It is assumed here that information of the main camera is used in controlling a tracking target subject and a zoom operation of the sub camera400as one example. Note that the main camera is a camera different from the overhead camera300; the main camera may be provided separately from the sub cameras, or one of the plurality of sub cameras may be assigned as the main camera. In a case where the main camera is provided separately from the sub cameras, it is sufficient that the main camera be a camera whose image capture is not controlled by the capture control apparatus100(operated by a photographer). In a case where the main camera is provided, for example, the user registers identification information of the main camera and the roles to be set on the respective sub cameras with the capture control application via the user input I/F106.

A tracking target subject of the main camera may be determined by the capture control apparatus100, or may be determined from a video of the main camera input to the capture control apparatus100, similarly to the overhead camera300. The CPU101can determine the tracking target subject of the main camera using, for example, a known main subject determination method. For example, the CPU101can specify identification information of the tracking target subject by identifying a subject region closest to the center of the image.

Also, the CPU101can determine a zoom operation of the main camera and the phase thereof by, for example, detecting a change in the angle of view of the video of the main camera. For example, the change in the angle of view may be detected from the sizes of the subject regions, a temporal change in an interval therebetween, and the like. Alternatively, the CPU101may detect a zoom operation of the main camera and the phase thereof by cyclically obtaining information of the angle of view from the main camera.

FIG.14shows examples of the types of roles that can be set on a sub camera400, and the corresponding contents of control. It is assumed here that one of “main follow”, “main counter”, “assist follow”, and “assist counter” can be set as a role (ROLE).

With respect to the sub camera400whose role (ROLE) is “main follow”, the capture control apparatus100(CPU101) sets the same tracking target subject as the main camera, and also performs in-phase zoom control on the sub camera400when a zoom operation has been performed on the main camera. Here, in-phase means that the zoom direction (telephoto direction or wide-angle direction) is the same, that is to say, the direction of change in the angle of view is the same. On the other hand, antiphase means that the zoom direction (telephoto direction or wide-angle direction) is the opposite direction, that is to say, the direction of change in the angle of view is the opposite direction. Note that the angle of view may not be the same as that of the main camera500even if the zoom direction is in-phase, and the degree of change in zooming (the speed of change, the rate of change, or the like) may not be the same as that of the main camera500whether the zoom direction is in-phase or antiphase.

With respect to the sub camera whose role (ROLE) is “main counter”, the capture control apparatus100(CPU101) sets the same tracking target subject as the main camera, and also performs antiphase zoom control on the sub camera when a zoom operation has been performed on the main camera. Therefore, in a case where a zoom-in operation has been performed on the main camera, the capture control apparatus100(CPU101) controls the sub camera of this role to zoom out. Note that to zoom in means to switch to zooming in the telephoto direction (the direction toward the telephoto end), whereas to zoom out means to switch to zooming in the wide-angle direction (the direction toward the wide-angle end). In a case where zoom control is performed by the image processing unit406, to zoom in means to reduce a region to be cut out from an image and increase the enlargement factor of the cut-out region compared to that before changing the region. On the other hand, to zoom out means to increase a region to be cut out from an image and reduce the enlargement factor of the cut-out region compared to that before changing the region.

With respect to the sub camera whose role (ROLE) is “assist follow”, the capture control apparatus100(CPU101) sets a tracking target subject different from that of the main camera, and also performs in-phase zoom control on the sub camera when a zoom operation has been performed on the main camera.

With respect to the sub camera whose role (ROLE) is “assist counter”, the capture control apparatus100(CPU101) sets a tracking target subject different from that of the main camera, and also performs antiphase zoom control on the sub camera when a zoom operation has been performed on the main camera.

Here, with respect to the sub camera400whose role (ROLE) is “assist follow” or “assist counter”, a subject located on the left side among the subjects other than a tracking target subject of the main camera in an image is set as a tracking target subject of the sub camera400. Note that the tracking target subject of the sub camera400may be set in accordance with other conditions. For example, a subject that is located on the right side, the upper side, or the lower side among the subjects other than a tracking target subject of the main camera in an image may be set as the tracking target subject of the sub camera.

Alternatively, a subject that is located nearest to the front or the back among the subjects other than a tracking target subject of the main camera may be set as the tracking target subject of the sub camera400.

Furthermore, only one of the setting of the tracking target subject and zoom control may be executed, and another control item may be further added.

Second Embodiment

Next, a second embodiment of the present invention will be described. The present embodiment may be similar to the first embodiment, except for the operations of the capture control apparatus100to determine a tracking target subject. Therefore, a description of configurations and operations of apparatuses that are the same as those of the first embodiment is omitted.

Using a flowchart shown inFIG.11, the following describes the operations of the capture control apparatus100to determine a tracking target subject according to the present embodiment. The operations shown inFIG.11can be executed in place of, for example, step S104ofFIG.4A. Furthermore, processes that are the same as those of the first embodiment are given the same reference numerals asFIG.9, and a description thereof is omitted.

In step S501, the CPU101determines whether the subject of interest has been changed as a result of processing of step S103; step S602is executed if it has been determined that the subject of interest has been changed, and step S601is executed if it has not been determined that the subject of interest has been changed.

In step S601, the CPU101executes processing for reserving a change in a tracking target subject. The details of this processing will be described later. Once the processing for reserving a change in a tracking target subject has finished, the CPU101ends the processing of the flowchart shown inFIG.11.

In step S602, the CPU101determines whether a reservation for subject change is stored in the RAM102. The reservation for subject change will be described later. The CPU101executes step S603if it has been determined that the reservation for subject change is stored in the RAM102, and step S502if it has not been thus determined.

In step S603, the CPU101deletes the reservation for subject change stored in the RAM102. This is because the subject of interest has been changed since the time when the reservation was stored. Thereafter, the CPU101executes step S502.

Thereafter, the CPU101executes processing similar to that of the first embodiment, except that step S606is executed in place of step S504in a case where it has been determined in step S503that the state of the sub camera targeted for the processing is “distributing”. In the present embodiment also, in the case of the state “previewing”, processing similar to that executed in the state “distributing” may be executed.

In step S606, the CPU101includes the identification information MAIN_SUBJECT of the subject of interest, which has been determined in step S103, in the reservation for change in the tracking target subject, and stores the reservation into the RAM102.

Next, using a flowchart shown inFIG.12, the details of processing for reserving a change in a tracking target subject, which is executed in step S601ofFIG.11, will be described. InFIG.12, processes that are the same as those of the first embodiment are given the same reference numerals asFIG.9, and a description thereof is omitted. Note that the processing for reserving a change in a tracking target subject may be executed at an arbitrary timing different from the timing of execution of step S104ofFIG.3. The processing for reserving a change in a tracking target subject is executed for each sub camera.

In step S701, the CPU101determines whether a reservation for a change in a tracking target subject is stored in the RAM102. The reservation for subject change will be described later. The CPU101executes step S502if it has been determined that the reservation for subject change is stored in the RAM102, and changes the sub camera targeted for the processing and executes step S701if it has not been thus determined. If there is no sub camera for which step S701has not been executed, the CPU101ends the processing shown in the flowchart ofFIG.12.

In step S502, the CPU101obtains state information of the sub camera targeted for the processing. The state information may be obtained directly from the sub camera, or the state information STREAMING may be obtained from the switcher1000. In a case where the state information STREAMING has been obtained from the switcher1000, as it includes state information of every sub camera, step S502need not be executed with respect to the remaining sub cameras.

Steps S503, S505, and S506are similar to those of the first embodiment. In step S505, with regard to the sub camera for which changing of a tracking target subject has been refused when a subject of interest was changed previously, as the state has been changed from “distributing” (the selection is no longer made by the switcher1000), the tracking target subject is changed to the latest subject of interest. After step S506, the CPU101executes step S706.

In step S706, the CPU101deletes the reservation for subject change stored in the RAM102. Then, the CPU101changes the target sub camera, and executes step S701. If there is no sub camera for which step S701has not been executed, the CPU101ends the processing shown in the flowchart ofFIG.12.

According to the present embodiment, with regard to a sub camera for which changing of a tracking target subject has been refused, the tracking target subject is changed automatically upon entering a state where the tracking target subject can be changed; therefore, the advantageous effect of offering convenience can be realized in addition to the advantageous effects similar to those of the first embodiment.

Third Embodiment

Next, a third embodiment of the present invention will be described. The present embodiment is related to control performed by the capture control apparatus100on an angle of view of a sub camera400. The capture control apparatus100can change the angle of view by transmitting a control command to the sub camera400. For example, in a case where an instruction for changing the angle of view of the sub camera400has been received from a non-illustrated external apparatus in the communication network600, there is a risk of a change in the angle of view that is against the intention of the operator of the switcher1000.

The capture control apparatus100of the present embodiment prevents a change in an angle of view that is not intended by the operator of the switcher1000with respect to a sub camera that is currently capturing a main video.

Upon receiving a zoom command via the network I/F105, the CPU101of the capture control apparatus100executes processing shown in a flowchart ofFIG.13.

In step S801, the CPU101stores the received zoom command into the RAM102. It is assumed here that the zoom command includes identification information of a target sub camera and zoom position information. Then, the CPU101obtains the zoom position information from the zoom command.

The zoom position changes between the wide-angle end and the telephoto end, thereby allowing the angle of view to be changed. For example, the zoom position information indicates the number of steps in a stepper motor that drives a zoom lens, and the zoom position changes between steps “0” to “300”. Here, the zoom position information may be, for example, a focal length (mm) of the image capture optical system corresponding to a full-size 35-mm image sensor; in this case, a zoom value representing the zoom position information on the telephoto side is larger than a zoom value representing the zoom position information on the wide-angle side. Note that the zoom command may include a zoom-in or zoom-out instruction in place of the zoom position information that uniquely indicates the angle of view. The zoom-in or zoom-out instruction is an instruction for changing the current angle of view by an angle of view of a certain magnitude. It is assumed that the certain magnitude is determined in advance based on, for example, a unit of zoom driving in the driving unit or a unit of control on a digital zoom magnification factor.

In step S803, the CPU101obtains the state information STREAMING from the switcher1000, and stores the same into the RAM102. Note that state information may be obtained from the sub camera400individually.

In step S804, the CPU101refers to the state information STREAMING stored in the RAM102, and obtains the state of the sub camera400designated through the zoom command. Then, the CPU101determines whether the state is “distributing”. The CPU101executes step S805if the state has been determined to be “distributing”, and step S807if the state has not been determined to be “distributing”.

In step S805, the CPU101reads out, from the RAM102, a setting value (permission setting value) indicating whether changing of the angle of view of the sub camera that is currently capturing a main video (a video that is currently output from the switcher1000) (or changing of the angle of view of the main video) is permitted. It is assumed that the permission setting value has been set in advance.

Note that when the zoom command has been received, data of a setting screen may be transmitted to the switcher1000, and the operator of the switcher1000may set the permission setting value. Upon receiving the permission setting value from the switcher1000via the network I/F105, the CPU101stores the permission setting value into the RAM102.

The CPU101refers to the permission setting value, and executes step S807if it has been determined that changing of the angle of view of the main video is permitted, and ends the processing shown in the flowchart ofFIG.13if it has not been thus determined. In this way, if it has not been determined that changing of the angle of view of the main video is permitted, the zoom command is not executed. In this case, the CPU101may notify the external apparatus that has transmitted the zoom command of the fact that zooming is not permitted.

In step S807, the CPU101generates a control command for the designated sub camera400based on the zoom position information, and stores the same into the RAM102.

In step S808, the CPU101reads out the control command from the RAM102, and transmits the same to the designated sub camera400via the network I/F105.

The capture control apparatus100of the present embodiment can control whether to permit changing of the angle of view of a sub camera that is capturing a main video (or changing of the angle of view of the main video). Also, in a case where the capture control apparatus100has received an instruction for changing the angle of view with respect to the sub camera that is capturing the main video, it changes the angle of view only when changing of the angle of view is permitted. This can prevent changing of the angle of view of the main video that is against the intention of the operator of the switcher.

Note that control is not limited to being performed on the angle of view; parameters which affect a video of a sub camera400and which can be controlled by the capture control apparatus100(e.g., a focusing distance, an exposure condition, and the like) can also be controlled similarly to changing of the angle of view. Furthermore, the permission setting value can be set on a per-parameter basis.

Other Embodiments

The above embodiments have been described in relation to a configuration in which the capture control apparatus100is an independent apparatus. However, another apparatus (e.g., the switcher1000or the sub cameras400) may have the functions similar to those of the capture control apparatus100.

This application claims the benefit of Japanese Patent Application No. 2023-145576, filed Sep. 7, 2023, which is hereby incorporated by reference herein in its entirety.