INFORMATION PROCESSING APPARATUS AND METHOD FOR CONTROLLING THE SAME

Imaging coverages of a plurality of imaging apparatuses are calculated based on position information about a position of each of the plurality of imaging apparatuses, azimuth and angle information about the plurality of imaging apparatuses, and imaging area information about the plurality of imaging apparatuses. A notification of an area capable of comprehensive imaging and an area not capable of comprehensive imaging is made based on the imaging coverages.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an information processing apparatus and a method for controlling the same.

Background Art

Nowadays, a large number of imaging apparatuses directed to various positions and at various angles are sometimes to be installed to capture video images from various viewpoints. While video images of imaging apparatuses at user-desired viewpoints are to be obtained from among the video images of the plurality of imaging apparatuses, it is difficult to appropriately store only the video images desired by the user among those of the large number of imaging apparatuses.

Patent Literature 1 discusses a method where imaging apparatuses each have gazing point information and their position information and direction information, and video images captured by the imaging apparatuses are selected based on user's attribute information.

Techniques about an imaging apparatus having pan and tilt functions and a function of automatically keeping track of a specific object have also been discussed. For example, Patent Literature 2 discusses a control method for determining differences between the center coordinates of a monitor and the position coordinates of an object and driving the pan and tilt angles to move the object to the screen center to output an image in which the object is present at the center of the monitor screen.

In a system that captures images from a plurality of viewpoints by using a plurality of imaging apparatuses, the exact installation positions of the imaging apparatuses and the optical axis directions and the angles of views of the imaging apparatuses are typically set before use. The imaging apparatuses can cooperatively perform framing through pan, tilt, and zoom driving based on the installation positions and the directions of the imaging apparatuses, whereas it is difficult to easily install and calibrate the plurality of imaging apparatuses. If the user carelessly installs the imaging apparatuses, the user has had difficulty in easily checking a range where images can be captured from a plurality of viewpoints or a range where imaging from a plurality of viewpoints is difficult.

The present invention is directed to providing an information processing apparatus that facilitates checking an imaging area in operating a plurality of imaging apparatuses in a cooperative manner and a method for controlling the same.

CITATION LIST

Patent Literature

SUMMARY OF THE INVENTION

An information processing apparatus includes a storage unit configured to store position information about a position of each of a plurality of imaging apparatuses, an obtaining unit configured to obtain azimuths and angles of the plurality of imaging apparatuses, a setting unit configured to set an imaging area of the plurality of imaging apparatuses, and a notification unit configured to make a notification of an area to be comprehensively imaged and an area to not be comprehensively imaged based on the position information stored in the storage unit, information obtained by the obtaining unit, and information about the imaging area set by the setting unit.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described in detail below with reference to the attached drawings.

Configuration of Imaging Apparatus

FIG. 1Ais a diagram schematically illustrating an imaging apparatus used in a control system of a plurality of cooperative imaging apparatuses according to a first exemplary embodiment.

An imaging apparatus101illustrated inFIG. 1Aincludes an operation member capable of a power switch operation (hereinafter referred to as a power button, whereas operations such as a tap, flick, and swipe on a touch panel may be used instead). A lens barrel102is a casing including an imaging lens group and an image sensor for capturing an image. The lens barrel102is equipped with rotation mechanisms that are attached to the imaging apparatus101and can drive the lens barrel102to rotate with respect to a fixing unit103. A tilt rotation unit104is a motor drive mechanism that can rotate the lens barrel102in a pitch direction illustrated inFIG. 1B. A pan rotation unit105is a motor drive mechanism that can rotate the lens barrel102in a yaw direction. The lens barrel102can thus rotate about one or more axial directions.FIG. 1Billustrates the definitions of the axes at the position of the fixing unit103. An angular velocity meter106and an acceleration meter107are both mounted on the fixing unit103of the imaging apparatus101. Vibrations of the imaging apparatus101are detected based on the angular velocity meter106and the acceleration meter107, and the tilt rotation unit104and the pan rotation unit105are driven to rotate based on the detected vibration angles. Thus, the imaging apparatus101is configured to correct shakes and tilts of the lens barrel102that is a movable unit.

FIG. 2is a block diagram illustrating a configuration of the imaging apparatus101according to the present exemplary embodiment.

InFIG. 2, a first control unit223includes a processor (such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, and a microprocessing unit (MPU)) and a memory (such as a dynamic random access memory (DRAM) and a static random access memory (SRAM)). Such components perform various types of processing to control various blocks of the imaging apparatus101, and control data transfer between the blocks. A nonvolatile memory (electrically erasable programmable read-only memory (EEPROM))216is an electrically erasable and recordable memory, and stores operating constants and programs intended for the first control unit223.

InFIG. 2, a zoom unit201includes a variable power zoom lens. A zoom driving control unit202controls driving of the zoom unit201. A focus unit203includes a lens for making a focus adjustment. A focus driving control unit204controls driving of the focus unit203.

An imaging unit206includes an image sensor that receives light incident through the lens groups and outputs information about charges corresponding to the amount of the light as analog image data to an image processing unit207. The image processing unit207performs analog-to-digital (A/D) conversion on the analog image data, applies image processing to the resulting digital image data, and outputs the image-processed digital image data. Examples of the image processing include distortion correction, white balance adjustment, and color interpolation processing. An image recording unit208converts the digital image data output from the image processing unit207into a recording format, such as a Joint Photographic Experts Group (JPEG) format, and transmits the converted digital image data to a memory215and/or a video output unit217to be described below.

A lens barrel rotation driving unit205drives the tilt rotation unit104and the pan rotation unit105to drive the lens barrel102in the tilt and pan directions.

An apparatus vibration detection unit209includes the angular velocity meter (gyro sensor)106and the acceleration meter (acceleration sensor)107, for example. The angular velocity meter106detects the angular velocity of the imaging apparatus101about the three axial directions. The acceleration meter107detects the accelerations of the imaging apparatus101about the three axial directions. The apparatus vibration detection unit209calculates the rotation angles of the imaging apparatus101and the amounts of shift of the imaging apparatus101based on the detected signals.

An audio input unit213obtains an audio signal around the imaging apparatus101from a microphone mounted on the imaging apparatus101, performs A/D conversion, and transmits the resulting digital audio signal to an audio processing unit214. The audio processing unit214performs audio-related processing, such as optimization processing, on the input digital audio signal. The first control unit223transmits the audio signal processed by the audio processing unit214to the memory215. The memory215temporarily stores the image signal and the audio signal obtained by the image processing unit207and the audio processing unit214.

The image processing unit207and the audio processing unit214read the image signal and the audio signal temporarily stored in the memory215, and encode the image signal and the audio signal to generate a compressed image signal and a compressed audio signal. The first control unit223transmits the compressed image signal and the compressed audio signal to a recording and reproduction unit220.

The recording and reproduction unit220records the compressed image signal and the compressed audio signal generated by the image processing unit207and the audio processing unit214, and other imaging-related control data, on a recording medium221. If the audio signal is not compression coded, the first control unit223transmits the audio signal generated by the audio processing unit214and the compressed image signal generated by the image processing unit207to the recording and reproduction unit220so that the audio signal and the compressed image signal are recorded on the recording medium221.

The recording medium221may be one built in the imaging apparatus101or a removable one. The recording medium221can record various types of data, including the compressed image signal, the compressed audio signal, and the audio signal generated by the imaging apparatus101. A medium having a larger capacity than the nonvolatile memory216is typically used as the recording medium221. Examples of the recording medium211may include all kinds of recording media, such as a hard disk, an optical disk, a magneto-optic disk, a compact disc recordable (CD-R), a digital versatile disc recordable (DVD-R), a magnetic tape, a nonvolatile semiconductor memory, and a flash memory.

The recording and reproduction unit220reads (reproduces) compressed image signals, compressed audio signals, audio signals, various types of data, and/or programs recorded on the recording medium221. The first control unit223transmits the read compressed image and audio signals to the image processing unit207and the audio processing unit214. The image processing unit207and the audio processing unit214temporarily store the compressed image and audio signals in the memory215, decode the signals by a predetermined procedure, and transmit the decoded signals to the video output unit217and an audio output unit218.

The audio input unit213includes a plurality of microphones mounted on the imaging apparatus101. The audio processing unit214can detect the direction of sound on a plane where the plurality of microphones is located. The direction of sound is used for a search and automatic imaging to be described below. The audio processing unit214also detects specific voice commands The audio processing unit214may be configured so that the user can register specific sound in the imaging apparatus101as a voice command aside from several commands registered in advance. The audio processing unit214also performs sound scene recognition. The sound scene recognition includes making a sound scene determination by using a network trained through machine learning based on a large amount of audio data in advance. For example, the audio processing unit214includes a network for detecting specific scenes, such as “cheers arising”, “hands clapping”, and “voice uttered”. The audio processing unit214is configured to output a detection trigger signal to the first control unit223in response to a specific sound scene or specific voice command being detected.

A power supply unit210supplies power for operating the first control unit223. The audio output unit218outputs a preset sound pattern from a speaker built in the imaging apparatus101during imaging, for example.

A light-emitting diode (LED) control unit224controls a preset on-off pattern of an LED mounted on the imaging apparatus101during imaging, for example.

The video output unit217includes a video output terminal, for example. The video output unit217transmits an image signal for displaying a video image on an external display connected. The audio output unit218and the video output unit217may be configured as an integrated terminal, such as a High-Definition Multimedia Interface (HDMI (registered trademark)) terminal.

A training processing unit219trains a neural network to the user's preferences by using a machine learning algorithm.

A communication unit222performs communication between the imaging apparatus101and an external apparatus. For example, the communication unit222transmits and receives data, such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal. The communication unit222also receives imaging-related control signals, such as imaging start and end commands and pan, tilt, and zoom driving control signals, and drives the imaging apparatus101based on instructions from an external apparatus capable of mutual communication with the imaging apparatus101. The communication unit222also transmits and receives information, such as various training-related parameters to be processed by the training processing unit219between the imaging apparatus101and the external apparatus. Examples of the communication unit222include an infrared communication module, a Bluetooth® communication module, a wireless local area network (LAN) communication module, and a Wireless Universal Serial Bus (USB) communication module, and a wireless communication module, such as a Global Positioning System (GPS) receiver.

Configuration of Control System Including Plurality of Cooperative Imaging Apparatuses

FIG. 3illustrates an example of a control system including a plurality of cooperative imaging apparatuses.

Imaging apparatuses101a,101b,101c,and101dcan communicate wirelessly with a controller unit (smart device)301having a communication function. The imaging apparatuses101a,101b,101c,and101dcan receive operation instructions transmitted to individual imaging apparatuses101a,101b,101c,and101dfrom the controller unit301(smart device) and transmit control information about the respective imaging apparatuses101a,101b,101c,and101dto the controller unit301. InFIG. 3, the imaging apparatuses101a,101b,101c,and101dand the smart device301each connect to an access point302, and communicate via the access point302to transfer information.

A configuration of the smart device301including a wireless LAN communication module will be described with reference toFIG. 4.

The smart device301is an information processing apparatus including, for example, a wireless LAN control unit401intended for a wireless LAN, a Bluetooth® Low Energy control unit402intended for Bluetooth® Low Energy, and a public wireless control unit406intended for public wireless communication. The smart device301further includes a packet transmission and reception unit403. The wireless LAN control unit401performs wireless LAN radio frequency (RF) control, communication processing, and protocol processing related to a driver for performing various types of control on wireless LAN communication compliant with the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard series and the wireless LAN communication. The Bluetooth® Low Energy control unit402performs Bluetooth® Low Energy RF control, communication processing, and protocol processing related to a driver for performing various types of control on Bluetooth® Low Energy communication and the Bluetooth® Low Energy communication. The public wireless control unit406performs public wireless communication RF control, communication processing, and protocol processing related to a driver for performing various types of control on public wireless communication and the public wireless communication. Examples of the public wireless communication include ones compliant with the International Multimedia Telecommunications (IMT) standard and the Long-Term Evolution (LTE) standard. The packet transmission and reception unit403performs processing for at least either transmitting or receiving packets related to the wireless LAN and the Bluetooth® Low Energy and public wireless communications. In this example, the smart device301is described to at least either transmit or receive packets during communication. However, communication modes other than packet switching, like circuit switching, may be used.

The smart device301further includes, for example, a control unit411, a storage unit404, a GPS reception unit405, a display unit407, an operation unit408, a motion data obtaining unit409, and a power supply unit410. The control unit411controls the entire smart device301by executing a control program stored in the storage unit404, for example. The storage unit404stores, for example, the control program to be executed by the control unit411, and various types of information, such as parameters to be used for communication. Various operations to be described below are implemented by the control unit411executing the control program stored in the storage unit404.

The power supply unit410supplies power to the smart device301. The display unit407has a function capable of visually perceptible information output as in a liquid crystal display (LCD) and an LED, and a function capable of sound output as in a speaker, for example. The display unit407displays various types of information. Examples of the operation unit408include a button for accepting the user's operation on the smart device301. The display unit407and the operation unit408may be implemented by a common member, such as a touch panel.

The motion data obtaining unit409includes an angular velocity meter (gyro sensor) for detecting the angular velocity of the smart device301about three axial directions. The motion data obtaining unit409also includes an acceleration meter (acceleration sensor) for detecting the acceleration of the smart device301about the three axial directions, and an azimuth meter (azimuth sensor, geomagnetic sensor) for detecting the earth's magnetic field. The control unit411calculates the rotation angle and the amount of displacement (the amounts of X-, Y-, and Z-axis movement) of the smart device301from the output values of the gyro sensor, the acceleration sensor, and the geomagnetic sensor. The motion data obtaining unit409may also include an atmospheric pressure sensor to obtain altitude based on a change in the atmospheric pressure, and use the altitude to detect the amount of displacement.

The GPS reception unit405receives GPS signals notified from satellites, analyzes the GPS signals, and estimates the current position (longitude and latitude information) of the smart device301. Alternatively, the current position of the smart device301may be estimated based on information about wireless networks nearby by using the Wi-Fi positioning system (WPS).

The smart device301exchanges data with the imaging apparatuses101by communication using the wireless LAN control unit401. For example, the imaging apparatuses101and the smart device301transmit or receive data such as an audio signal, an image signal, a compressed audio signal, and a compressed image. The smart device301receives imaging start information and object detection information from the imaging apparatus101. The smart device301issues imaging and other operation instructions to the imaging apparatus101.

In the configuration illustrated inFIG. 3, to use the smart device301as an imaging apparatus controller, the smart device301is configured to serve as a server and transmit and receive information to/from the imaging apparatuses101a,101b,101c,and101dvia the access point302. However, the smart device301may be configured to control the plurality of imaging apparatuses101a,101b,101c,and101dby other methods.

FIG. 6is a diagram illustrating an example of transfer via a server. The imaging apparatuses101a,101b,101c,and101dconnect wirelessly to a server602via the access point302and transfer information. A personal computer (PC)603connects to the server602via an access point601, and obtains information transferred to the server602by the imaging apparatuses101a,101b,101c,and101d.While in this example the imaging apparatuses101a,101b,101c,and101dand the PC603connect to the server602via the different access points302and601, the imaging apparatuses101a,101b,101c,and101dand the PC603may connect to the server602via the same access point. The PC603and the server602are not limited to the wireless connection, and may be connected in a wired manner.

The imaging apparatuses101a,101b,101c,and101dare not limited to the wireless connection, either, and may be connected in a wired manner. Power over Ethernet (PoE) may be used to supply power to the imaging apparatuses101a,101b,101c,and101dduring operation.

Imaging Apparatus Layout Setting Method

Take a case where a fixedly-installed plurality of imaging apparatuses101performs automatic framing imaging in a cooperative manner by controlling driving of their tilt rotation units104, pan rotation units105, and zoom units201. In such a case, layout information about the imaging apparatuses101and angle information about the optical axis directions of the imaging apparatuses101are to be found out in advance.

A simple method for obtaining the layout information and angle information about the installed imaging apparatuses101will be described.

An application dedicated to controlling a plurality of imaging apparatuses101is prepared in the smart device301. The application is configured so that the user can easily register the installation positions of the imaging apparatuses101by using the application, and the plurality of imaging apparatuses101can perform framing control in a cooperative manner during imaging.

Initially, imaging apparatuses101a,101b,101c,101d,and101eare installed at appropriate locations. A user700activates the application in the smart device301.FIG. 8illustrates a screen example of the activated application. The user700taps an imaging apparatus layout setting tab801, and the screen transitions to an imaging apparatus layout setting screen.FIGS. 9A and 9Billustrates the imaging apparatus layout setting screen. The user700moves to a location where the imaging apparatus101ais installed, brings the smart device301as close to the imaging apparatus101aas possible, and taps an imaging apparatus registration set button901to obtain layout information about the imaging apparatus101a.The user700similarly moves to the locations where the imaging apparatuses101b,101c,101d,and101eare installed, brings the smart device301as close to the imaging apparatuses101b,101c,101d,and101eas possible, and taps the imaging apparatus registration set button901to obtain layout information about the imaging apparatuses101b,101c,101d,and101e(FIG. 7). The installation positions of the registered imaging apparatuses101can be checked on a display section903. The display section903displays the installation positions of the imaging apparatuses101as seen from above in the direction of gravity by default.

A case where the user700moves from the installation location of the imaging apparatus101dto that of the imaging apparatus101eand registers the imaging apparatus101ewill be described as an example. When the imaging apparatus101dis registered, the number of registered imaging apparatuses101a,101b,101c,and101dis four. A number of registered imaging apparatuses display902is thus “4”. If the user700moves to the location of the imaging apparatus101eand taps the imaging apparatus registration set button901, the number of registered imaging apparatuses display902changes from “4” to “5”. The display section903additionally displays a new imaging apparatus position904as the installation position of the imaging apparatus101e.

Details of the method for displaying the installation positions of the imaging apparatuses101on the display section903will be described.

Initially, if the user700taps the imaging apparatus registration set button901at a position as close to the first imaging apparatus101aas possible, the XYZ coordinates of that position are registered as (0, 0, 0). The user700then moves to the installation position of the next imaging apparatus101band registers the imaging apparatus101b.Here, the installation position of the imaging apparatus101bis registered by calculating the moving distance from the point where the reference coordinates (0, 0, 0) are initially registered to the point where the imaging apparatus registration set button901is next tapped based on the gyro sensor, the acceleration sensor, and the azimuth sensor in the smart device301. Alternatively, GPS information in the smart device301may be used. The imaging apparatuses101may include GPS modules. The moving distance may be detected by estimating the current position by trilateration based on differences in the intensity of radio waves received from a plurality of wireless communication apparatuses. The moving distance maybe calculated by using such methods in combination.

The angles of the imaging apparatuses101can be calculated from the detection results of the acceleration sensors and azimuth sensors in the imaging apparatuses101.

Alternatively, the angles of the imaging apparatuses101may be calculated based on the detection results of the acceleration sensor and azimuth sensor in the smart device301, on the assumption that the user registers each imaging apparatus101with the relative angle between the imaging apparatus101and the smart device301at a fixed value. Suppose that the X, Y, and Z axes of an imaging apparatus101are defined as illustrated inFIG. 10A, and the X′, Y′, and Z′ axes of the smart device301are defined as illustrated inFIG. 10B. As illustrated inFIGS. 10C and 10D, the user700adjusts the Y′ direction of the smart device301to the optical axis direction (Z direction) of the imaging apparatus101situated at default pan and tilt angles, and taps the imaging apparatus registration set button901to set the position and angle of the imaging apparatus101. In such a manner, the azimuth angle of the imaging apparatus101can be obtained by using sensor information in the smart device301without equipping the imaging apparatus101with an azimuth sensor.

Through the foregoing method, the three axial angles of the imaging apparatus101can be calculated.

FIG. 11illustrates a processing procedure of the control unit411of the smart device301after the imaging apparatus registration set button901is tapped.

In step S1101, the control unit411determines whether the number of registered imaging apparatuses, N, is greater than 0. If N is 0 (i.e., no imaging apparatus is registered) (NO in step S1101), the processing proceeds to step S1102. In step S1102, the control unit411initially registers the installation position of the imaging apparatus [N] (imaging apparatus101a) as an initial position (0, 0, 0). In step S1101, if N is greater than 0 (YES in step S1101), the processing proceeds to step S1103. In step S1103, to determine the coordinates of the imaging apparatus [N], the control unit411calculates the relative position from the installation position of the imaging apparatus [N-1] to the current position. The control unit411then registers the installation position of the imaging apparatus [N].

After the processing of step S1102or S1103, the processing proceeds to step S1104. In step S1104, the control unit411obtains the installation angle of the imaging apparatus [N]. The processing proceeds to step S1105. In step S1105, the control unit411additionally displays the installation position of the imaging apparatus [N] in the display section903displaying the installation positions of imaging apparatuses. The processing proceeds to step S1106. In step S1106, the control unit411increments the number of registered imaging apparatuses N, and updates the number of registered imaging apparatuses displayed in the number of registered imaging apparatuses display902. The processing ends.

The procedure is repeated each time the imaging apparatus registration set button901is tapped. If the registration is reset by a separately-prepared reset setting, the control unit411resets the number of registered imaging apparatuses N to 0, and resets the information in the display section903displaying the installation positions of the imaging apparatuses.

Imaging Area Setting Method

If the screen returns to that ofFIG. 8and the user700taps an imaging area setting tab802, the screen transitions to an imaging area setting screen.FIGS. 12A to 12Cillustrate the imaging area setting screen. A display section1201displays the installation positions of the imaging apparatuses101, where the layout of the imaging apparatuses101set on the foregoing imaging apparatus layout setting screen is displayed. As illustrated inFIGS. 12A to 12C, the user700may paste a picture into the display section1201(in the illustrated example, a picture of a basketball court is pasted). A separately-prepared image captured from above may be pasted. An area1202displays an actual live video image of an imaging apparatus101. If the user700taps the installation position of an imaging apparatus101in the display section1201, the video image of the specified imaging apparatus101can be live displayed in the area1202. Buttons1203,1204, and1205are each provided for the purpose of changing mode. The button1203is provided as a mode switch for entering an “imaging area check mode” screen. The button1204is provided as a mode switch for entering an “overall imaging area setting mode” screen. The button1205is provided as a mode switch for entering an “individual imaging area setting mode” screen.

The “imaging area check mode” will initially be described.

Imaging Area Check Mode

If the button1203is tapped, the screen transitions to the “imaging area check mode” screen (FIGS. 12A to 12C).

As described above, the layout of the imaging apparatuses101is displayed in the display section1201. If no imaging area has been set by the user in the “overall imaging area setting mode” or “individual imaging area setting mode” to be described below, the display section1201displays all the areas where the imaging apparatuses101can capture an image by driving the tilt rotation units104, the pan rotation units105, and the zoom units201. The more areas where a plurality of imaging apparatuses101can capture an image overlap, the darker the display color. If areas where a plurality of imaging apparatuses101can capture an image do not overlap, the areas are displayed in light color. Based on the maximum focal length of each imaging apparatus101, up to what distance is treated as the imaging area of the imaging apparatus101is determined with the imaging apparatus101as an origin. Specifically, the radius from the installation position of an imaging apparatus101is determined to satisfy the condition that the imaging magnification at the maximum zoom position is higher than or equal to a predetermined value. Such a display enables the user to check in which area a multiple viewpoint video image can be obtained by a plurality of imaging apparatuses101and in which area imaging using a plurality of imaging apparatuses is difficult.

The display section1201displaying the installation positions of the imaging apparatuses101is capable of zooming in, zooming out, and rotation. The screen display can be zoomed out to observe a wider area by an operation of pinching the display section1201on the touch screen with two fingers (pinch-in operation). The screen can be zoomed in to observe the imaging areas more closely by an operation of spreading out the display section1201on the touch screen with two fingers (pinch-out operation).FIGS. 12A and 12Billustrate an example of the pinch-in operation. The angle of the screen display can be changed by an operation of sliding two fingers over the display section1201on the touch screen. For example, a vertical sliding operation can rotate the screen display with a horizontal axis as the rotation axis. A horizontal sliding operation can rotate the screen display with a vertical axis as the rotation axis (FIG. 12Cillustrates an example).

The center position of the screen display can be moved by a sliding operation with a single finger. The live video image of a specified imaging apparatus101can be displayed in the area1202by tapping the installation location of the imaging apparatus101.

Overall Imaging Area Setting Mode

If the button1204is tapped, the screen transitions to the “overall imaging area setting mode” screen (FIGS. 13A to 13F).

In the “overall imaging area setting mode”, the user700manually operates an area (imaging area) to capture an image on the screen displayed in the display section1201displaying the installation positions of the imaging apparatuses101, so that a user-intended video image is more likely to be captured when the plurality of imaging apparatuses101performs automatic framing imaging. The imaging apparatuses101are automatically controlled to capture few images in areas not specified as the imaging area here.FIG. 13Aillustrates an example where the user700specifies an imaging area1301in the screen display of the display section1201displaying the installation positions of the imaging apparatuses101.

As in the method described with reference toFIGS. 12A to 12C, the display section1201displaying the installation positions of the imaging apparatuses101is capable of zooming in, zooming out, and rotation. The screen display can be zoomed out to observe a wider area by an operation of pinching the display section1201on the touch screen with two fingers (pinch-in operation). The screen display can be zoomed in to observe the imaging area1301more closely by an operation of spreading out the display section1201on the touch screen with two fingers (pinch-out operation). The angle of the screen display can be changed by an operation of sliding two fingers over the display section1201on the touch screen in the same direction. For example, a vertical sliding operation can rotate the screen display with a horizontal axis as the rotation axis. A horizontal sliding operation can rotate the screen display with a vertical axis as the rotation axis (FIG. 13Cillustrates an example). With the direction of gravity as the Z-axis direction, an imaging area can be specified within the range of the X- and Y-axes and whether an area is capable of being comprehensively imaged can be checked in a view displayed in the Z-axis direction as illustrated inFIGS. 13A and 13B. In addition, an imaging range in the Z-axis direction can also be specified.FIG. 13Dillustrates an example where the screen display is rotated to a screen angle at which the Z-axis direction can be specified, and in which state the user700specifies the imaging area in the Z-axis direction. An area capable of being comprehensively imaged by the arranged imaging apparatuses101is displayed as with an area1304. An area determined to not be capable of being comprehensively imaged is displayed as with an area1305.

The area1202displays the live video image of a specified imaging apparatus101. If the user700taps the installation position of an imaging apparatus101in the display section1201as inFIG. 13E, the live video image of the specified imaging apparatus101is displayed in the area1202.FIG. 13Fillustrates a display example. The display section1201displays the specified imaging apparatus101in different color, shape, or size so that which imaging apparatus101is specified can be observed. An angle of view range display1307indicating the current display angle of view of the specified imaging apparatus101is also displayed at the same time. The live video image in the area1202is also configured to indicate the range of the imaging area1301specified manually the user700(for example, an area1308is an area specified as the imaging area; an area1306not specified as the imaging area is displayed in gray).

In such a manner, the user can specify an imaging area with a simple operation, and can visualize an area capable of being comprehensively imaged and an area not capable of being comprehensively imaged. In addition, the user can observe the inside and outside of the range of the specified area while monitoring the actual live video image of an imaging apparatus101.

Individual Imaging Area Setting Mode

If the button1205is tapped, the screen transitions to the “individual imaging area setting mode” screen (FIGS. 14A to 14F).

In the “individual imaging area setting mode”, the imaging areas of the respective imaging apparatuses101can be specified in detail. If the user700taps the installation position of an imaging apparatus101of which the user wants to set the imaging area on the screen displayed in the display section1201displaying the installation positions of the imaging apparatuses101as illustrated inFIG. 14A, the live video image of the specified imaging apparatus101is displayed in the area1202.FIG. 14Billustrates a display example. The display section1201displays the specified imaging apparatus101in different color, shape, or size so that which imaging apparatus101is specified can be visually observed. An angle of view range display1401indicating the current display angle of view of the specified imaging apparatus101is also displayed at the same time. The user700specifies an imaging area within the display screen by making a sliding operation on the area1202displaying the live video image on the touch screen with a single finger. The user700can specify the range of the imaging area by making a sliding operation of horizontally sliding the finger over the screen, such as a movement from the point inFIG. 14Bto the point inFIG. 14C. If the sliding operation reaches near the screen end, the specified imaging apparatus101is driven to pan in the horizontal (pan) direction. The user700can thereby specify the imaging area while changing the optical axis of the imaging apparatus101. A sliding operation in the vertical (tilt) direction can also be made in a similar manner. The user700can specify the range of the imaging area by making a sliding operation of vertically sliding the finger over the screen such as a movement from the point inFIG. 14Cto the point inFIG. 14D. If the sliding operation reaches near the screen end, the specified imaging apparatus101is driven to tilt in the vertical (tilt) direction. This facilitates specifying the imaging area by changing the optical axis. The user700may want to change the optical axis of the imaging apparatus101by pan and tilt driving without specifying an imaging area. In such a case, the screen display can be moved in the vertical and horizontal (tilt and pan) directions without specifying an imaging area, for example, by a sliding operation with two fingers. An operation of pinching the area1202on the touch screen with two fingers (pinch-in operation) increases the angle of view by zoom driving, so that the screen display can be zoomed out to allow the user700to observe a wider area. An operation of spreading out the area1202on the touch screen with two fingers (pinch-out operation) reduces the angle of view by zoom driving, so that the screen display can be zoomed in to allow the user700to observe the imaging area more closely.

An area where the imaging apparatus101performs automatic zoom driving may be specified by the user700.

If the user700wants to cancel a specified imaging area, for example, the user700taps the specified imaging area twice to display a “whether to cancel” message on the screen. If cancel OK is specified, the specified imaging area is cancelled. Alternatively, “specify” and “cancel” touch buttons may be provided on the touch screen, and the user700may tap the “cancel” button to cancel the imaging area specified by the sliding operation on the area1202.

By using the foregoing method, the imaging areas of the respective imaging apparatuses101can be specified one by one. If the button1203is tapped after the imaging areas of the respective imaging apparatuses101are specified on the “individual imaging area setting mode” screen, the screen transitions to the “imaging area check mode” screen (described with reference toFIGS. 12A to 12C).FIG. 14Eillustrates a display example of the imaging areas individually specified by the method described in conjunction withFIGS. 14A to 14D. Based on the specified imaging areas of the respective imaging apparatuses101, whether each area is comprehensively imaged can be visually observed in terms of the depth of color (the more areas where a plurality of imaging apparatuses101can capture an image overlap, the darker the display color. If areas where a plurality of imaging apparatuses101can capture an image do not overlap, the areas are displayed in light color).

FIGS. 14E and 14Fillustrate an example of a pinch-in operation. The screen display is zoomed out by the pinch-in operation, and the imaging areas can be observed on a screen display where a wide area can be observed.

In such a manner, the user700can individually specify the imaging areas of the respective imaging apparatuses101with simple operations, and can visualize areas capable of being comprehensively imaged and areas not capable of being comprehensively imaged.

The user700can easily specify imaging areas using a plurality of imaging apparatuses101through the foregoing method. Cooperative framing adjustment in the specified imaging areas by the plurality of imaging apparatuses101and automatic imaging around the specified imaging areas by the plurality of imaging apparatuses101are supported.

Remote Control Setting Mode

If the screen returns to that ofFIG. 8and the user700taps a remote control tab803, the screen transitions to a remote control screen.FIG. 5Aillustrates the remote control screen. A display section1701displays the installation positions of the imaging apparatuses101. The display section1701displays the layout of the imaging apparatuses101set on the foregoing imaging apparatus layout setting screen and the imaging areas set on the imaging area setting mode screen in an identifiable manner. The display section1701can also visualize in which direction each imaging apparatus101is currently facing and what angle of view range each imaging apparatus101has. If the angle of view or the direction of the optical axis of an imaging apparatus101is automatically or manually changed, the directional display also changes at the same time. An area1702displays the actual live video image of an imaging apparatus101. If the user700taps the installation position of an imaging apparatus101on the display section1701, the video image of the specified imaging apparatus101can be live displayed in the area1702. Which imaging apparatus101is currently under live display can be seen in the display section1701. The display section1701displays the installation position of the imaging apparatus101in different color, shape, or size like an installation position1706. An imaging button1704can be used to give instructions to start capturing a moving image, capture a still image, or start automatic imaging. An imaging apparatus setting button1705can be used to change the settings of the imaging apparatus101. If the imaging apparatus setting button1705is tapped, an imaging apparatus setting menu is displayed. For example, resolution, frame rate, and white balance settings can be manually operated from the imaging apparatus setting menu.

If the user700wants to remotely operate the imaging apparatus101, the user700taps a remote control operation button1703to enter a remote operation screen.FIG. 5Billustrates the remote operation screen. The remote operation screen displays an operation section1707capable of pan and tilt operations and an operation section1708capable of zoom operations. The user700drives the specified imaging apparatus101to tilt by touching at up and down icons in the operation section1707and to pan by touching at left and right icons. The optical axis of the imaging apparatus101can thereby be changed. Moreover, the imaging apparatus101is driven to zoom in a direction of narrowing the angle of view (to a telescopic side) by an upward sliding operation on a switch icon in the operation section1708, and to zoom in a direction of widening the angle of view (to a wide side) by a downward sliding operation on the switch icon. The angle of view can be thus changed.

In the foregoing example, the operation sections1707and1708are described to be displayed. However, the imaging apparatus101may be driven to zoom in and out by making pinch-out and pinch-in operations within the area1702on the touch screen, and driven to pan and tilt by making a sliding operation, without displaying the operation sections1707and1708.

If the user700wants to automatically drive the imaging apparatus101to pan, tilt, and zoom so that a specified object is kept positioned at a predetermined position on the screen (for example, near the screen center), the user700may specify the object by a touch operation as illustrated inFIG. 5C. If an object is specified by a touch operation, the imaging apparatus101controls automatic object tracking and displays an object frame1709so that the currently-tracked object is visually identifiable as illustrated inFIG. 5D. The object tracking can be continued until cancelled. For example, a cancellation button may be displayed and the object tracking may be cancelled if the cancellation button is touched. Alternatively, the object tracking may be cancelled if the optical axis or the angle of view of the imaging apparatus101is manually changed by using the operation section1707or1708.

During an automatic framing imaging operation, a specified imaging area can become unable to be imaged because of a dead angle behind an obstacle (for example, a person can come and remain in front of the imaging apparatus101). A warning display in such a case will now be described.

Since the imaging areas are specified as described in conjunction withFIGS. 9 to 14F, a dead angle is determined to occur if a foreground object is detected closer to an imaging apparatus101than the range set as its imaging area and the object occupies an area greater than a predetermined value in the imaging range. In such a case, the application in the smart device301displays a warning. For example, the smart device301is notified of the imaging apparatus101in which the dead angle occurs. The application informs the user of the warning by blinking the icon of the imaging apparatus101in the display section1701or providing an error display. The distance may be measured by any method, including a focus-based method and one using an external sensor. Alternatively, suppose that a plurality of imaging apparatuses101is tracking the same object and capturing an image thereof at the same time, or capturing an image of the same imaging area. In such a case, a warning may be displayed if image information of one imaging apparatus101does not coincide with that of another. Examples of the image information include object detection information and feature information such as hue and saturation in the images.

Automatic Framing/Imaging Processing

Next, details of automatic imaging processing will be described. If an instruction to start imaging control is given from the button1704, an imaging operation is started. Imaging mode processing will be described with reference toFIG. 15.

In step S1501, the image processing unit207generates an image intended for object detection by performing image processing on the signal captured by the imaging unit206. The first control unit223performs object detection, such as human detection and general object detection based on the generated image.

In the case of human detection, the first control unit223detects the object's face or human body. For face detection processing, patterns for determining a human face are provided in advance, and a region matching a pattern in the captured image can be detected as a human face image. The first control unit223also calculates a degree of reliability indicating the likelihood of the object being a face at the same time. For example, the degree of reliability is calculated based on the size of the face region in the image and the degree of matching with the pattern. Similarly, in the case of general object detection, the first control unit223can recognize a general object matching a previously registered pattern. Alternatively, a characteristic object can be extracted through a method using hue and saturation histograms of the captured image. Here, distributions derived from the hue or saturation histograms of an object image captured within the imaging angle of view are divided into a plurality of intervals. Processing for classifying the captured image interval by interval is then performed.

For example, the first control unit223generates histograms of a plurality of color components of the captured image. The first control unit223divides the histograms into unimodal intervals, classifies images captured in regions belonging to the combination of the same intervals, and recognizes the object image regions. The first control unit223calculates evaluation values for the respective object image regions recognized, and thus, the object image region having the highest evaluation value can be determined as a main object region. Convolutional neural networks (CNNs) may be trained to detect intended objects in advance, and the CNNs may be applied to the face detection and the general object detection. By using such a method, pieces of object information can be obtained from the captured image.

In step S1502, the first control unit223performs object search processing. The object search processing includes the following processes:

(1) Area division

(2) Area-by-area calculation of importance levels

(3) Determination of the area to be searched

The processes will be described below in order.

(1) Area Division

The area division will be described with reference toFIGS. 16A to 16D.

The first control unit223performs area division all over with the position of the imaging apparatus101at the center (with the position of the imaging apparatus101as an origin O) as illustrated inFIG. 16A. In the example ofFIG. 16A, the areas are divided in units of 22.5° in both the tilt and pan directions. If the entire area is divided as illustrated inFIG. 16A, the horizontal circumference becomes shorter and the areas smaller as the angle in the tilt direction of the imaging apparatus101gets farther away from 0°.

As illustrated inFIG. 16B, the areas at a tilt angle of 45° or more are therefore horizontally divided in units greater than 22.5°.

FIGS. 16C and 16Dillustrate examples of area division within the imaging angle of view. An axis1601represents the direction of the imaging apparatus101when initialized. The area division is performed with the directional angle as a reference position. An angle of view area1602represents the image being captured.FIG. 16Dillustrates an example of the captured image here. The image captured within the angle of view is divided into areas1603to1618ofFIG. 16Dbased on the area division.

(2) Area-by-Area Calculation of Importance Levels

The first control unit223calculates importance levels indicating the order of priority in a search. The first control unit223calculates the importance levels of each of the areas divided as described above based on the state of an object or objects in the area and the state of the scene of the area. The importance level based on the state of an object or objects is calculated, for example, based on the number of human figures in the area, face sizes and face directions of the human figures, the probability of face detection, facial expressions of the human figures, and personal authentication results of the human figures. The importance level based on the state of the scene is calculated, for example, based on a general object recognition result, a scene discrimination result (such as blue sky, backlight, and twilight view), the level of sound from the direction of the area, a voice recognition result, and motion detection information within the area.

If the imaging areas are specified by the user700by using the method described in conjunction withFIGS. 12A to 14F, the importance levels of areas located in unspecified imaging areas are fixed to a minimum value so that the imaging apparatuses101make searching and framing operations within the respective specified imaging areas. This precludes a search of those areas.

If the importance levels of the areas remain unchanged under the foregoing conditions alone, the area of the highest importance level remains the same and thus the area to be searched remains unchanged unless a change occurs in the respective areas. To avoid this, the first control unit223changes the importance levels based on past imaging information. Specifically, the importance level of an area continuously specified as a search area for a predetermined period of time may be lowered. The importance level of an area where an image is captured in step S1508to be described below may be lowered for a predetermined period of time. The first control unit223does not change but maintains the importance levels of the areas not specified as imaging areas by the user at the minimum value.

(3) Determination of Area to be Searched

With the importance levels of the areas calculated as described above, the first control unit223determines the area having the highest importance level to be the area to be searched. The first control unit223then calculates pan and tilt search target angles for capturing the search target area within the angle of view.

In step S1503, the first control unit223performs pan and tilt driving. The first control unit223calculates the amounts of pan and tilt driving by adding driving angles obtained by control sampling based on the pan and tile search target angles. The lens barrel rotation driving unit205controls driving of the tilt rotation unit104and the pan rotation unit105.

In step S1504, the zoom driving control unit202controls the zoom unit201for zoom driving. Specifically, the zoom driving control unit202drives the zoom unit201to zoom based on the state of an object to be searched for determined in step S1502. For example, if the object to be searched for is a human face, too small a face on the image can fall below a minimum detectable size and be lost track of due to a detection failure. In such a case, the zoom driving control unit202controls the zoom unit201so that the zoom unit201zooms in to the telescopic side to increase the size of the face on the image. On the other hand, if the face on the image is too large, the object can easily go out of the angle of view due to movement of the object or the imaging apparatus101itself. In such a case, the zoom driving control unit202controls the zoom unit201so that the zoom unit201zooms out to the wide side to reduce the size of the face on the image. Such zoom control can maintain a state suitable to keep track of the object.

While the object search in steps S1502to S1504is described to be performed by pan, tilt, and zoom driving, an imaging system that captures images in all directions at a time by using a plurality of wide angle lenses may be used for object search. In the case of an omnidirectional imaging apparatus, performing image processing, such as object detection, using all the captured signals as an input image involves an enormous amount of processing. In such a case, the first control unit223crops a part of the image and performs object search processing within the cropped image. The first control unit223calculates the importance levels of respective areas in a manner similar to the foregoing method, changes the cropping position based on the importance levels, and makes an automatic imaging determination to be described below. Such a configuration can reduce the power consumption of the image processing and enables fast object search.

In step S1505, the first control unit223determines whether a manual imaging instruction is given. If the imaging instruction is given (YES in step S1505), the processing proceeds to step S1506. The manual imaging instruction can be given by pressing a shutter button, by lightly tapping the casing of the imaging apparatus101with a finger, by voice command input, or as an instruction from an external device. The method for giving an imaging instruction based on a tap operation uses a series of high-frequency accelerations detected in a short time by the apparatus vibration detection unit209as an imaging trigger when the user taps the casing of the imaging apparatus101. The method for giving an imaging instruction by voice command input uses a voice recognized by the audio processing unit214as an imaging trigger when the user utters a predetermined cue phrase for imaging instruction (such as “take a picture”). The method for giving an imaging instruction as an instruction from an external device uses as a trigger a shutter instruction signal that is transmitted from, for example, a smart phone connected to the imaging apparatus101over wireless communication via a dedicated application.

In step S1506, the first control unit223makes an automatic imaging determination. The automatic imaging determination determines whether to perform automatic imaging.

Determination About Whether to Perform Automatic Imaging

Whether to perform automatic imaging is determined based on the following two determinations. One is a determination based on the area-specific importance levels obtained in step S1502. If the importance levels exceed a predetermined value, the first control unit223determines to perform automatic imaging. The other is a determination based on a neural network. The neural network is used to estimate an output value from input values. A neural network trained with input values and exemplary output values for the input values in advance can estimate an output value following the trained examples from new input values. The training method will be described below. In the determination based on the neural network, objects captured in the current angle of view and feature amounts based on the states of the scene and the imaging apparatus101are input to neurons in an input layer. A value output from an output layer through calculations based on a multilayer perceptron forward propagation method is thereby obtained. If the output value is greater than or equal to a threshold, automatic imaging is determined to be performed. Examples of object features include the current zoom magnification, a general object recognition result in the current angle of view, a face detection result, the number of faces captured in the current angle of view, a degree of smiling and a degree of eye closure of the face or faces, face angles, face authentication identification (ID) numbers, and the line of sight angle of an object person. In addition, a scene discrimination result, the elapsed time from the previous imaging, the current time, GPS position information, the amount of change from the previous imaging position, the current sound level, the person uttering a voice, and the presence or absence of handclapping and cheers may be used. Vibration information (acceleration information or the state of the imaging apparatus) and environmental information (temperature, atmospheric pressure, illuminance, humidity, and the amount of ultraviolet rays) may also be used. The first control unit223converts such features into numerical values in a predetermined range, and inputs the numerical values to the respective neurons of the input layer as feature amounts. As many neurons of the input layer as the number of feature amounts to be used are thus used.

The training processing unit219can change the connection weights between the neurons to change the output value, and thus a result of the neural network-based determination can be adapted to the training result.

Determination of Imaging Method

In determining an imaging method, the first control unit223determines which imaging method to perform, still image capturing or moving image capturing, based on the state of an object or objects nearby detected in step S1501. For example, if the object(s) (person(s)) is/are standing still, the first control unit223determines to perform still image capturing. If the object(s) is/are moving, the first control unit223determines to perform moving image capturing or continuous shooting. A neural network-based determination may be made. The user can manually change the settings of the imaging apparatus101by using a dedicated application. The imaging apparatus101can be set to capture only still images, only moving images, or capture and save both.

In step S1507, if automatic imaging is determined to be performed by the automatic imaging determination in step S1506(YES in step S1507), the processing proceeds to step S1508. If not (NO in step S1507), the imaging mode processing ends.

In step S1508, the imaging apparatus101starts imaging. Here, the imaging apparatus101starts to capture an image through the imaging method determined in step S1506. In the meantime, the focus driving control unit204performs automatic focus control. The imaging apparatus101also performs exposure control by using a not-illustrated aperture control unit, sensor gain control unit, and shutter control unit so that the object(s) has/have appropriate brightness. After the imaging, the image processing unit207performs various types of conventional image processing, such as automatic white balance processing, noise reduction processing, and gamma correction processing, and generates an image.

In the case of moving image capturing, the imaging apparatus101captures the moving image while performing framing operations by pan, tilt, and zoom driving based on the object detection as described in steps S1501to S1504even during imaging and recording. As in the foregoing method, a search based on the area-by-area importance levels may be performed. A large-scale search operation may be disabled during moving image capturing. A specific object may be registered, and the imaging apparatus101may capture a moving image while keeping track of the registered object within a specified imaging area by pan, tilt, and zoom driving so that the registered object is kept positioned near the screen center.

In step S1509, the first control unit223performs editing processing for processing the image generated in step S1508or adding the image to a moving image. Specific examples of the image processing include trimming processing based on a human face or an in-focus position, image rotation processing, and application of effects, such as a high dynamic range (HDR) effect, a blurring effect, and a color conversion filter effect. These processes may be combined to generate a plurality of processed images from the image generated in step S1508, and the processed images may be stored separate from the image generated in step S1508. In the case of moving image processing, the first control unit223may apply special effect processing such as sliding, zooming, and fading to the captured moving image or still image, and add the resulting image to an already-generated edited moving image.

In step S1510, the first control unit223updates the past imaging information. Specifically, the first control unit223increments the following counts corresponding to the image captured this time by one: the numbers of captured images in the respective areas described in step S1506, the numbers of captured images of respective authenticated and registered persons, the numbers of captured images of respective objects recognized by general object recognition, and the numbers of captured images of respective scenes discriminated through scene discrimination.

By using the foregoing method, the user700can easily specify imaging areas using a plurality of imaging apparatuses101. Cooperative framing adjustment in the specified imaging areas by the plurality of imaging apparatuses101and automatic imaging around the specified imaging areas by the plurality of imaging apparatuses101are thus supported.

By using the foregoing method, the user700can specify imaging areas with a simple operation. The plurality of imaging apparatuses101then cooperatively makes a framing adjustment in the specified imaging areas and performs automatic imaging around the specified imaging areas, so that automatic imaging highly likely to capture a user-desired video image can be implemented.

The present exemplary embodiment has been described by using an example where a plurality of imaging apparatuses101having the pan, tilt, and zoom configurations illustrated inFIGS. 1A and 1Bis used. All the plurality of imaging apparatuses101used may have the pan, tilt, and zoom configurations illustrated inFIGS. 1A and 1B. Imaging apparatuses having the zoom configuration without a pan or tilt configuration may be used. Imaging apparatuses having the pan and tilt configurations without a zoom configuration may be used. Some of the imaging apparatuses101may have a fixed focal length without a zoom, pan, or tilt configuration. An omnidirectional imaging apparatus that includes a plurality of image sensors and a wide angle optical system and captures images in all directions at a time may be used.

Other Embodiments

An exemplary embodiment of the present invention can be implemented by processing for supplying a program for implementing one or more of the functions of the foregoing exemplary embodiment to a system or an apparatus via a network or a recording medium, and reading and executing the program by one or more processors in a computer of the system or apparatus. A circuit for implementing one or more functions (for example, application specific integrated circuit (ASIC)) may be used for implementation.

An exemplary embodiment of the present invention is not limited to imaging by a digital camera or a digital video camera, and can also be implemented on an information processing apparatus that communicates with imaging apparatuses, such as a surveillance camera, a web camera, and a mobile phone. The information processing apparatus is not limited to a mobile phone such as a smartphone, and may be a tablet computer.

According to an exemplary embodiment of the present invention, an information processing apparatus that facilitates checking an imaging area in operating a plurality of imaging apparatuses in a cooperative manner and a control method thereof can be provided.

The present invention is not limited to the foregoing exemplary embodiments, and various changes and modifications can be made without departing from the spirit and scope of the present invention. The following claims are therefore attached to make public the scope of the present invention.