Photographing control apparatus and photographing control method

A photographing control apparatus includes a memory configured to store a first image that is photographed by a first camera and the second image that is photographed by a second camera the second image including an image of a user who has visibility of the first image being displayed in a display, and a processor coupled to the memory and configured to acquire the first image and the second image, detect the line of sight of the user based on the second image, identify an object displayed at the position of the line of sight of the user from the first image displayed in the display, and control a photographing direction or photographing magnification of the first camera or both photographing direction and photographing magnification of the first camera so that the object is continuously included in the first image for a predetermined time period or longer.

FIELD

The embodiment discussed herein is related to a photographing control apparatus and a photographing control method.

BACKGROUND

The following technique has been proposed: when an operational button is pressed, the visual field direction of a camera is automatically controlled so that an object displayed at the position of an operation marker superimposed and displayed on an image photographed by the camera is located at the center of the visual field of the camera. In this technique, a predetermined region whose center is located at the position of the operational marker when the operational button is pressed is registered as a template image, and a portion that is included in the photographed image and has the highest similarity to the template image is automatically traced by the camera until the portion is located at the center of the image.

An example of related art is Japanese Laid-open Patent Publication No. 2002-359766.

SUMMARY

According to an aspect of the invention, a photographing control apparatus includes a memory configured to store a first image that is photographed by a first camera and the second image that is photographed by a second camera the second image including an image of a user who has visibility of the first image being displayed in a display, and a processor coupled to the memory and configured to acquire the first image and the second image, detect the line of sight of the user based on the second image, identify an object displayed at the position of the line of sight of the user from the first image displayed in the display, and control a photographing direction or photographing magnification of the first camera or both photographing direction and photographing magnification of the first camera so that the object is continuously included in the first image for a predetermined time period or longer.

DESCRIPTION OF EMBODIMENT

For example, in a facility in which care such as nursing care or child care is provided, a caregiver is responsible for the care of multiple persons to be cared in general, and during the time when the caregiver helps a certain person to be cared, the caregiver may temporarily pay insufficient attention to the other persons. It is therefore considered that the persons who are to be cared and to which insufficient attention is temporarily paid by the caregiver are watched by displaying a monitoring image photographed by a monitoring camera and indicating a care site in a display within another room and visually checking the monitoring image by a user such as administrative staff of the facility. However, since the user such as the administrative staff of the facility is responsible for the task of visual checking the monitoring image and a task other than the visual checking, it is requested that the additional task of visually checking the monitoring image and watching the persons to be cared, for example, a task of controlling the monitoring camera to trace a target person, be reduced as much as possible.

When the aforementioned technique is applied and a user performs a task of temporarily specifying a person to be cared and watched on the monitoring image, the specified person to be cared is automatically traced by the monitoring camera. In this case, however, since the monitoring camera is controlled, the task of specifying the person to be cared and watched on the monitoring image by the user is not omitted. In the care site, a situation may frequently change, a person to be cared and watched may be switched, or multiple persons to be cared may be to be watched simultaneously. It is, therefore, desirable to reduce the burden on the user who visually checks the monitoring image.

According to an aspect, an object of the present disclosure is to provide a technique for controlling a camera configured to photograph a monitoring image in which a target to be monitored is included in a photographing region.

Hereinafter, an embodiment of the technique disclosed herein is described in detail with reference to the accompanying drawings.FIG. 1illustrates a photographing control apparatus10. The photographing control apparatus10is connected to multiple monitoring cameras42that are installed at different locations in a care site of a care facility in which care such as nursing care or child care is provided and are configured to photograph the care site and output monitoring images. Photographing range changers44configured to change photographing ranges of the monitoring cameras42by driving forces such as motors are attached to the monitoring cameras42and connected to the photographing control apparatus10.

The embodiment describes a case where the photographing range changers44change the photographing ranges of the corresponding monitoring cameras42by changing photographing directions of the corresponding monitoring cameras42. The photographing control apparatus10controls the photographing ranges of the monitoring cameras42by outputting, to the corresponding photographing range changers44, a control signal for controlling the photographing directions of the monitoring cameras42.

In addition, a main display46, an auxiliary display48, a user camera50, and a notifier52are connected to the photographing control apparatus10and installed in, for example, a room that is different from the care site and is located in the care facility and in which a user such as administrative staff of the care facility performs a task, for example. The photographing control apparatus10causes multiple monitoring images photographed by the multiple monitoring cameras42to be displayed in the main display46and the auxiliary display48.

Normally, each of the monitoring images includes one or more persons to be cared as objects. During a task for which the user is responsible, the user visually checks a monitoring image displayed in the main display46or the auxiliary display48and confirms whether or not a person who is to be cared and to which insufficient attention is temporarily paid by a caregiver exists. If the person who is to be cared and to which insufficient attention is temporarily paid by the caregiver exists, the user watches the person to be cared on the monitoring image and performs a care support task or contacts the caregiver or the like.

The user camera50photographs the user visually checking the monitoring image displayed in the main display46or the auxiliary display48and outputs a user image. The user camera50may be a visible light camera or an infrared light camera having an infrared light source. The notifier52includes a buzzer for notifying the user of the occurrence of an abnormality.

The photographing control apparatus10includes an acquirer12, a display controller14, a line-of-sight detector16, an identifying section18, a photographing controller20, a first storage section32, and a second storage section38. The identifying section18includes an object extractor22, a continuous photographing target identifying section24, a registered person identifying section26, an abnormality determiner28, and a notification signal output section30.

The acquirer12acquires the multiple monitoring images photographed by the multiple monitoring cameras42and the user image photographed by the user camera50. The line-of-sight detector16detects the direction of the line of sight of the user based on the user image photographed by the user camera50. The display controller14causes one of the multiple images photographed by the multiple monitoring cameras42to be displayed as a main monitoring image on an overall screen of the main display46and causes the remaining images to be displayed as thumbnail images in a matrix in the auxiliary display48(refer toFIG. 3A). In addition, the display controller14acquires the direction of the line of sight of the user from the line-of-sight detector16and switches the monitoring image displayed in the main display46to another monitoring image based on the direction of the line of sight of the user.

The identifying section18identifies, from objects included in the monitoring images photographed by the monitoring cameras42, an object to be set as a continuous photographing target and continuously photographed for a predetermined time period or longer. Specifically, the object extractor22of the identifying section18extracts all the objects (persons) included in the monitoring images and causes the results of extracting the objects to be stored as object information34in the first storage section32for the monitoring cameras42. Then, the continuous photographing target identifying section24identifies, from the objects extracted by the object extractor22from the monitoring images, the object displayed at the position of the line, detected by the line-of-sight detector16, of sight of the user on the display screen of the main display46as the continuous photographing target.

The registered person identifying section26identifies, as a continuous photographing target from the objects extracted by the object extractor22from the monitoring images, an object having a characteristic stored as registered person information40in the second storage section38. The abnormality determiner28determines whether or not each of the objects extracted by the object extractor22from the monitoring images has made an abnormal motion. The abnormality determiner28identifies, as a continuous photographing target, an object that has made an abnormal motion. If the abnormality determiner28detects an object that has made an abnormal motion, the notification signal output section30causes the notifier52to operate and output a notification signal for notifying the user of the abnormal motion.

The photographing controller20outputs a control signal for controlling a photographing range of a monitoring camera42to a photographing range changer44corresponding to the monitoring camera42so that the object identified by the identifying section18as the continuous photographing target is continuously photographed as a monitoring image for the predetermined time period or longer. The first storage section32has, stored therein, monitoring camera information36indicating positional relationships (for example, a positional relationship indicating that the photographing range of a monitoring camera y is adjacent to the photographing range of a monitoring camera x on the right side of the photographing range of the monitoring camera x and the like) between the photographing ranges of the monitoring cameras42.

The acquirer12is an example of an acquirer included in the technique disclosed herein. The line-of-sight detector16is an example of a detector included in the technique disclosed herein. The identifying section18including the continuous photographing target identifying section24is an example of an identifying section included in the technique disclosed herein. The photographing controller20is an example of a controller included in the technique disclosed herein. The object extractor22is an example of an extractor included in the technique disclosed herein. The registered person identifying section26is an example of a registered person identifying section included in the technique disclosed herein. The abnormality determiner28is an example of an abnormality determiner included in the technique disclosed herein. The notification signal output section30is an example of a notification signal output section included in the technique disclosed herein.

The photographing control apparatus10may be achieved by a computer60illustrated inFIG. 2, for example. The computer60includes a CPU62, a memory64, and a nonvolatile storage section66. The memory64serves as a temporal storage region. The computer60also includes a camera interface (I/F)68and a display I/F70. The camera I/F68is connected to the user camera50and the monitoring cameras42including the photographing range changers44. The display I/F70is connected to the main display46, the auxiliary display48, and the notifier52. The computer60also includes an input and output I/F74and a reading and writing device (R/W)78. The input and output I/F74is connected to an input and output device72. The R/W78reads and writes data from and in a recording medium76. The CPU62, the memory64, the storage section66, the camera I/F68, the display I/F70, the input and output I/F74, and the R/W78are connected to each other via a bus79.

The storage section66is achieved by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. In the storage section66, a photographing control program80for causing the computer60to function as the photographing control apparatus10is stored. The CPU62reads the photographing control program80from the storage section66, loads the read photographing control program80into the memory64, and sequentially executes processes included in the photographing control program80.

The photographing control program80includes an acquisition process82, a display control process84, a line-of-sight detection process86, an identification process88, and a photographing control process90. The identification process88includes an object extraction process92, a continuous photographing target identification process94, a registered person identification process96, an abnormality determination process98, and a notification signal output process100.

The CPU62executes the acquisition process82, thereby operating as the acquirer12illustrated inFIG. 1. The CPU62executes the display control process84, thereby operating as the display controller14illustrated inFIG. 1. The CPU62executes the line-of-sight detection process86, thereby operating as the line-of-sight detector16illustrated inFIG. 1. The CPU62executes the identification process88, thereby operating as the identifying section18illustrated inFIG. 1. The CPU62executes the photographing control process90, thereby operating as the photographing controller20illustrated inFIG. 1.

The CPU62executes the object extraction process92, thereby operating as the object extractor22illustrated inFIG. 1. The CPU62executes the continuous photographing target identification process94, thereby operating as the continuous photographing target identifying section24illustrated inFIG. 1. The CPU62executes the registered person identification process96, thereby operating as the registered person identifying section26illustrated inFIG. 1. The CPU62executes the abnormality determination process98, thereby operating as the abnormality determiner28illustrated inFIG. 1. The CPU62executes the notification signal output process100, thereby operating as the notification signal output section30illustrated inFIG. 1. Thus, the computer60executes the photographing control program80, thereby functioning as the photographing control apparatus10.

The storage section66has an object information storage region102, a monitoring camera information storage region104, and a registered person information storage region106. In the object information storage region102, the object information34is stored. In the monitoring camera information storage region104, the monitoring camera information36is stored. In the registered person information region106, the registered person information40is stored. Thus, the storage section66functions as the first storage section32and the second storage section38that are illustrated inFIG. 1.

The computer60may not be installed in the care facility and may be a server computer that is installed in a location other than the care facility and communicates with devices such as the monitoring cameras42installed in the care facility via a network. The photographing control apparatus10may be achieved by a semiconductor integrated circuit such as an application specific integrated circuit (ASIC), for example.

As an effect of the embodiment, the display controller14is described below. When a power supply of the photographing control apparatus10is turned on and multiple monitoring images are input to the photographing control apparatus10from the multiple monitoring cameras42, the display controller14causes one of the monitoring images to be displayed on the overall screen of the main display46as an example, as illustrated inFIG. 3A. The monitoring image displayed in the main display46may be input from a predetermined monitoring camera42or may be displayed in the main display46immediately before the turning-off of the power supply of the photographing control apparatus10. The display controller14causes the remaining monitoring images to be displayed as thumbnail images in a matrix in the auxiliary display48, while the remaining monitoring images are among the multiple monitoring images and are not displayed in the main display46.

The display controller14periodically acquires the direction, detected by the line-of-sight detector16, of the line of sight of the user from the line-of-sight detector16and monitors whether or not the line of sight of the user remains in a specific thumbnail image displayed in the auxiliary display48for a predetermined time period or longer. If the display controller14detects that the line of sight of the user has remained in the specific thumbnail image for the predetermined time period or longer, the display controller14switches the main monitoring image displayed in the main display46to the specific image.

As an example, if the display controller14detects that the line of sight of the user remains on a thumbnail image photographed by the monitoring camera y for the predetermined time period or longer in a state in which a monitoring image photographed by the monitoring camera x is displayed in the main display46as illustrated inFIG. 3B, the display controller14switches the main monitoring image to the image photographed by the monitoring camera y as illustrated inFIG. 3C. Specifically, the monitoring image photographed by the monitoring camera y is displayed as a main monitoring image on the overall screen of the main display46, and the monitoring image photographed by the monitoring camera x is displayed as a thumbnail image at a position at which the thumbnail image photographed by the monitoring camera y has previously been displayed on a display screen of the auxiliary display48.

Thus, if an event that prompts the user to confirm a situation occurs in the photographing range of a monitoring camera42that has photographed an image displayed as a thumbnail image in the auxiliary display48, and the user carefully watches the thumbnail image for the predetermined time period or longer, the thumbnail image is enlarged and displayed in the main display46due to the careful watching. Thus, the burden on the user who performs a task other than the care support task may be reduced.

The object information34stored in the first storage section32includes information (indicated as display target inFIG. 7) indicating whether or not each of monitoring images photographed by the monitoring cameras42is being displayed in the main display46. The display controller14updates the aforementioned information included in the object information34every time a main monitoring image displayed in the main display46is switched.

Next, a photographing control process to be executed by the photographing control apparatus10when the power supply of the photographing control apparatus10is turned on and the CPU62executes the photographing control program80is described with reference toFIGS. 4 to 6.

In step150of the photographing control process, the object extractor22clears the object information storage region102of the storage section66. In step152, the acquirer12acquires the user image from the user camera50. In step154, the line-of-sight detector16extracts eye regions of the user from the user image acquired by the acquirer12and detects the direction of the line of sight of the user based on the extracted eye regions of the user.

The line of sight of the user may be detected by estimating the positions of the centers of eyeballs of the user from the eye regions of the user, calculating the positions of the centers of pupils of the user, and detecting the directions of straight lines extending through the positions of the centers of the eyeballs and the positions of the centers of the pupils, for example. If the user camera50is a visible light camera, the inner corners of the eyes of the user may be treated as standard points, the irises of the eyes of the user may be treated as moving points, and the direction of the line of sight of the user may be detected based on positional relationships between the standard points (inner corners) and the moving points (irises), for example. In addition, if the user camera50is an infrared light camera, corneal infrared light reflection points of the eyes of the user may be treated as standard points, the pupils of the user may be treated as moving points, and the direction of the line of sight of the user may be detected based on positional relationships between the standard points (corneal reflection points) and the moving points (pupils), for example.

In step156, the photographing controller20sets a variable i for identifying a monitoring camera42to 1. In step158, the acquirer12acquires a monitoring image (hereinafter referred to as monitoring image i) from an i-th monitoring camera42. In step160, the object extractor22extracts regions of all objects (persons) included in the monitoring image i acquired by the acquirer12.

The object extractor22may use any of known various methods to extract the objects (persons). As an example, a difference between the monitoring image i and a background image photographed by the i-th monitoring camera42in advance is calculated and a differential region indicating the difference is extracted. The photographing ranges of the monitoring cameras42are variable in the embodiment. However, while the photographing ranges may be different from each other, multiple background images may be acquired and the difference may be calculated using a background image that is most approximate to the photographing range of the current i-th monitoring camera42.

Next, the extracted differential region is binarized, noise is removed, and labeling is executed to separate the differential region into multiple regions that may correspond to the objects (persons). Then, characteristic amounts such as the sizes and shapes of the separated regions are calculated for the separated regions, the accuracy of the regions corresponding to the objects (persons) is checked based on the calculated characteristic amounts, and a region corresponding to an item other than the objects (persons) is removed. Thus, the regions of all the objects (persons) included in the monitoring image i are extracted. It goes without saying that the objects (persons) may be extracted by a method other than the aforementioned method. After the regions of the objects are extracted, the regions of the objects (persons) may be traced using other characteristic amounts such as positions, colors (color histogram), or motion vectors on the monitoring image i or the like.

In step162, the object extractor22registers or updates the object information34stored in the first storage section, based on the results of extracting the regions of the objects from the monitoring image i of the i-th monitoring camera42. As illustrated inFIG. 7, the object information34includes attribute information of objects included in the monitoring images for the monitoring cameras42. The attribute information of the objects includes object IDs, characteristic amount information, positional coordinates, image ranges, continuous photographing target flags, weight level flags, registered person flags, start times of continuous photographing, cumulative time periods for which the line of sight of the user remains on the objects, and line-of-sight remain flags.

The object extractor22registers or updates the object IDs, the characteristic amount information, the positional coordinates, and the image ranges among the aforementioned attribute information. The object IDs identify the objects. Hereinafter, a j-th object extracted from the monitoring image i is referred to as object Sij. The characteristic amount information indicates characteristics of the objects and includes at least any group of sizes (heights or the like), shapes, colors (for example, color histograms), and motion vectors or includes a combination of the sizes, the shapes, the colors, and the motion vectors. The positional coordinates are the positions of the objects on the monitoring image i or are, for example, the centers of gravity of the objects. The image ranges are information indicating ranges occupying the objects on the monitoring image i. If the characteristic amount information includes the sizes, the ranges of the objects may be omitted.

If information of the objects extracted from the monitoring image i is not registered, the object extractor22newly registers, for the objects, the object IDs, the characteristic amount information, the positional coordinates, and the image ranges in the object information34. If the information of the objects extracted from the monitoring image i is already registered, the object extractor22compares the characteristic amount information, the positional coordinates, and the image ranges with registered information for the objects and determines whether or not the information indicates registered objects. If the object extractor22determines that information of an object is already registered, the object extractor22updates characteristic amount information, positional coordinates, and image range of the object that have been registered in the object information34. If the object extractor22determines that the information of the object is not registered, the object extractor22newly registers the information and an object ID of the object in the object information34.

The object extractor22associates, for the objects, (changes in) characteristic amount information, positional coordinates, and image ranges, obtained in the object extraction process executed multiple times, of the objects with the object IDs and causes the characteristic amount information, the positional coordinates, the image ranges, and the object IDs to be stored in the memory64for determination by the abnormality determiner28described later.

In step164, the identifying section18sets a variable j for identifying an object to 1. In the next step166, the identifying section18reads, from the object information34, information of the j-th object Sijextracted from the monitoring image i. In step168, the identifying section18executes a continuous photographing target identification process of identifying whether or not the object Sijis to be subjected to the continuous photographing, based on the information of the object Sijthat has been read in step166.

The continuous photographing target identification process is described below with reference toFIG. 5. In step200, the registered person identifying section26of the identifying section18determines whether or not the object Sijis already set as a registered person, based on whether or not a registered person flag included in the read information of the object Sijindicates ON. The registered person flag is set to ON when the object is determined as a registered person having the characteristic stored as the registered person information40in the second storage section38. The registered person flag is initially set to OFF.

If the answer to the determination of step200is negative, the process proceeds to step202. In step202, the registered person identifying section26determines whether or not characteristic amount information of the object Sijmatches the characteristic stored as the registered person information40in the second storage section38. In the registered person information40, the characteristic of the person who is among persons using the care facility and is desirable to be carefully watched or for which the degree of care to be provided is high or the like is registered. The registered person information40may include information such as a characteristic indicating how the person who is desirable to be carefully watched walks. If the characteristic amount information of the object Sijdoes not match the characteristic of the registered person indicated in the registered person information40, the answer to the determination of step202is negative and the process proceeds to step204.

In step204, the continuous photographing target identifying section24determines whether or not the object Sijis already set as a continuous photographing target, based on whether or not a continuous photographing target flag included in the information of the object Sijindicates ON and whether or not a weight level wijof the object Sijis equal to or larger than a lower limit value. The continuous photographing target flag is set to ON when the line of sight of the user remains on an image of the object for the predetermined time period or longer. The continuous photographing target flag is initially set to OFF. If the answer to the determination of step204is negative, the process proceeds to step206.

In step206, the continuous photographing target identifying section24determines whether or not the monitoring image i is being displayed as a main monitoring image in the main display46. If the answer to the determination of step206is negative, the continuous photographing target identification process is terminated. Thus, continuous photographing target flags (described later) and weight levels w (described later) of objects included in the monitoring image i currently displayed as a thumbnail image in the auxiliary display48are not updated. In other words, the continuous photographing target flags and weight levels w of the objects included in the monitoring image i currently displayed as the thumbnail image in the auxiliary display48are maintained at the same values as those when the monitoring image i has previously been displayed as the main monitoring image in the main display46.

On the other hand, if the answer to the determination of step206is positive, the process proceeds to step208. In step208, the continuous photographing target identifying section24acquires the direction, detected by the line-of-sight detector16, of the line of sight of the user from the line-of-sight detector16. Then, the continuous photographing target identifying section24determines whether or not the line of sight of the user is located on an image of the object Sij, based on whether or not a position on the line of sight of the user is included in the image range of the object Sijon the display screen of the main display46. If the line of sight of the user is not located on the image of the object Sij, the answer to the determination of step208is negative and the process proceeds to step210. In step210, the continuous photographing target identifying section24sets, to 0, a remain flag of the object Sijand a cumulative time period T for which the line of sight of the user remains on the image of the object Sij. Then, the continuous photographing target identifying section24terminates the continuous photographing target identification process.

If the line of sight of the user is located on the image of the object Sij, the answer to the determination of step208is positive and the process proceeds to step212. In step212, the continuous photographing target identifying section24determines whether or not the remain flag of the object Sijis already set to 1. If the answer to the determination of step212is negative, the process proceeds to step214. In step214, the continuous photographing target identifying section24sets the remain flag of the object Sijto 1. Then, the continuous photographing target identifying section24terminates the continuous photographing target identification process.

If the answer to the determination of step212is positive, the process proceeds to step216. In step216, the continuous photographing target identifying section24updates the cumulative time period T by adding a predetermined time period (corresponding to a time interval in which each object is processed) to the cumulative time period T for which the line of sight of the user remains on the image of the object Sij. In the next step218, the continuous photographing target identifying section24determines whether or not the cumulative time period T updated in step216is equal to or longer than a threshold. If the answer to the determination of step218is negative, the continuous photographing target identifying section24terminates the continuous photographing target identification process. Thus, if the line of sight of the user becomes separated from the image of the object Sijbefore the cumulative time period T for which the line of sight of the user remains on the image of the object Sijreaches the threshold, the remain flag of the object Sijand the cumulative time period T for which the line of sight of the user remains on the image of the object Sijare set to 0 again.

If the line of sight of the user remains on the image of the object Sijfor a time period equal to or longer than the threshold, the answer to the determination of step218is positive and the process proceeds to step220. In step220, the continuous photographing target identifying section24sets the continuous photographing target flag of the object Sijto 1, thereby setting the objects Sijas a continuous photographing target to be continuously photographed by the i-th monitoring camera42for a time period equal to or longer than the predetermined time period. In addition, the continuous photographing target identifying section24sets the start time tijof the continuous photographing to be executed on the object Sijto the current time.

In the next step222, the continuous photographing target identifying section24sets the weight level wijof the object Sijto an initial value corresponding to the position of the object Sijon the monitoring image i. In the embodiment, as illustrated inFIG. 8A, a region on the monitoring image is divided into an inner circumferential region including the center of the monitoring image, an outer circumferential region surrounding the circumference of the inner circumferential region, and a takeover region located around a circumferential edge of the outer circumferential region. As illustrated inFIG. 8B, the weight level w of the object set as the continuous photographing target is changed so that the initial value of the weight level w of the object located in the outer circumferential region is larger than the initial value of the weight level w of the object located in the inner circumferential region and that a reduction in the weight level w over time when the object is located in the outer circumferential region is smaller than a reduction in the weight level w over time when the object is located in the inner circumferential region. Since the weight level w is changed in the aforementioned manner, the object set as the continuous photographing target is continuously photographed for a time period equal to or longer than the predetermined time period to the time when the weight level w becomes lower than the lower limit value.

Thus, in step222, the weight level wijis set to the initial value that is based on whether the object Sijis located in the outer circumferential region or the inner circumferential region. If the object Sijis located in the takeover region, the object Sijis removed from continuous photographing targets to be continuously photographed by the i-th monitoring camera42in a target takeover and abnormality determination process (described later), and the weight level wijof the object Sijis not set in step222. In the next step224, the continuous photographing target identifying section24sets, to 0, the cumulative time period T for which the line of sight of the user remains on the image of the object Sij. Then, the continuous photographing target identifying section24terminates the continuous photographing target identifying process.

If the object Sijis already set as the continuous photographing target to be continuously photographed by the i-th monitoring camera42in step204, the answer to the determination of step204is positive and the process proceeds to step226. In step226, the continuous photographing target identifying section24calculates a time elapsed from the start time tijof the continuous photographing executed on the object Sij. In the next step228, the continuous photographing target identifying section24determines whether or not the monitoring image i is being displayed as the main monitoring image in the main display46. If the answer to the determination of step228is negative, the continuous photographing target identifying section24terminates the continuous photographing target identification process.

If the answer to the determination of step228is positive, the process proceeds to step230. In step230, the continuous photographing target identifying section24acquires the direction, detected by the line-of-sight detector16, of the line of sight of the user from the line-of-sight detector16. Then, the continuous photographing target identifying section24determines whether or not the line of sight of the user is located on the image of the object Sij. If the line of sight of the user is not located on the image of the object Sij, the answer to the determination of step230is negative and the process proceeds to step210. In step210, the continuous photographing target identifying section24sets the remain flag of the object Sijto 0. Then, the continuous photographing target identifying section24terminates the continuous photographing target identification process.

If the line of sight of the user is located on the image of the object Sij, the answer to the determination of step230is positive and the process proceeds to step234. In step234, the continuous photographing target identifying section24determines whether or not the remain flag of the object Sijindicates 1. If the answer to the determination of step234is negative, the process proceeds to step236. In step236, the continuous photographing target identifying section24sets the remain flag of the object Sijto 1. Then, the continuous photographing target identifying section24terminates the continuous photographing target identification process.

If the answer to the determination of step234is positive, the process proceeds to step238. In step238, the continuous photographing target identifying section24updates the cumulative time period T by adding the predetermined time period (corresponding to the time interval in which each object is processed) to the cumulative time period T for which the line of sight of the user remains on the image of the object Sij. In the next step240, the continuous photographing target identifying section24determines whether or not the cumulative time period T updated in step238is equal to or longer than the threshold.

If the answer to the determination of step240is negative, the process proceeds to step242. In this case, it may be determined that the object Sij is not carefully watched by the user. Thus, in step242, the continuous photographing target identifying section24subtracts, from the weight level wijof the object Sij, a predetermined value based on the position of the object Sijon the monitoring image i and a time elapsed from the start time tijof the continuous photographing executed on the object Sij. Specifically, a value that is obtained by multiplying the elapsed time by an inclination (refer toFIG. 8B) based on whether the object Sij is located in the outer circumferential region or the inner circumferential region is calculated as the predetermined value, and the calculated predetermined value is subtracted from the weight level wijof the object Sij. Then, the continuous photographing target identification process is terminated.

If the answer to the determination of step240is positive, the process proceeds to step244. In this case, it may be determined that the degree at which the user carefully watches the object Sijis high. Thus, in step244, the continuous photographing target identifying section24adds a predetermined value to the weight level wijof the object Sij. Then, the continuous photographing target identifying section24terminates the continuous photographing target identification process. The predetermined value added in step244may be a fixed value or may be changed based on whether the object Sijis located in the outer circumferential region or the inner circumferential region.

If the characteristic amount information of the object Sijmatches the characteristic stored as the registered person information40in the second storage section38in step202, the answer to the determination of step202is positive and the process proceeds to step246. In step246, the registered person identifying section26sets the registered person flag of the object Sijand the continuous photographing target flag of the object Sijto 1, thereby setting the object Sijas a registered person and a continuous photographing target. In step248, the registered person identifying section26sets the weight level wijof the object Sijto a predetermined value. Then, the registered person identifying section26terminates the continuous photographing target identification process.

The answer to the determination of step200executed on the object Sijset as the registered person is positive every time the determination of step200is made. If the answer to the determination of step200is positive, the continuous photographing target identification process is terminated and the setting state of the object Sijset as the registered person and the continuous photographing target is maintained.

When the aforementioned continuous photographing target identification process is terminated, the photographing control process (illustrated inFIG. 4) proceeds to step170. In step170, the identifying section18executes the target takeover and abnormality determination process. The target takeover and abnormality determination process is described with reference toFIG. 6.

In step260of the target takeover and abnormality determination process, the identifying section18determines whether or not the continuous photographing target flag of the object Sijindicates 1 and the identifying section18determines whether or not the object Sijis a continuous photographing target. If the answer to the determination of step260is negative, the process proceeds to step270. If the object Sijis the continuous photographing target, the answer to the determination of step260is positive and the process proceeds to step262. In step262, the identifying section18determines whether or not the object Sijis located in the takeover region of the monitoring image i. If the answer to the determination of step262is negative, the process proceeds to step270. If the answer to the determination of step262is positive, the process proceeds to step264.

If the object Sijis the continuous photographing target and located in the takeover region of the monitoring image i, the identifying section18references the monitoring camera information36stored in the first storage section32in step264. Then, the identifying section18recognizes the i-th monitoring camera42and a monitoring camera42whose photographing range is adjacent to the photographing range of the i-th monitoring camera42on the side on which the object Sijis located. The monitoring camera recognized in step264is hereinafter referred to as “k-th monitoring camera42” for descriptive purposes.

In the next step266, the identifying section18copies information indicating the object Sijand included in object information corresponding to the i-th monitoring camera42and included in the object information34stored in the first storage section32and adds the copied information to object information corresponding to the k-th monitoring camera42. Positional coordinates Pijof the object Sijare temporarily converted into three-dimensional coordinate values set in the care site of the care facility and are reconverted into positional coordinates Pkxon a monitoring image photographed by the k-th monitoring camera42, and the positional coordinates Pkxare set. Thus, the object Sijis additionally set as a continuous photographing target to be continuously photographed by the k-th monitoring camera42, and the continuous photographing executed on the object Sijis taken over by the k-th monitoring camera42from the i-th monitoring camera42. In the next step268, the identifying section18sets, to 0, the continuous photographing target flag, included in the object information corresponding to the i-th monitoring camera42, of the object Sij, thereby removing the object Sijfrom continuous photographing targets to be continuously photographed by the i-th monitoring camera42.

In step270, the abnormality determiner28determines whether or not the object Sijhas made an abnormal motion, based on changes in the characteristic amount information, positional coordinates, and image range of the object Sijthat have been stored in the memory64in the object extraction process executed by the object extractor22multiple times. A typical example of the abnormal motion is a fall. The abnormal motion such as a fall may be determined based on a rapid change in the shape of the region of the object Sijor the like. If the answer to the determination of step270is negative, the target takeover and abnormality determination process is terminated.

If the object Sijhas made an abnormal motion or fallen or the like, the answer to the determination of step270is positive and the process proceeds to step272. In step272, the notification signal output section30causes the notifier52to operate and output a notification signal for notifying the user of the abnormal motion. Thus, the abnormal motion of the object Sijthat is the fall or the like is notified to the user. In addition, if the abnormal motion of the objects Sijis detected in a monitoring image displayed as a thumbnail image in the auxiliary display48, the abnormal motion is notified to the user.

In step274, the abnormality determiner28sets, to 1, the continuous photographing target flag, included in the object information corresponding to the i-th monitoring camera42, of the object Sij, thereby adding the object Sijto the continuous photographing targets to be continuously photographed by the i-th monitoring camera42. In step276, the abnormality determiner28sets, to a predetermined value, the weight level wijof the object Sijthat is included in the object information corresponding to the i-th monitoring camera42. Then, the abnormality determiner28terminates the target takeover and abnormality determination process.

The process executed on the object Sijwhen the object Sijhas made an abnormal motion or fallen or the like is not limited to the aforementioned process. The registered person flag may be set to 1 in advance and the object Sijmay be set as the continuous photographing target for a longer time period.

When the aforementioned target takeover and abnormality determination process is terminated, the identifying section18determines whether or not all the objects included in the monitoring image i have been processed in step172of the photographing control process (illustrated inFIG. 4). If the answer to the determination of step172is negative, the process proceeds to step174. In step174, the identifying section18increments the variable j by 1. When the process of step174is executed, the process returns to step166. Then, steps166to174are repeated until the answer to the determination of step172becomes positive. Thus, the continuous photographing target identification process and the target takeover and abnormality determination process are executed on all the objects included in the monitoring image i.

When the execution of the continuous photographing target identification process and the target takeover and abnormality determination process on all the objects included in the monitoring image i is completed, the answer to the determination of step172is positive and the process proceeds to step176. In step176, the photographing controller20references the object information corresponding to the i-th monitoring camera42and included in the object information34and determines whether or not an object set as a continuous photographing target to be continuously photographed by the i-th monitoring camera42exists. If information of one or more objects of which continuous photographing target flags have been set to 1 does not exist in the object information corresponding to the i-th monitoring camera42, the answer to the determination of step176is negative and the process proceeds to step186.

If the information of the one or more objects of which the continuous photographing target flags have been set to 1 exists in the object information corresponding to the i-th monitoring camera42, the answer to the determination of step176is positive and the process proceeds to step178. In step178, the photographing controller20reads, from the object information corresponding to the i-th monitoring camera42, weight levels w of the one or more objects of which the continuous photographing target flags have been set to 1.

In step180, the photographing controller20calculates a weighted average position of the objects set as the continuous photographing targets to be continuously photographed by the i-th monitoring camera42. For example, if the number of all the objects set as the continuous photographing targets to be continuously photographed by the i-th monitoring camera42is n, a weight level of an m-th continuous photographing target is wm, and positional coordinates of the m-th object are (xm, ym), a weight ratio Wmof the m-th continuous photographing target is calculated according to the following Equation (1). The weighted average position (X, Y) of the objects set as the continuous photographing targets is calculated according to the following Equations (2).

In step182, the photographing controller20determines whether or not the weighted average position (X, Y), calculated in step180, of the objects set as the continuous photographing targets is in the photographing range of the i-th monitoring camera42. If the answer to the determination of step182is positive, the process proceeds to step184.

In step184, the photographing controller20outputs a control signal for controlling the photographing range of the i-th monitoring camera42to a photographing range changer44corresponding to the i-th monitoring camera42so that the center of the photographing range of the i-th monitoring camera42matches the calculated weighted average position (X, Y). Specifically, the control signal causes the center of the photographing range of the i-th monitoring camera42to match the calculated weighted average position (X, Y) of the objects set as the continuous photographing targets to be continuously photographed by the i-th monitoring camera42. If the answer to the determination of step182is negative, step184is skipped and the process proceeds to step186.

In step186, the photographing controller20determines whether or not the aforementioned process has been executed on all the monitoring cameras42. If the answer to the determination of step186is negative, the process proceeds to step188. In step188, the photographing controller20increments the variable i by 1 and the process returns to step158. Then, steps158to188are repeated until the answer to the determination of step186becomes positive and the aforementioned process is executed on all the monitoring cameras42. If the answer to the determination of step186is positive, the process returns to step152and all the processes are repeated in order from the detection of the line of sight of the user.

Effects of the aforementioned photographing control process are described with reference to theFIGS. 9A to 14B. As described above, an object that is among objects included in a main monitoring image displayed in the main display46and toward which the user directs his or her line of sight for a time period equal to or longer than the threshold is set as a continuous photographing target.

As illustrated inFIG. 9A, when a user120becomes aware of an object (person to be cared)122B that is included in a main monitoring image and is not set as a continuous photographing target, the user120directs his or her line of sight toward the concerned object122B as a natural motion. Then, when the user120continues to direct his or her line of sight toward the concerned object122B for a time period equal to or longer than the threshold, the object122B toward which the user120directs his or her line of sight is additionally set as a continuous photographing target, as illustrated inFIG. 9B. Due to the setting, the center of the photographing range of a monitoring camera42photographing the main monitoring image is moved to a weighted average position of an object122A already set as a continuous photographing target and the object122B additionally set as the continuous photographing target. Thus, the range of the main monitoring image is controlled to a range in which it is appropriate to watch the objects122A and122B set as the continuous photographing targets without any operation by the user120.

In addition, as illustrated inFIG. 9B, when the user120becomes aware of an object (person to be cared)122C that is included in the main monitoring image and is not set as a continuous photographing target, the user120directs his or her line of sight toward the concerned object122C as a natural motion. Then, if the user120continues to direct his or her line of sight toward the concerned object122C for a time period equal to or longer than the threshold, the object122C toward which the user120directs his or her line of sight is additionally set as a continuous photographing target, as illustrated inFIG. 9C. Due to the setting, the center of the photographing range of the monitoring camera42photographing the main monitoring image is moved to a weighted average position of the objects122A,122B, and122C set as the continuous photographing targets. Thus, the range of the main monitoring image is controlled to a range in which it is appropriate to watch the objects122A,122B, and122C set as the continuous photographing targets without any operation by the user120.

Next, as illustrated inFIG. 10A, the following state is considered: the two objects122A and122B are already set as continuous photographing targets, and the center of the photographing range of the monitoring camera42photographing the main monitoring image is located at a weighted average position of the objects122A and122B set as the continuous photographing targets. It is assumed that a time of 30 seconds elapses after the object122A is set as the continuous photographing target due to the careful watching of the object122A by the user120. It is assumed that a time of 100 seconds elapses after the object122B is set as the continuous photographing target due to the careful watching of the object122B by the user120. In this state, when the user120becomes aware of the object122C and continues to direct his or her line of sight toward the object122C for a time period equal to or longer than the threshold, the object122C is additionally set as a continuous photographing target.

Time elapses after the user120starts to carefully watch the objects122A and122B, but it is considered that the user120is carefully watching the object122C and most aware of the object122C at the current time. In the embodiment, weight levels w of the objects are reduced over time after the objects are set as the continuous photographing targets (refer to step242illustrated inFIG. 5). A relative weight level w (weight ratio W) of the object122C is larger than those of the objects122A and122B since there are differences between time that elapses after the objects are set as the continuous photographing targets. Thus, as illustrated inFIG. 10B, the center of the photographing range of the monitoring camera photographing the main monitoring image is moved toward the object122C and controlled to a range in which it is appropriate to watch the object122C without any operation by the user120.

When a long time elapses after a certain object is set as a continuous photographing target, and the line of sight of the user120remains on an image of the certain object for a cumulative time period equal to or longer than the threshold, the weight level w of the certain object is increased (refer to steps240and244illustrated inFIG. 5). Thus, when the user120is aware of an object (or directs his or her line of sight toward the object) even after the setting of the object as a continuous photographing target, the object is maintained as the continuous photographing target and is continuously photographed by the monitoring camera42photographing the main monitoring image.

Subsequently, as illustrated inFIG. 11A, the following state is considered: four objects122A to122D are already set as continuous photographing targets and the center of the photographing range of the monitoring camera42photographing the main monitoring image is located at a weighted average position of the objects122A to122D set as the continuous photographing targets. In this state, if the object (person to be cared)122C moves from the inner circumferential region of the main monitoring image to the outer circumferential region of the main monitoring image, a probability at which the object122C departs from the range of the main monitoring image increases.

In the embodiment, however, a weight level w of an object set as a continuous photographing target is changed so that a reduction over time in the weight level w of the object located in the outer circumferential region is smaller than a reduction over time in the weight level w of the object located in the inner circumferential region. Thus, a reduction over time in the weight level w of the object122C located in the outer circumferential region of the main monitoring image is smaller than reductions over time in the weight levels w of the objects122A,122B, and122D located in the inner circumferential region of the main monitoring image, and the relative weight level w (weight ratio W) of the object122C becomes larger than those of the objects122A,122B, and122D. Thus, as illustrated inFIG. 11B, the center of the photographing range of the monitoring camera42photographing the main monitoring image is moved toward the object122C, and this movement of the center of the photographing range may inhibit the object122C from departing from the range of the main monitoring image without any operation by the user120.

Next, as illustrated inFIG. 12A, it is assumed that the four objects122A to122D are set as continuous photographing targets and that an object122E (person to be cared) that is a registered person came into the photographing range of the monitoring camera42photographing the main monitoring image. In the embodiment, when an object whose characteristic amount information matches the characteristic of the registered person indicated in the registered person information40is detected, the object is set as a registered person and a continuous photographing target. Thus, as illustrated inFIG. 12B, the object122E that came into the photographing range is additionally set as a continuous photographing target, and the center of the photographing range of the monitoring camera42photographing the main monitoring image is moved to a position included in the photographing range including the object122E.

Thus, since a characteristic of a person who is to be cared and desirable to be carefully watched or for which the degree of care to be provided is high is registered in the registered person information40in advance, the person to be cared is automatically additionally set as a continuous photographing target without any operation by the user120. In the embodiment, whether or not objects included in monitoring images displayed as thumbnail images in the auxiliary display48are registered persons is determined, and a weight level w of an object set as a registered person is not changed. Thus, the registered person is maintained as a continuous photographing target and is continuously photographed by one or more monitoring cameras42among the multiple monitoring cameras42.

Next, as illustrated inFIG. 13A, it is assumed that the object122E moves to the outside of the photographing range of a monitoring camera42photographing a main monitoring image in a state in which the five objects122A to122E are already set as continuous photographing targets. In the embodiment, when an object set as a continuous photographing target is located in the takeover region of the monitoring image i, the continuous photographing executed on the object located in the takeover region is taken over by the k-th monitoring camera42whose photographing range is adjacent to the photographing range of the i-th monitoring camera42from the i-th monitoring camera42that has acquired the monitoring image i.

Thus, when the object122E moves to the takeover region of the monitoring image illustrated inFIG. 13A, the continuous photographing executed on the object122E is taken over by a monitoring camera42(whose photographing range includes a place to which the object122E has moved) photographing a monitoring image illustrated inFIG. 13B. Thus, even when an object set as a continuous photographing target moves to the outside of the photographing range of a certain monitoring camera42, the continuous photographing executed on the object is taken over by another monitoring camera42whose photographing range is adjacent to the photographing range of the certain monitoring camera42without any operation by the user120. Then, the center of the photographing range of the other monitoring camera42that has taken over the continuous photographing is moved to a position included in the photographing range including the object122E that has moved.

Subsequently, as illustrated inFIG. 14A, it is assumed that while the three objects122A to122C are already set as continuous photographing targets, the object122D that is not set as a continuous photographing target has fallen. In the embodiment, when an abnormal motion such as a fall is detected, the abnormal motion is notified to the user regardless of whether or not an object that has made the abnormal motion is a continuous photographing target, and the object of which the abnormal motion is detected is additionally set as a continuous photographing target.

Thus, as illustrated inFIG. 14B, when the abnormal motion (fall) of the object122D is detected, the abnormal motion is notified to the user120, the object122D is additionally set as a continuous photographing target, and the center of the photographing range of the monitoring camera42is moved toward the object122D. Thus, the user120may immediately confirm the state of the person who is to be cared and of which the abnormal motion has been detected, and the user120may notify a caregiver of the state of the person and immediately go to the care site or take appropriate measures. If a monitoring image that includes the object122D is displayed as a thumbnail image in the auxiliary display48, the user carefully watches the thumbnail image, thereby causing the monitoring image including the object122D to be displayed in the main display46.

According to the embodiment, the acquirer12acquires a monitoring image photographed by a monitoring camera42and a user image photographed by the user camera50and indicating the user who has visibility of the monitoring image displayed in the main display46. The line-of-sight detector16detects the line of sight of the user based on the user image acquired by the acquirer12. The identifying section18identifies an object included in the monitoring image on the display screen of the main display46and displayed at the position of the line, detected by the line-of-sight detector16, of sight of the user on the display screen of the main display46. The imaging control section20outputs a control signal for controlling the photographing direction of the monitoring camera42so that the object identified by the identifying section18is continuously photographed as the monitoring image for the predetermined time period or longer. Thus, the technique for controlling the cameras configured to photograph monitoring images in which objects to be monitored are included in photographing regions may be provided. In addition, the burden on the user who visually checks the monitoring images may be reduced, and the user may watch, on a monitoring image, a person who is to be cared and to which a caregiver temporarily pays insufficient attention during a task for which the user is responsible.

According to the embodiment, the identifying section18identifies an object displayed in a region in which the line of sight of the user remains for a predetermined time period or longer on the display screen of the main display46. Thus, an image of an object that crosses the line of sight of the user during the time when the user changes his or her line of sight may be removed from continuous photographing targets, and only an object included in an image carefully watched by the user may be identified as a continuous photographing target.

According to the embodiment, if the identifying section18identifies multiple objects, the photographing controller20outputs a control signal so that all the multiple objects are simultaneously photographed as the monitoring image. Thus, the user who visually checks the single monitoring image and may simultaneously watch the multiple objects to be subjected to the continuous photographing.

According to the embodiment, the photographing controller20outputs a control signal for controlling the photographing direction of the monitoring camera42so that the center of the photographing range of the monitoring camera42is located at a position separated by a predetermined distance or less from a standard position of a range including all the multiple objects. Thus, the photographing range of the monitoring camera42is controlled to a range in which it is appropriate to simultaneously watch the multiple objects to be subjected to the continuous photographing.

According to the embodiment, the identifying section18gives weight levels to the multiple objects so that the weight levels are reduced as time elapses after the setting of the objects as continuous photographing targets. Then, the photographing controller20sets a weighted average position of the multiple objects to the standard position of the range including all the objects. Thus, the center of the photographing range of the monitoring camera42may become close to an object image carefully watched by the user for the shortest elapsed time, and the photographing range of the monitoring camera42may be controlled to a photographing range intended by the user.

According to the embodiment, the identifying section18measures, for the objects set as the continuous photographing targets, cumulative time periods for which the line of sight of the user remains on the objects, and the identifying section18gives weight levels to the objects so that the weight levels are increased as the cumulative time periods for which the line of sight of the user remains on the objects become longer. Then, the photographing controller20sets a weighted average position of the multiple objects to the standard position of the range including all the multiple objects. Thus, the center of the photographing range of the monitoring camera42may become close to an object on which the line of sight of the user remains for the longest cumulative time period, and the photographing range of the monitoring camera42may be controlled to a photographing range intended by the user.

According to the embodiment, if multiple monitoring cameras42whose photographing ranges are different from each other exist, the identifying section18causes, for the multiple monitoring cameras42, information of the identified objects to be stored in the first storage section32as information of continuous photographing targets. In addition, the identifying section18removes, from information corresponding to a first monitoring camera42and indicating continuous photographing targets, information of a displayed object whose position has moved into a predetermined region included in a monitoring image photographed by the first monitoring camera42. Furthermore, the identifying section18adds the removed information to information indicating continuous photographing targets and corresponding to a second monitoring camera42whose photographing range is adjacent to the photographing range of the first monitoring camera42. Then, the photographing controller20outputs a control signal for controlling photographing directions of the multiple monitoring cameras42, based on the information corresponding to the multiple monitoring cameras42and indicating the continuous photographing targets. Thus, even when an object set as a continuous photographing target moves to the outside of the photographing range of the first monitoring camera42, the continuous photographing of the object is taken over by the second monitoring camera42whose photographing range is adjacent to the photographing range of the first monitoring camera42, and the takeover may inhibit the photographing of the object set as the continuous photographing target from being stopped.

According to the embodiment, the object extractor22extracts one or more objects included in the monitoring image and the abnormality determiner28determines whether or not each of the objects extracted by the object extractor28has made an abnormal motion. If the abnormality determiner28determines that an object has made an abnormal motion, the notification signal output section30outputs a notification signal for notifying the user of the abnormality motion. Thus, the user may immediately confirm, on the monitoring image, the object determined to have made the abnormal motion and the notification signal may promote the user to take measures.

According to the embodiment, the object extractor22extracts one or more objects included in the monitoring image. The registered person identifying section26determines whether or not each of the objects extracted by the object extractor22has a specific object characteristic stored in the second storage section38, and the registered person identifying section26identifies an object determined to have the specific object characteristic. Thus, the object (registered person) that has the specific object characteristic may be automatically set as a continuous photographing target.

The case where the photographing range changer44changes the photographing direction of a monitoring camera42, thereby changing the photographing range of the monitoring camera42is described above. The embodiment, however, is not limited to this. The photographing range changer44may change the magnification of a monitoring camera42, thereby changing the photographing range of the monitoring camera42. Alternatively, the photographing range changer44may change the magnification and photographing direction of the monitoring camera42, thereby changing the photographing range of the monitoring camera42.

In addition, the case where the weighted average of the current positions of the multiple objects to be subjected to the continuous photographing is used as the standard position of the range including all the multiple objects is described above. The standard position, however, is not limited to the weighted average of the current positions of the multiple objects. For example, the standard position may be a weighted average of moving average positions of the multiple objects or a simple average of the current positions of the multiple objects.

In addition, the case where the monitoring camera42is controlled so that the center of the photographing range of the monitoring camera42matches the standard position of the range including all the multiple objects is described above. The embodiment, however, is not limited to this. For example, a future change in the standard position may be estimated based on a motion vector of the standard position of the range including all the multiple objects, and the monitoring camera42may be controlled so that the center of the photographing range of the monitoring camera42is moved in advance of the future change in the standard position.

In addition, the case where the photographing control program80that is an example of the photographing control program according to the technique disclosed herein is stored (installed) in the storage section66in advance is described above. The photographing control program, however, may be stored in a recording medium such as a CD-ROM, a DVD-ROM, or a memory card and provided.

All cited documents, patent applications, and technical standards mentioned in the present specification are incorporated by reference in the present specification to the same extent as if the individual cited documents, patent applications, and technical standards were specifically and individually incorporated by reference in the present specification.