Image and audio reproduction device and method

An image and audio reproduction device obtains an image captured by an image capturing device and a plurality of acoustic signals whose arrival directions are respectively a plurality of locations in an image capturing scope of the image capturing device, and sets, as one group, arrival directions of acoustic signals with a high degree of similarity of amounts of characteristics on the basis of a degree of similarity of the amounts of characteristics between the acoustic signals that are from the arrival directions adjacent to each other. The image and audio reproduction device adds information to the image captured by the image capturing device so as to reproduce the image, the information representing a position of the group in an image capturing scope of the image capturing device.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2015-171263, filed on Aug. 31, 2015, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are related to a device that reproduces an image and audio.

BACKGROUND

As a technique related to an image and audio reproduction device that reproduces an image captured by an image capturing device and audio picked up by a sound pickup device, a technique that selectively reproduces audio from an attention area in an image that is being reproduced is known (see Document 1 for example).

Also, as a technique of estimating a direction of a target sound, a technique of estimating a direction of a sound source by using a spatial spectrum calculated on the basis of a plurality of correlation matrixes of acoustic signals and correlation matrixes of noise signals is known (see Document 2 for example).

Also, as a technique of estimating a direction of a sound source of a particular type, a technique of identifying the type of a sound source of an acoustic signal on the basis of the likelihood of the type of a sound source associated with an amount of acoustic characteristics so as to estimate the direction of the sound source on the basis of an acoustic signal of a sound source of the identified type is known (see Document 3 for example).

Further, a technique of performing an auditory lateralization process etc. on an audio signal by using a signal process device provided separately from a host device for reproducing an image, so that the loads on a central processing unit (CPU) are reduced is known (see Document 4 for example).

SUMMARY

According to an aspect of the embodiment, an image and audio reproduction device comprising: a memory configured to store an image captured by an image capturing device and a plurality of acoustic signals whose arrival directions are respectively a plurality of locations, at which a sound is picked up by a sound pickup device, in an image capturing scope of the image capturing device; and a processor configured to set, as one group, arrival directions of acoustic signals with a high degree of similarity of amounts of characteristics on the basis of a degree of similarity of the amounts of characteristics between acoustic signals that are from the arrival directions adjacent to each other, and to add information to the image so as to reproduce the image, the information representing a position of the group in an image capturing scope of the image capturing device.

DESCRIPTION OF EMBODIMENTS

In facilities, such as a factory, having various types of equipment including power generating equipment etc., prompt discovery of abnormality of each piece of equipment is vital. Thus, in maintenance services of factories etc., prompt detection of an abnormal sound of equipment and prompt identification of the source of the abnormal sound are vital.

Also, maintenance services of factories have increased its efficiency by using the Information and Communication Technology (ICT) such as for example one utilizing mobile terminals, e.g., a tablet computer in recent years. Regarding the increase in efficiency by the use of the Information and Communication Technology, a sound arriving from a direction corresponding to a location (attention area) selected in an image (window) during the reproduction of the image captured by an image capturing device can selectively be reproduced by utilizing the technique disclosed by Document 1 etc.

However, when sounds of a plurality of locations in a window are to be reproduced, each of such locations is to be selected so as to conduct the reproduction in a one-by-one manner. This makes the confirmation operation troublesome when the presence or absence of abnormal sounds is to be confirmed for a plurality of locations in the image capturing scope in a maintenance service.

First Embodiment

FIG. 1illustrates a configuration example of an inspection system according to a first embodiment.

As illustrated inFIG. 1, an inspection system1A according to the present embodiment includes an image capturing device2, a plurality of sensors3(301,302and310), a relay device4and an inspection terminal5.

The image capturing device2is a device that captures images of pieces of equipment6(601and602) that are inspection targets.

The plurality of sensors3are sensors that include sound pickup devices (microphones) that pick up acoustic signals. The present embodiment uses, as the sensor3, a compound sensor that includes a sound pickup device for picking up an acoustic signal and a vibration sensor for detecting vibrations. The sensor3is located in the vicinity of a location (for example a power generation unit, a movable unit, etc.) in which sound is generated in the inspection-target equipment. Hereinafter, the sensor3is also referred to as a compound sensor.

The relay device4is a device that obtains an image captured by the image capturing device2and a signal output from each compound sensor3(acoustic signal and vibration signal) so as to transfer them to the inspection terminal5. The inspection terminal5is a device that reproduces an image and an acoustic signal obtained via the relay device4. The relay device4and the inspection terminal5are connected to each other in such a manner that communications are possible in a form of a wireless communication based on for example a communications standard standardized by the Institute of Electrical and Electronic Engineers (IEEE).

FIG. 2is a block diagram illustrating a functional configuration of an inspection terminal used in the inspection system according to the first embodiment.

As illustrated inFIG. 2, the inspection terminal5according to the present embodiment includes an image and audio reproduction device7A, a display device8, a speaker9and an input device10.

The image and audio reproduction device7A includes a communication unit701, a correlation value calculation unit702, a group setting unit703, an acoustic-characteristic amount calculation unit704, a vibration-characteristic amount calculation unit705, a state decision unit706, a reproduction unit707, a storage unit708and an input reception unit709.

The communication unit701conducts a wireless communication with a communication unit401of the relay device4so as to obtain an image captured by the image capturing device2, signals output from the compound sensors301,302and310(acoustic signal and vibration signal), and sensor position information. Sensor position information is stored in for example a storage unit402of the relay device4. The communication unit701outputs an obtained image to the reproduction unit707. Also, the communication unit701outputs an obtained acoustic signal to the correlation value calculation unit702and the acoustic-characteristic amount calculation unit704. The communication unit701also outputs an obtained vibration signal to the vibration-characteristic amount calculation unit705. Further, the communication unit701outputs sensor position information to the correlation value calculation unit702and stores it in the storage unit708.

On the basis of the input acoustic signals and sensor position information, the correlation value calculation unit702calculates a correlation value between acoustic signals obtained from sound pickup devices (compound sensors3) that are adjacent to each other. The group setting unit703compares the calculated correlation value between acoustic signals with a prescribed correlation threshold so as to determine the presence or absence of a correlation between acoustic signals. Also, the group setting unit703performs grouping in which a pair of arrival directions of acoustic signals with a correlation between them is set as one group, on the basis of the determination results about the presence or absence of correlations between acoustic signals. The group setting unit703outputs information on a set group to the reproduction unit707.

The acoustic-characteristic amount calculation unit704analyzes an input acoustic signal so as to calculate a prescribed amount of acoustic characteristics. The vibration-characteristic amount calculation unit705analyzes an input vibration signal so as to obtain an amount of vibration characteristics. On the basis of the calculated amount of acoustic characteristics and amount of vibration characteristics, the state decision unit706decides whether or not a signal output from each compound sensor3is a signal in a normal condition. The state decision unit706outputs the decision result to the reproduction unit707.

When the decision result of the state decision unit706indicates that the compound sensor3having output an acoustic signal or a vibration signal that is not normal exists, the reproduction unit707outputs information representing the set position of that compound sensor3in the image (in the window of the display device) to the display device8together with the image. Also, when there exists the compound sensor3that output an acoustic signal or a vibration signal that is not normal, the reproduction unit707outputs to the compound sensor3the image and information representing a group including the compound sensor3having output a signal that is not normal from among the groups set by the group setting unit703. Note that when the acoustic signals and vibration signals of all the compound sensors3are normal, the reproduction unit707outputs only an input image to the display device8.

Further, the reproduction unit707selects an acoustic signal arriving from a direction corresponding to a point in an image specified by the operator so as to output the signal to the speaker9. The operator manipulates the input device10so as to specify a point in an image. In response to the manipulation of the input device10by the operator, an input signal in accordance with the manipulation content is input to the input reception unit709of the image and audio reproduction device7A from the input device10. When the input signal is a signal related to the reproduction of an acoustic signal, the input reception unit709outputs the input signal to the reproduction unit707.

FIG. 3illustrates an example of an image displayed on a display device of the inspection terminal.

The display device8of the inspection terminal5displays an image captured by the image capturing device2. In the case of the inspection system1A illustrated inFIG. 1, the display device8displays an image of an area in which two pieces of equipment601and602are arranged as illustrated inFIG. 3. To the equipment601and602displayed on the display device8, the compound sensors3having sound pickup devices are attached. One compound sensor3may be connected to one piece of equipment as in the case of the equipment602or a plurality of compound sensors3may be connected as in the case of the equipment601. The equipment601includes a first portion601A, a second portion601B and a third portion601C. In the equipment601, a workpiece that received a prescribed process in the first portion601A for example is conveyed to the second portion601B via the third portion601C. Three compound sensors (sound pickup devices301A through303A) are attached to the first portion601A of the equipment601. Also, one compound sensor (sound pickup device304A or305A) is connected to each of the second portion601B and the third portion601C. The setting positions of these sensors are registered in the sensor position information.

FIG. 4illustrates an example of sensor position information.

As illustrated inFIG. 4, sensor position information410according to the present embodiment includes for example a two-dimensional coordinate value representing the position of a compound sensor in an image (in a window of a display device) and an identifier (sensor ID) for identifying a piece of equipment in which a compound sensor is set. A coordinate value representing the position of a compound sensor is calculated from for example a positional relationship between an image capturing device for capturing an image and a piece of equipment, the direction of the optical axis of an image capturing device, a view angle (image-capturing scope) and the position of a compound sensor in the equipment.

FIG. 5is a flowchart explaining contents of a reproduction process performed by an image and audio reproduction device according to the first embodiment.

In the inspection system1A according to the present embodiment, when for example the operator manipulates the input device10of the inspection terminal5so that the inspection terminal5and the relay device4are connected in such a manner that communications are possible, the image and audio reproduction device7A of the inspection terminal5performs a reproduction process as illustrated inFIG. 5.

First, the image and audio reproduction device7A starts to obtain an image signal, an acoustic signal and a vibration signal and also obtains sensor position information (step S100). The process in step S100is performed by the communication unit701.

Next, the image and audio reproduction device7A performs a group detection process (step S101) and a state decision process (step S102). The group detection process in step S101is performed by the correlation value calculation unit702and the group setting unit703. The correlation value calculation unit702calculates a correlation value between acoustic signals of adjacent compound sensors (sound pickup devices) on the basis of the acoustic signals and sensor position information from the respective compound sensors. The group setting unit703determines whether or not the correlation value between the acoustic signals calculated by the correlation value calculation unit702is higher than a prescribed correlation value, and conducts grouping in which compound sensors with a high correlation (degrees of similarity) of the acoustic signals are set as one group.

The state decision process in step S102is performed by the acoustic-characteristic amount calculation unit704, the vibration-characteristic amount calculation unit705and the state decision unit706. The acoustic-characteristic amount calculation unit704calculates the amounts of characteristics in acoustic signals obtained from the respective compound sensors. The vibration-characteristic amount calculation unit705calculates the amounts of characteristics in vibration signals obtained from the respective compound sensors. For each compound sensor, the state decision unit706decides whether or not an acoustic signal is normal on the basis of the amounts of characteristics of the acoustic signal and the vibration signal. Note that in the present embodiment, the state decision unit706uses a value of a state flag for representing the decision result for each compound sensor. When the acoustic signal is normal, the state decision unit706sets zero as the value of the state flag. When the acoustic signal is not normal (when there is abnormality), the state decision unit706sets “1” as the value of the state flag.

Completing the group detection process and the state decision process, the image and audio reproduction device7A determines whether or not there is a group including an acoustic signal whose state flag is 1 (step S103). The determination in step S103is performed by the reproduction unit707. When there is a group including an acoustic signal with a state flag of 1, (Yes in step S103), the reproduction unit707adds, to the image (window), information representing a group that includes an acoustic signal with a state flag of 1 (step S104). Thereafter, the reproduction unit707checks whether or not a signal of an instruction to reproduce an acoustic signal has been input (step S105). When there is no group including an acoustic signal with a state flag of 1 (No in step S103), the reproduction unit707skips the process in step S104, and performs the check in step S105.

When it is determined in step S105that a signal of an instruction to reproduce an acoustic signal has not been input (No in step S105), the reproduction unit707reproduces only the image signal and outputs it to the display device8(step S106). When it is determined in step S105that a signal of an instruction to reproduce an acoustic signal has been input (Yes in step S105), the reproduction unit707performs an image and audio output process (step S107). In step S107, the reproduction unit707selects an acoustic signal in accordance with the input signal so as to output the image signal to the display device8and output the selected acoustic signal to the speaker9.

When the reproduction unit707has output an image signal or has output both an image signal and an acoustic signal, the image and audio reproduction device7A determines whether or not to continue the processes in steps S101through S107(step S108). When the processes are to be continued (Yes in step S108), the image and audio reproduction device7A performs the processes in step S101through S187for acoustic signals in the next process unit period. When the processes are to be terminated (No in step S108), the image and audio reproduction device7A terminates the process of obtaining an image signal, an acoustic signal and a vibration signal, and also terminates the reproduction process, although this is omitted in the flowchart inFIG. 5.

The group detection process (step S101) in the flowchart illustrated inFIG. 5is performed by the correlation value calculation unit702and the group setting unit703as described above. The correlation value calculation unit702and the group setting unit703perform the process illustrated inFIG. 6as a group detection process.

FIG. 6illustrates a flowchart explaining contents of a group detection process performed in the reproduction process according to the first embodiment.

In a group detection process, the correlation value calculation unit702first identifies the position of a sound pickup device in the window of an image (step S101A). The correlation value calculation unit702identifies the position of a sound pickup device in a compound sensor on the basis of the sensor position information410.

The correlation value calculation unit702next calculates a correlation value between acoustic signals of adjacent sound pickup devices (step S101B). On the basis of the sensor position information410, the correlation value calculation unit702sequentially calculates correlation values between acoustic signals of pairs of adjacent sound pickup devices in the same equipment. A correlation value between acoustic signals is obtained by for example calculating a power of an acoustic signal picked up by each sound pickup device and calculating a correlation value in a time domain between powers for a pair of adjacent sound pickup devices. A correlation value in a time domain is calculated by any one of the known functions that are used for calculating a degree of similarity or a mutual correlation coefficient for acoustic signals in two time domains. After calculating correlation values for all pairs of sound pickup devices, the correlation value calculation unit702outputs to the group setting unit703the calculated correlation values and the pairs of sound pickup devices in an associated manner.

The group setting unit703compares each of the input correlation values with a prescribed correlation threshold so as to determine whether or not there is a pair of sound pickup devices with a high correlation between acoustic signals (step S101C). When there is a pair of sound pickup devices with a high correlation between acoustic signals (Yes in step S101C), the group setting unit703conducts grouping in which a pair of sound pickup devices with a high correlation are set as one group (step S101D). When one of the sound pickup devices constituting a pair has already been included in one group in step S101D, the group setting unit703adds the other sound pickup device to that group. When there is no pair of sound pickup devices with a high correlation between acoustic signals (No in step S101C), the group setting unit703skips the process in step S101D.

When the group setting unit703terminates the processes in steps S101C and S101D and outputs the process result to the reproduction unit707, the group detection process for the acoustic signals in the current process unit period is terminated.

Also, the state decision process (step S102) in the flowchart illustrated inFIG. 5is performed by the acoustic-characteristic amount calculation unit704, the vibration-characteristic amount calculation unit705and the state decision unit706as described above. The acoustic-characteristic amount calculation unit704, the vibration-characteristic amount calculation unit705and the state decision unit706perform the process illustrated inFIG. 7as a state detection process.

FIG. 7is a flowchart explaining contents of a state decision process in the reproduction process according to the present embodiment.

In the state decision process, the acoustic-characteristic amount calculation unit704first calculates the amount of acoustic characteristics of each acoustic signal (step S102A) and the vibration-characteristic amount calculation unit705calculates the amount of vibration characteristics of each vibration signal (step S102B). The acoustic-characteristic amount calculation unit704outputs the calculated amount of acoustic characteristics to the state decision unit706. Also, the vibration-characteristic amount calculation unit705outputs the calculated amount of vibration characteristics to the state decision unit706.

In the above situation, any amount of characteristics that permits the characterization of an acoustic signal in the current process target period and that can decide whether it is a situation with equipment operating normally or a situation involving abnormality can be used as an amount of acoustic characteristics. Similarly, any amount of characteristics that permits the characterization of a vibration signal in the current process target period and that can decide whether it is a situation with equipment operating normally or a situation involving abnormality can be used as an amount of vibration characteristics. Note that the processes in step S102A and102B may be performed in parallel or the process in step S102B may be performed first.

In response to input of an amount of acoustic characteristics and an amount of vibration characteristics, the state decision unit706performs the processes in steps S102C through S102F. The state decision unit706first selects one acoustic signal in an undetermined state (i.e., an acoustic signal not having had its state determined) (step S102C).

Next, the state decision unit706determines whether or not the amount of acoustic characteristics and the amount of vibration characteristics of the selected acoustic signal are within a normal range (step S102D). In step S102D, the state decision unit706compares the amount of acoustic characteristics and the amount of vibration characteristics of the selected acoustic signal with an amount of acoustic characteristics and an amount of vibration characteristics that are prepared in advance, and determines whether or not the selected amount of acoustic characteristics and amount of vibration characteristics are within a normal range.

When the amount of acoustic characteristics and the amount of vibration characteristics of the selected acoustic signal are within a normal range (Yes in step S102D), the state decision unit706sets “0” as the state flag of the selected acoustic signal (step S102E). When the amount of acoustic characteristics or the amount of vibration characteristics of the selected acoustic signal is out of a normal range (No in step S102D), the state decision unit706sets “1” as the state flag of the selected acoustic signal (step S102F).

After setting a value for a state flag in step S102E or step S102F, the state decision unit706checks whether or not there is an acoustic signal whose state is not set (step S102G). When there is an acoustic signal whose state is not set (Yes in step S102G), the state decision unit706performs the processes in steps S102C through S102F for an acoustic signal whose state is not set. When the processes in step S102C through S102F have been performed on all acoustic signals (No in step S102G), the state decision unit706outputs a decision result to the reproduction unit707and terminates the state determination process for the acoustic signal in the current unit period.

After performing the group detection process and the state decision process, the reproduction unit707performs the processes in steps S103through S107described above in the image and audio reproduction device7A.

FIG. 8illustrates an example of a process result of the reproduction process according to the first embodiment.FIG. 8illustrates a process result of a case when the above reproduction process is performed as a process for reproducing the image illustrated inFIG. 3.

The display device8of the inspection terminal5is displaying two pieces of equipment601and602as an image captured by the image capturing device2. The equipment601, which is one of the two pieces of the equipment601and602, includes the first portion601A, the second portion601B and the third portion601C. In this equipment601, a workpiece that received a prescribed process in the first portion601A for example is conveyed to the second portion601B via the third portion601C. Three compound sensors are attached to the first portion601A of the equipment601. Also, one compound sensor (sound pickup device304A or305A) is connected to each of the second portion601B and the third portion601C. The equipment602, which is the other one of the two pieces of equipment601and602, is smaller than the equipment601, and has for example one compound sensor (sound pickup device) attached to it.

When there is no abnormality in the equipment601or602during the reproduction of an image including the two pieces of equipment601and602, the state flags become “0” for all acoustic signals in step S102. Accordingly, the image and audio reproduction device7A skips the process in step S104illustrated inFIG. 5. When there is no input of a signal of an instruction to reproduce an acoustic signal (No in step S105), the reproduction unit707outputs only an image captured by the image capturing device2to the display device8. Accordingly, when there is no abnormality in the equipment601or602and there is no input of a signal of an instruction to reproduce an acoustic signal, the display device8of the inspection terminal5displays, on its window, an image captured by the image capturing device2.

By contrast, when for example an abnormal sound is generated at point P on the conveyance path (the third portion601C) for conveying a workpiece from the first portion601A to the second portion601B in the equipment601, the sound pickup devices303A through305A located near the conveyance path pick up the abnormal sound. Accordingly, in the group detection process (step S101) first, three compound sensors (sound pickup devices303A through305A) are set as one group. Also, when the sound pickup devices303A through305A have picked up an abnormal sound, the state flags are set to “1” for the acoustic signals picked up by the sound pickup devices303A through305A in the state decision process (step S102). Accordingly, the reproduction unit707in the image and audio reproduction device7A performs the process in step S104illustrated inFIG. 5. In other words, on the basis of the sensor position information410, the reproduction unit707outputs, to the display device8, for example an image signal obtained by superimposing frame line L including the sound pickup devices303A through305A having picked up the abnormal sound on an image captured by the image capturing device2. Accordingly, when the sound pickup devices303A through305A have picked up an abnormal sound, frame line L, which represents an area in which an abnormal sound was picked up in the image capturing scope, is displayed on the display device8of the inspection terminal5as illustrated inFIG. 8. Thereby, the operator of the inspection terminal5can easily recognize that an abnormal sound has been generated on the conveyance path (third portion601C) that conveys a workpiece from the first portion601A to the second portion601B in the equipment601.

Also, because a plurality of compound sensors (sound pickup devices) with a high correlation between the acoustic signals are treated as one group, the same type of an abnormal sound is picked up at the respective locations in frame line L displayed on the display device8. This enables the operator to recognize easily what type of a sound the abnormal sound is and which of the areas in the image capturing scope the sound was detected in just by selecting one point in frame line L displayed on the display device8and reproducing the acoustic signal. When for example the inspection terminal5can be used as the input device10having a touch panel arranged on the display device8, the operator of the inspection terminal5can hear an abnormal sound in the area just by touching a portion in frame line L by using a finger or a stylus, etc. Thereby, the operator can conduct, with less efforts, a confirmation process of identifying whether or not an abnormal sound was generated in the image capturing scope and of identifying the source of the abnormal sound, making it possible to conduct maintenance services efficiently.

Note that in a reproduction process according to the present embodiment, not only when a sensor-provided location in frame line L, representing the location at which an abnormal sound was generated, is selected, but also when a sensor-provided location out of frame line L on the window or an area near that location is selected, it is possible to confirm the sound in the selected location. In other words, in an image and audio output process (step S107) in the flowchart illustrated inFIG. 5, it is possible to output a sound at an arbitrary location on the window. The image and audio output process in step S107is perform d by the reproduction unit707as described above. The reproduction unit707performs the process illustrated inFIG. 9as an image and audio output process.

FIG. 9is a flowchart explaining contents of an image and audio output process in the reproduction process according to the first embodiment.

In the image and audio output process, the reproduction unit707first identifies, from input information, the eye direction at a selected position in an image (window on the display device) (step S107A). Next, the reproduction unit707identifies a compound sensor provided at the position closest to the selected position, on the basis of the orientation of the image capturing device, the eye direction and the sensor position information, and determines an acoustic signal to be reproduced (step S107B). Thereafter, the reproduction unit707outputs to the speaker9a sound reproduced from the acoustic signal that was determined in step S107B, and outputs the image to the display device8(step S107C).

As described above, according to the present embodiment, when there is a location at which an abnormal sound was picked up (detected) in an image captured by the image capturing device2, information representing the location of the generation of the abnormal sound is added to the image and the image is displayed on the display device8of the inspection terminal5. This enables the operator of the inspection terminal5to recognize the location of the generation of an abnormal sound easily.

Also, in a reproduction process according to the present embodiment, when adjacent locations have a high correlation of acoustic signals that were picked up at a plurality of locations in the image capturing scope, such locations are grouped. When the group includes a location of the generation of an abnormal sound, information of frame line L etc. representing the area included in that group is superimposed on the image, and the image is displayed. This permits easy recognition that the same type of an abnormal sound was picked up (detected) in the area enclosed by one frame line L in an image. Further, the operator can confirm what type of an abnormal sound is generated in an area just by selecting one location in an area enclosed by one frame line in an image and reproducing an acoustic signal. Accordingly, the present embodiment enables the operator to confirm the presence or absence of an abnormal sound in the image capturing scope of the image capturing device2and identify the location of the generation of an abnormal sound with less efforts.

Note that the inspection terminal5of the present embodiment is not limited to an integrated device including the image and audio reproduction device7A, the display device8, the speaker9and the input device10. The inspection terminal5may employ a configuration in which the display device8, the ee9 and the input device10that are prepared separately from the image and audio reproduction device7A are connected to the image and audio reproduction device7A via a transmission cable.

When the inspection terminal5is used for performing an inspection operations, the connection between the relay device4and the image and audio reproduction device7A is not limited to the above wireless communications but may be through a transmission cable.

Further, in the above embodiment, the state of an acoustic signal is decided (decision of whether or not it is normal) on the basis of a amount of acoustic characteristics and an amount of vibration characteristics, however, the state of an acoustic signal may be decided on the basis of an amount of acoustic characteristics alone. Also, the state of an acoustic signal may be decided on the basis of a combination of an amount of acoustic characteristics and a physical amount (such as a temperature etc.) that varies between a normal state and an abnormal state in monitoring-target equipment.

Second Embodiment

FIG. 10illustrates a configuration example of an inspection system according to a second embodiment.

As illustrated inFIG. 10, an input system1B according to the present embodiment includes the image capturing device2, a microphone array11, the relay device4and the inspection terminal5.

The image capturing device2is a device that captures images of the equipment6(601and602) that are inspection targets.

The microphone array11is a device in which a plurality of sound pickup devices (microphones) are arrayed. The microphone array11illustrated inFIG. 10includes five sound pickup devices1101through1105that are arranged in line at prescribed intervals. As illustrated inFIG. 10, this microphone array11is located near the image capturing device2and is oriented in a direction of picking up a sound arriving from the image capturing scope (equipment6).

The relay device4is a device that obtains an image captured by the image capturing device2and an acoustic signal output from the microphone array11(sound pickup devices1101through1105) so as to transfer them to the inspection terminal5. The inspection terminal5is a device that reproduces an image and an acoustic signal obtained via the relay device4. The relay device4and the inspection terminal5are connected to each other by for example wireless communications based on a prescribed wireless communication standard.

FIG. 11is a block diagram illustrating a functional configuration of an inspection terminal used in the inspection system according to the second embodiment.

As illustrated inFIG. 11, the inspection terminal5according to the present embodiment includes an image and audio reproduction device7B, the display device8, the speaker9and the input device10.

The image and audio reproduction device7B includes a communication unit721, an arriving sound generation unit722, a conversion unit723, a correlation value calculation unit724and a group setting unit725. Also, the image and audio reproduction device7B includes a power calculation unit726, a characteristic amount calculation unit727, a state decision unit728, a reproduction unit729, a storage unit730and an input reception unit731.

The communication unit721conducts a wireless communication with the relay device4, and obtains an image captured by the image capturing device2and a plurality of acoustic signals output from the microphone array11. The communication unit721outputs an obtained image to the reproduction unit729. Also, the communication unit721outputs an obtained acoustic signal to the arriving sound generation unit722.

On the basis of a plurality of acoustic signals obtained from the microphone array11, the arriving sound generation unit722generates acoustic signals that arrive at the microphone array11(image capturing device2) respectively from a plurality of locations in an obtained image. The arriving sound generation unit722outputs a generated acoustic signal (arriving sound) to the conversion unit723.

The conversion unit723converts an acoustic signal (arriving sound) generated by the arriving sound generation unit722into a signal in a frequency domain (frequency spectrum) from a signal in a time domain. The conversion unit723outputs an acoustic signal in a frequency domain obtained by the conversion to the correlation value calculation unit724, the power calculation unit726and the characteristic amount calculation unit727.

The correlation value calculation unit724uses a plurality of input acoustic signals so as to calculate a correlation value between acoustic signals that arrived from adjacent directions. The group setting unit725compares a calculated correlation value between acoustic signals with a prescribed correlation threshold and determines the presence or absence of a correlation between acoustic signals. Also, on the basis of a result of the determination of the presence or absence of a correlation between acoustic signals, the group setting unit725conducts grouping in which a pair of arrival directions of acoustic signals with a correlation is set as a one group. The group setting unit725outputs information on a set group to the reproduction unit729.

The power calculation unit726calculates powers of a plurality of input acoustic signals. The power calculation unit726outputs the calculated power of each acoustic signal to the reproduction unit729.

The characteristic amount calculation unit727analyzes a plurality of input acoustic signals so as to calculate a prescribed amount of acoustic characteristics. The characteristic amount calculation unit727outputs the calculated amount of acoustic characteristics to the state decision unit728. The state decision unit728compares the amount of acoustic characteristics calculated by the characteristic amount calculation unit727with an amount of acoustic characteristics in a normal condition stored in the storage unit730, and decides whether or not the acoustic signal (arriving sound) in each arrival direction is normal. The state decision unit728outputs the decision result to the reproduction unit729.

On the basis of the decision result by the state decision unit728and the power of each acoustic signal, the reproduction unit729determines whether or not the signal is decided to be not normal and there is an acoustic signal having a power equal to or greater than a threshold. When there is a corresponding acoustic signal, the reproduction unit729outputs, to the display device8, information representing the source of an abnormal acoustic signal on the image together with the image. When there is an acoustic signal decided to be not normal, the reproduction unit729outputs, to the display device, information representing a group including an arrival direction including the abnormality among the groups set by the group setting unit725together with the image. Note that when there is no acoustic signal decided to be not normal, the reproduction unit729outputs only an input image to the display device8.

Further, the reproduction unit729selects an acoustic signal arriving from a direction corresponding to a point specified by the operator in the image, and outputs it to the speaker9. The operator manipulates the input device10so as to specify the point in the image. In response to a manipulation of the input device10by the operator, an input signal in accordance with the manipulation content is input to the input reception unit731of the image and audio reproduction device7B from the input device10. The input reception unit731outputs an input signal to the reproduction unit729when the input signal is a signal related to the reproduction of an acoustic signal.

FIG. 12explains a method of setting an arrival direction of an arriving sound generated by an arriving sound generation unit.

The arriving sound generation unit722in the image and audio reproduction device7B according to present embodiment generates a sound arriving at the microphone array11(sound pickup device) from each of a plurality of locations in the image as described above. In the above generation, the arriving sound generation unit722generates a sound arriving from a location corresponding to each grid point Q in a two-dimensional grid set in an image displayed on the display device8of the inspection terminal5as illustrated in for exampleFIG. 12. The intervals between grid points Q can be set arbitrarily, and are set to be for example a value appropriate to the types and arrangements of pieces of equipment existing in the inspection target area (image capturing scope).

An arriving sound from a position corresponding to each grid point Q set in an image may be generated by using a known generation method. An arriving sound from grid point Q can be generated by folding a transmission characteristic corresponding to the position of grid point Q in the image (arrival direction) into a plurality of acoustic signals picked up by the microphone array11. An arrival direction can be calculated on the basis of a direction of an optical axis in the camera coordinate system of the image capturing device2and the eye direction passing through grid point Q.

In the inspection system1B according to present embodiment, when for example the operator manipulates the input device10of the inspection terminal5and the inspection terminal5and the relay device4are connected so as to allow communications between them, the image and audio reproduction device7B of the inspection terminal5performs a reproduction process as illustrated inFIG. 13.

FIG. 13is a flowchart explaining contents of a reproduction process performed by an image and audio reproduction device according to the second embodiment.

The image and audio reproduction device7B first starts to obtain an image signal and an acoustic signal (step S200). The process in step S200is performed by the communication unit721. The communication unit721outputs an obtained image signal to the reproduction unit729and outputs an obtained acoustic signal to the arriving sound generation unit722.

Next, the image and audio reproduction device7B generates arriving sounds from a plurality of arrival directions in the image capturing scope (step S201). The process in step S201is performed by the arriving sound generation unit722. As explained by referring toFIG. 12, the arriving sound generation unit722treats, as arrival direction, a direction corresponding to the position of each grid point Q set on the image so as to generate an arriving sound by folding the transmission characteristic corresponding to the arrival direction into a plurality of acoustic signals. The arriving sound generation unit722associates the generated arriving sound with the arrival direction so as to output them to the conversion unit723.

Next, the image and audio reproduction device7B calculates a power spectrum for each arrival direction by converting a plurality of generated arriving sounds from signals in a time domain into signals in a frequency domain (frequency spectrum) (step S202). The process in step S202is performed by the conversion unit723. The conversion unit723performs Fast Fourier Transform (FFT) on each of the plurality of arriving sounds so as to convert each arriving sound into a frequency spectrum. The conversion unit723outputs the calculated power spectrum to the correlation value calculation unit724, the power calculation unit726and the characteristic amount calculation unit727.

Next, the image and audio reproduction device7B performs a group detection process (step S203), a power calculation process (step S204) and a state decision process (step S205). Note that the order of the processes in steps S203, S204and S205is not limited to the order illustrated inFIG. 13, and these processes may be performed in different orders or may be executed in parallel.

The group detection process in step S203is performed by the correlation value calculation unit724and the group setting unit725. On the basis of the power spectrum for each arrival direction, the correlation value calculation unit724calculates a correlation value between power spectrums in adjacent arrival directions. In this example, adjacent arrival directions refer to two arrival directions that correspond to two grind points Q that are adjacent in the horizontal directions or in the vertical directions on the window among the plurality of grind points Q illustrated inFIG. 12. The correlation value calculation unit724calculates, as a correlation value, for example a mutual correlation coefficient for a power spectrum in a frequency domain. The correlation value calculation unit724associates a calculated correlation value with a pair of arrival directions (grind points Q) so as to output them to the group setting unit725.

The group setting unit725compares each of the input correlation values with a prescribed correlation threshold, and sets a pair of arrival directions with a high correlation as one group when there is a pair of arrival directions with a high correlation between acoustic signals. When one of the arrival directions constituting a pair has already been included in one group, the group setting unit725adds the other arrival direction to that group. Completing the setting of a group of arrival directions with high correlations, the group setting unit725outputs information regarding the set group to the reproduction unit729.

The power calculation process in step S204is performed by the power calculation unit726. The power calculation unit726uses the power spectrum for each arrival direction so as to calculate the power of an arriving sound for each arrival direction. The power calculation unit726outputs a calculated power for each arrival direction to the reproduction unit729.

The state decision process in step S205is performed by the characteristic amount calculation unit727and the state decision unit728. The characteristic amount calculation unit727calculates the amount of characteristics of the power spectrum of each arrival direction, and outputs the calculated amount of characteristics to the state decision unit728. The state decision unit728compares, for each arrival direction, the calculated amount of characteristics of the power spectrum with a prescribed amount of characteristics of a power spectrum in a normal condition, and decides whether or not the arriving sound is normal. The state decision unit728reads the amount of characteristics of the power spectrum in a normal condition from the storage unit730. In a state decision process according to present embodiment, similarly to a state decision process according to the first embodiment, the state flag of a normal arriving sound (power spectrum) is set to “0” and the state flag of a arriving sound (power spectrum) that is not normal is set to “1”. The state decision unit728outputs a state flag representing the state of a power spectrum to the reproduction unit729.

Completing a group detection process, a power calculation process and a state decision process, the image and audio reproduction device7B determines whether or not there is a group including an abnormal arriving sound (step S206). The determination in step S206is performed by the reproduction unit729. When there is a group that includes an arriving sound with the state flag of “1” and that has a power equal to or greater than a prescribed threshold, the reproduction unit729determines that group to be a group including an abnormal arriving sound. When there is a group including an abnormal arriving sound (Yes in step S206), the reproduction unit729adds information representing the arrival direction of an arriving sound included in that group to the image (window) (step S207). Thereafter, the reproduction unit729checks whether or not a signal of an instruction to reproduce an acoustic signal has been input (step S208). When there is no group including an abnormal arriving sound (No in step S206), the reproduction unit729skips the process in step S207, and performs the check in step S208.

When it is determined in step S208that a signal of an instruction to reproduce an acoustic signal has not been input (No in step S208), the reproduction unit729reproduces only the image signal and outputs it to the display device8(step S209). When it is determined in step S208that a signal of an instruction to reproduce an acoustic signal has been input (Yes in step S208), the reproduction unit729performs an image and audio output process (step S210). In step S210, the reproduction unit729selects an acoustic signal in an arrival direction specified by the input signal so as to output the image signal to the display device8and outputs the selected acoustic signal to the speaker9.

When the reproduction unit729has output an image signal or has output both an image signal and an acoustic signal, the image and audio reproduction device7B determines whether or not to continue the processes in steps S201through S210(step S211). When the processes are to be continued (Yes in step S211), the image and audio reproduction device7B performs the processes in step S201through S210for acoustic signals in the next process unit period. When the processes are to be terminated (No in step S211), the image and audio reproduction device7B terminates the process of obtaining an image signal and an acoustic signal, and also terminates the reproduction process, although this is omitted in the flowchart inFIG. 13.

The group detection process (step S203) in the flowchart illustrated inFIG. 13is performed by the correlation value calculation unit724and the group setting unit725as described above. The correlation value calculation unit724and the group setting unit725perform the process illustrated inFIG. 14as a group detection process.

FIG. 14illustrates a flowchart explaining contents of a group detection process performed in the reproduction process according to the second embodiment.

In a group detection process according to the present embodiment, the correlation value calculation unit724first calculates a correlation value of arriving sounds arriving from two adjacent arrival directions (step S203A). In step S203A, the correlation value calculation unit724performs, on all pairs of arrival directions that can be selected, a process in which two arrival directions are selected from among a plurality of arrival directions set in the image and a correlation value is calculated for the pair of the two selected arrival directions. The correlation value calculation unit724associates a calculated correlation value with the pair of the arrival directions and outputs them to the group setting unit725.

Next, the group setting unit725compares a correlation value calculated by the correlation value calculation unit724with a prescribed correlation value so as to determine whether or not there is a pair of arrival directions with a high correlation of arriving sounds (step S203B). When there is a pair of arrival directions with a high correlation of arrival sounds (Yes in step S203B), the group setting unit725sets a pair of arrival directions with a high correlation into one group (step S203C). When one of the arrival directions constituting a pair of arrival directions has already been included in one group in step S203C, the group setting unit725adds the other arrival direction to that group. When there is no pair of arrival directions with a high correlation of arriving sounds (No in step S203B), the group setting unit725skips the process in step S203C.

When the group setting unit725terminates the processes in steps S203B and S203C and outputs the process result to the reproduction unit729, the group detection process for the acoustic signals in the current process unit period is terminated.

Also, the state decision process (step S205) in the flowchart illustrated inFIG. 13is performed by the characteristic amount calculation unit727and the state decision unit728as described above. The characteristic amount calculation unit727and the state decision unit728perform the process illustrated inFIG. 15as a state detection process.

FIG. 15is a flowchart explaining contents of a state decision process in the reproduction process according to the second embodiment.

In the state decision process, first, the characteristic amount calculation unit727calculates the amount of acoustic characteristics of an acoustic signal converted into a frequency spectrum (step S205A). The characteristic amount calculation unit727outputs the calculate amount of acoustic characteristics to the state decision unit728. In the above situation, any amount of characteristics that permits the characterization of an arriving sound in the current process target period and that can decide whether it is a situation with equipment operating normally or a situation involving abnormality can be used as an amount of acoustic characteristics.

In response to input of an amount of acoustic characteristics calculated by the characteristic amount calculation unit727, the state decision unit728performs the processes in step S205B through S205F. The state decision unit728first reads an amount of acoustic characteristics in a normal condition from the storage unit730(step S205B). Next, the state decision unit728selects one arriving sound in an undetermined state (i.e., an arriving sound not having had its state determined) so as to compare the amount of acoustic characteristics with an amount of acoustic characteristics in a normal condition (step S205C), and determines whether or not the amount of acoustic characteristics of the selected arriving sound is within a normal range (step S205D). In step S205C and S205D, the state decision unit728obtains for example a difference between the selected amount of acoustic characteristics and the amount of acoustic characteristics in a normal condition so as to determine whether or not the difference is within an allowable range of a normal condition.

When a selected amount of acoustic characteristics is within a normal range (Yes in step S205D), the state decision unit728sets the state flag of the amount of acoustic characteristics of a selected arriving sound to “0” (step S205E). When a selected amount of acoustic characteristics is out of a normal range (No in step S205D), the state decision unit728sets the state flag of the amount of acoustic characteristics of a selected arriving sound to “1” (step S205F).

After setting a value of a state flag in step S205E or S205F, the state decision unit728checks whether or not there is an arriving sound whose state is not set (i.e., an arriving sound not having had its state flag set) (step S205G). When there is an acoustic signal whose state is not set (Yes in step S205G), the state decision unit728performs the processes in step S205C through S205F for an arriving sound whose state is not set. When the processes in steps S205C through S205F have been conducted for all arriving sounds (No in step S205G), the state decision unit728outputs the decision result to the reproduction unit729, and terminates the state determination process of the arriving sounds in the current process unit period.

After the group detection process and the state decision process above are performed, the reproduction unit729in the image and audio reproduction device7B performs the processes in steps S206through S210. The process in step S206through S210may respectively be similar to those in steps S103through S107explained in the first embodiment.

As described above, the present embodiment uses a plurality of acoustic signals picked up by the microphone array11so as to generate a sound (arriving sound) arriving from a position according to a grid point set in an image. When arriving sounds from adjacent arrival directions have a high correlation, the image and audio reproduction device7B treats such arrival directions as one group. Also, when it is decided that an arriving sound from an arrival direction included in one group involves abnormality, the image and audio reproduction device7B adds information representing the area of that group in the image to the image and displays the image on the display device8. This permits the operator of the inspection terminal5to recognize easily the location of the generation of an abnormal sound in an image.

Also, arriving sounds from a plurality of arrival directions included in one group have a high correlation of amounts of acoustic characteristics in acoustic signals converted into frequency domains. This permits the operator of the inspection terminal5to recognize easily that abnormal sounds of the same type have been picked up (detected) in an area enclosed by one frame, which represents the location of the generation of abnormal sounds in an image. This further enables the operator to recognize easily what type of a sound the abnormal sound is just by selecting one location in an area enclosed by one frame in an image. Accordingly, the present embodiment permits the operator to conduct, with less efforts, confirmation of the presence or absence of an abnormal sound and the identification of the location of the generation of the abnormal sound in the image capturing scope of the image capturing device2.

Further, the present embodiment treats as an arrival direction a position corresponding to a two-dimensional grid point set in an image and determines whether or not a sound arriving from each arrival direction is normal. Thereby, the present embodiment permits the investigation of whether or not an abnormal sound has been generated with a higher degree of fineness than in the first embodiment.

Note that explanations of the above group detection process (step S203) used an example in which a correlation value is calculated between power spectrums of arriving sounds after being converted into frequency spectrums by the correlation value calculation unit724. However, the correlation value calculation unit724is not limited to this example, and may for example calculate a time change amount of power spectrum in an arriving sound after being converted into a frequency spectrum so as to calculate a correlation value between time change amounts.

Third Embodiment

FIG. 16illustrates a configuration example of a monitoring system according to a third embodiment.

As illustrated inFIG. 16, a monitoring system1C according to the present embodiment can collectively monitor a plurality of monitored areas12A through12C. The monitored area12A is provided with the image capturing device2A and the microphone array11A. A monitored area12B is provided with the image capturing device2B and the microphone array11B. The monitored area12C is provided with the image capturing device2C and the microphone array11C. Images captured by image capturing devices and acoustic signals picked up by the microphone array in the monitored areas12A through12C are respectively transmitted to the image and audio reproduction device7C via a communication network13such as the Internet etc.

The image and audio reproduction device7C performs a reproduction process explained in the second embodiment or a process similar to a reproduction process explained in the second embodiment so as to generate information representing a location at which an abnormal sound is being generated in an image when an arriving sound involves abnormality. The image and audio reproduction device7C transfers to monitoring equipment14information representing an image and an acoustic signal that were obtained and a location at which an abnormal sound is being generated.

The monitoring equipment14is a piece of equipment provided to a facility that displays, on the display device8, information representing an image and an acoustic signal that were obtained and a location at which an abnormal sound is being generated, so as to monitor the monitored areas12A through12C. The monitoring equipment14includes the display device8, the speaker9, the input device10and a control device15.

FIG. 17is a block diagram illustrating a functional configuration of an image and audio reproduction device used in the monitoring system according to the third embodiment.

As illustrated inFIG. 17, the image and audio reproduction device7C according to the present embodiment includes a communication unit741, an arriving sound generation unit742, a conversion unit743, a correlation value calculation unit744, a group setting unit745, an SNR calculation unit746and a display position calculation unit747. Also, the image and audio reproduction device7C further includes a state decision unit748, a display position determination unit749, a first storage unit750, a second storage unit751, an arriving sound transmission unit752and an image transmission unit753.

The communication unit741conducts communications with a different communication device via the communication network13so as to obtain an image captured by the image capturing devices2(2A through2C) and a plurality of acoustic signals output from the microphone arrays11(11A through11C). The communication unit741makes the first storage unit750store an obtained image and acoustic signals.

The arriving sound generation unit742reads a plurality of acoustic signals from the first storage unit750so as to generate acoustic signals arriving at the microphone array11from a plurality of locations in a captured image. The arriving sound generation unit742outputs generated acoustic signals (arriving sounds) to the conversion unit743.

The conversion unit743converts the acoustic signals (arrival sounds) generated by the arriving sound generation unit742from signals in a time domain into signals in a frequency domain (frequency spectrums). The conversion unit743outputs the acoustic signals after the conversion to the correlation value calculation unit744and the SNR calculation unit746.

The correlation value calculation unit744uses a plurality of input acoustic signals so as to calculate a correlation value between acoustic signals that are from adjacent directions. The correlation value calculation unit744outputs the calculated correlation value to the group setting unit745.

The group setting unit745determines the presence or absence of a correlation value between acoustic signals by comparing the calculated correlation values between the acoustic signals with a prescribed correlation threshold, and conducts grouping in which adjacent arrival directions with a high correlation between acoustic signals are set as one group. The group setting unit745outputs information on a set group to the display position calculation unit747.

The SNR calculation unit746uses a plurality of input acoustic signals (arriving sounds) so as to calculate a signal-to-noise ratio (SNR) of each arriving sound. The SNR unit outputs the calculated SNR of each arriving sound to the display position calculation unit747.

On the basis of information of grouped arrival directions and an SNR of each arriving sound, the display position calculation unit747calculates a display position for superimposing information representing a group of arrival directions on an image and displaying the image. According to the present embodiment, the display position calculation unit747treats, as a display candidate, only a group in which the SNR calculated on the basis of the SNR of each arriving sound is equal to or higher than a prescribed threshold from among groups of arrival directions, and calculates a display position of that display candidate. The display position calculation unit747outputs a calculated display position to the state decision unit748and the display position determination unit749so as to make the second storage unit751store it.

The state decision unit748compares a display position calculated by the display position calculation unit747and a past display position stored in the second storage unit751regarding a group of arrival directions, and decides whether or not the state of the sound arriving from the display position is the same as that in a normal condition. The state decision unit748outputs a decision result to the display position determination unit749.

On the basis of the decision result by the state decision unit748, the display position determination unit749determines, to be a display position of information to be superimposed on an image, a display position of a group whose sound state was determined to be not normal among display positions of groups of arrival directions calculated by the display position calculation unit747. The display position determination unit749transmits a determined display position to control device15.

The arriving sound transmission unit752transmits to the control device15arriving sounds in a frequency domain obtained by converting acoustic signals in a time domain by using the conversion unit743. The arriving sound transmission unit752selects for example an acoustic signal arriving from a direction corresponding to a point in an image specified by the operator of the monitoring equipment14, and outputs it to the speaker9. The operator manipulates the input device10of the monitoring equipment14so as to specify a point in an image. In response to the manipulation of the input device10by the operator, an input signal in accordance with the manipulation content is input to the image and audio reproduction device7C via the control device15.

The image transmission unit753transmits to the control device15an image signal that was from the image capturing device2that is stored in the first storage unit750.

FIG. 18is a block diagram illustrating a functional configuration of a control device used in the monitoring system according to the third embodiment. The control device15of the monitoring equipment14includes a reproduction unit1501and a control unit1502as illustrated inFIG. 18. The reproduction unit1501includes an image process unit1501A and an audio process unit1501B.

The image process unit1501A obtains an image signal transmitted from the image transmission unit753of the image and audio reproduction device7C, converts it into a data format that allows it to be displayed on the display device8, and outputs it to the display device8.

The audio process unit1501B obtains an acoustic signal transmitted by the arriving sound transmission unit752of the image and audio reproduction device7C, converts it into a data format that allows it to be output from the speaker9, i.e., into a signal in a time domain from a signal in a frequency domain (frequency spectrum), and outputs it to the speaker9.

The control unit1502, when receiving a display position determined by the display position determination unit749of the image and audio reproduction device7C, adds information of a frame line etc. to the obtained display position in an image signal when the image process unit1501A processes the image signal, the information representing an area in which an abnormal sound is being generated. Also, when the operator has selected a point in an area in which an abnormal sound is being generated in an image by manipulating the input device10, the control unit1502transmits the selected position to the arriving sound transmission unit752of the image and audio reproduction device7C. Receiving a signal representing a selected position from the control device15, the arriving sound transmission unit752of the image and audio reproduction device7C extracts an arriving sound whose arrival direction is the selected position from the arrival sound generated by the arriving sound generation unit742and outputs it to the audio process unit1501B of the control device15.

Receiving an instruction to start a reproduction process in response to for example a manipulation of the input device10by the operator of the monitoring equipment14, the image and audio reproduction device7C in the monitoring system1C of the present embodiment performs the processes illustrated inFIG. 19.

FIG. 19is a flowchart explaining contents of a reproduction process performed by the image and audio reproduction device according to the third embodiment. The flowchart illustrated inFIG. 19explains a series of processes performed on an image obtained from an image capturing device arranged in one monitored area and a plurality of acoustic signals obtained from a microphone array. In other words, the image and audio reproduction device7C performs the processes illustrated inFIG. 19for each monitored area.

In a reproduction process according to the present embodiment, the image and audio reproduction device7C first starts to obtain an image signal and an acoustic signal (step S300). The process in step S300is performed by the communication unit741. The communication unit741makes the first storage unit750store obtained image signal and acoustic signal in such a manner that they can be identified for each monitored area.

Next, the image and audio reproduction device7C generates arriving sounds that are from a plurality of arrival directions in the image capturing scope (step S301). The process in step S301is performed by the arriving sound generation unit742. As explained by referring toFIG. 12, the arriving sound generation unit742generates arriving sounds by folding a transmission characteristic into a plurality of acoustic signals, the transmission characteristic corresponding to arrival directions that are treated as directions corresponding to respective grid points Q set in an image. The arriving sound generation unit742outputs the generated arriving sounds to the conversion unit743in a state that they are associated with the arrival directions.

Next, the image and audio reproduction device7C calculates a power spectrum for each arrival direction by converting a plurality of generated arriving sounds from signals in a time domain into signals in a frequency domain (frequency spectrum) (step S302). The process in step S302is performed by the conversion unit743. The conversion unit743performs Fast Fourier Transform on each of the plurality of arriving sounds so as to convert each arriving sound into a signal in a frequency domain. The conversion unit743outputs a calculated power spectrum to the correlation value calculation unit744and the SNR calculation unit746.

Next, the image and audio reproduction device7C performs a group detection process (step S303) and a process of calculating the SNR of each signal (step S304). Note that the order of the processes in step S303and s304is not limited to the order illustrated inFIG. 19, and these processes may be performed in the reverse order or may be executed in parallel.

The group detection process in step S303is performed by the correlation value calculation unit744and the group setting unit745. The correlation value calculation unit744calculates a correlation value between power spectrums that are from arrival directions adjacent to each other, on the basis of the power spectrum for each arrival direction. The correlation value calculation unit744calculates for example a mutual correlation coefficient for a power spectrum in a frequency domain as a correlation value. The correlation value calculation unit744associates a calculated correlation value with a pair of arrival directions (grid points Q) so as to output them to the group setting unit745.

The group setting unit745compares each of the input correlation values with a prescribed correlation threshold, and sets a pair of arrival directions with a high correlation as one group when there is a pair of arrival directions with a high correlation between acoustic signals. When one of the arrival directions constituting a pair has already been included in one group, the group setting unit745adds the other arrival direction to that group. Completing the setting of a group of arrival directions with high correlations, the group setting unit745outputs information regarding the set group to the display position calculation unit747.

The process of calculating an SNR in step S304is performed by the SNR calculation unit746. The SNR calculation unit746calculates an SNR (signal-to-noise ratio) for the arriving sound for each arrival direction. The SNR calculation unit746outputs the calculated SNR of each arrival sound to the display position calculation unit747.

Completing the processes in step S303and S304, the image and audio reproduction device7C calculates a display position on the basis of an SNR (step S305). Step S305is performed by the display position calculation unit747. After for example calculating the SNR of each of the groups of arrival directions on the basis of the SNR of each arriving sound, the display position calculation unit747calculates a display position of a case when a group in which the SNR calculated among the groups of arrival directions is higher than a prescribed threshold is superimposed on an image and the group is displayed. The display position calculation unit747associates a calculated display position with a group of arrival directions so as to output them to the state decision unit748and the display position determination unit749, and makes the second storage unit751store them.

Completing the process in step S305, the image and audio reproduction device7C performs a decision process (step S306). The decision process in step S306is performed by the state decision unit748. The state decision unit748compares a calculated display position with a past display position so as to decide whether or not a change in the display position is within a scope of changes in a normal condition. The state decision unit748outputs a decision result to the display position determination unit749.

Completing the process in step S306, the display position determination unit749checks whether or not the change in a display position is within a normal range on the basis of the input decision result (step S307). When the change in a display position is not within a normal range (No in step S307), the display position determination unit749determines a display position of a group of arrival directions involving an abnormal sound and outputs the determined position to the control device15(step S308). Thereafter, the image and audio reproduction device7C transmits an image signal to the control device15(step S309). The transmission process in step S309is performed by the image transmission unit753. The timing for the transmission process in step S309is not limited to one after step S308, and may be an arbitrary timing.

When the change in a display position is within a normal condition (Yes in step S307), which means that there is no group involving an abnormal arriving sound (abnormal sound), the display position determination unit749skips the process in step S308. Thereafter, the image and audio reproduction device7C transmits an image signal to the control device15(step S309).

After the transmission process in step S309, the image and audio reproduction device7C determines whether or not to continue the processes in step S301through S309(step S310). When the processes are to be continued (Yes in step S310), the image and audio reproduction device7C performs the processes in step S301through S309for acoustic signals in the next process unit period. When the process are to be terminated (No in step S310), the image and audio reproduction device7C terminates the reproduction process.

When information representing an area in which the abnormal sound was generated is included in an image displayed on the display device8, the operator of the monitoring equipment14can reproduce the abnormal sound by manipulating the input device10. In response to a manipulation of the input device10by operator to select an arbitrary position in the area in which the abnormal sound was generated, the control device15performs the process illustrated inFIG. 20.

FIG. 20explains contents of a process that is performed by the controller device according to the third embodiment in cooperation with the image and audio reproduction device.

The control device15first receives a signal representing a selected position from the input device10, and transmits the signal of the selected position to the image and audio reproduction device7C (step S320). The process in step S320is performed by the control unit1502. A signal representing the selected position transmitted by the control unit1502is received by the arriving sound transmission unit752of the image and audio reproduction device7C. Receiving a signal representing the selected position, the arriving sound transmission unit752extracts an arriving sound from the arrival direction corresponding to the selected position so as to transmit it to the control device15.

Receiving an arriving sound from an arrival direction corresponding to a selected position, the control device15converts the received arriving sound from a signal in a frequency domain (frequency spectrum) into a signal in a time domain (step S321). The process in step S321is performed by the audio process unit1501B. After step S312, the audio process unit1501B outputs the arriving sound after being converted into a signal in a time domain to the speaker9(step S322).

As described above, the present embodiment uses an image and acoustic signal obtained from the image capturing devices2(2A through2C) and the microphone arrays11(11A through11C) connected to the image and audio reproduction device7C via the communication network13, and decides the presence or absence of the generation of an abnormal sound in an image. This permits the monitoring of the generation of an abnormal sound in a plurality of monitored areas by using a pair of the image and audio reproduction device7C and the monitoring equipment14.

Also, differently from the first or second embodiment, the present embodiment decides whether or not an abnormal sound has been generated, on the basis of the display position of a case when a group of arrival directions with a high correlation is displayed and on the basis of the SNR of the arriving sound. When an abnormal sound has been generated, that sound is sometimes a sound different from those in a usual condition, such as a large sound, at a position close to the generation source. Thereby, when an abnormal sound has been generated, changes can easily occur to the position or the shape of an area representing a group of arrival directions with a high correlation at a position close to the generation source. Also, when a great change has occurred to the display position, the present embodiment determines that an abnormal sound is generated only when the SNR is equal to or higher than a prescribed threshold. This can prevent a situation in which a change in the display position caused by a noise, which is not an operation sound of a monitoring-target equipment, is mistaken as a change caused by the generation of an abnormal sound.

The reproduction processes explained in the first through third embodiments above are just exemplary and part of the processes may be changed in accordance with the necessity. It is also possible for example to analyze an image captured by the image capturing device so as to estimate the distance to each object in the image and generate an arriving sound by folding a transmission characteristics based on the arrival direction and the distance into a plurality of acoustic signals on the basis of the estimated distance and arrival direction in the arriving sound generation units722and742.

A correlation between arriving sounds from adjacent arrival directions is not limited to a correlation of powers in a time domain or power spectrums in a frequency domain, or may be a correlation of time change amounts of power spectrums in a frequency domain.

Also, the image and audio reproduction devices7A through7C above may be implemented by for example a computer and a program that causes the computer to execute the above reproduction processes. Hereinbelow, a computer that can be operated as the image and audio reproduction devices7A through7C is explained by referring toFIG. 21.

FIG. 21illustrates a hardware configuration of a computer.

As illustrated inFIG. 21, a computer20includes a processor2001, a main memory2002, an auxiliary memory2003, an input device2004, a display device2005, a communication device2006, an interface device2007and a medium driving device2008. These constituents2001through2008in the computer20are connected to each other via a bus2010so that data can be exchanged between the constituents.

The processor2001is a processing device such as a central processing unit (CPU) etc., and controls the entire operation of the computer20by executing various types of programs including an operating system.

The main memory2002includes a Read Only Memory (ROM) and a Random Access Memory (RAM). The ROM in advance stores for example a prescribed basic control program etc. that is read by the processor2001upon the activation of the computer20. Also, the RAM is used as a working storage area in accordance with the necessity when the processor2001executes various programs.

The auxiliary memory2003is a storage device having a capacity larger than that of the main memory2002such as a hard disk drive (HDD), a solid state drive (SSD), etc. The auxiliary memory2003is made to store various types of programs executed by the processor2001, various types of data, etc. An example of a program stored in the auxiliary memory2003is for example a program that causes the computer20to execute for example the process illustrated inFIG. 5throughFIG. 7, inFIG. 13throughFIG. 15or inFIG. 19. An example of data stored n the auxiliary memory2003is an image signal obtained from the image capturing device2, an acoustic signal obtained from the sound pickup device, etc.

An example of the input device2004is a keyboard or a mouse. The input device2004, manipulated by the operator of the computer20, transmits input information associated with that manipulation to the processor2001. Also, a touch panel that is overlapped on the display surface of the display device2005for example may be used as the input device2004.

The communication device2006is a device that connects the computer20to a different communication device so that communications are possible in accordance with a prescribed communication standard. An example of the communication device2006may be a device that connects the computer20and the relay device4by wireless short-distance or long-distance communications or may be a device that connects the computer20to the communication network13such as the Internet etc. via wired or wireless communications.

An example of the interface device2007is an input/output device for connecting the speaker9or other electronic devices to the computer20.

The medium driving device2008reads a program and data stored in a portable recording medium (not illustrated) and writes data etc. stored in the auxiliary memory2003to a portable recording medium. A flash memory having a USB-based connector for example may be used as a portable recording medium. Also, when the computer20is provided with a reader/writer of a memory card that can be used as the medium driving device2008, a memory card compatible with the reader/writer (SD-based memory card etc. for example) can be used as a portable recording medium. Further, when the computer20is provided with an optical disk drive that can be used as the medium driving device2008, an optical disk such as a Compact Disc (CD), a Digital Versatile Disc (DVD), a Blu-ray Disc (Blu-ray is a registered trade mark), etc. can be used as a portable recording medium.

In this computer20, the processor2001reads a program including the above described processes from the auxiliary memory2003and executes a reproduction process on an acoustic signal obtained from a sound pickup device in cooperation with the main memory2002, the auxiliary memory2003, etc. During the execution of a reproduction process, the processor2001functions (or operates) as for example the correlation value calculation unit702, the group setting unit703, the acoustic-characteristic amount calculation unit704, the vibration-characteristic amount calculation unit705, the state decision unit706and the reproduction unit707. Note that the processor2001may have part of the arithmetic operations in reproduction processes executed by a different arithmetic process device (not illustrated) such as a digital signal processor (DSP) etc.

Also, the computer20used as the image and audio reproduction devices7A through7C does not have to include all the constituents illustrated inFIG. 21, and some of the constituents may be omitted in accordance with the purposes or conditions. Also, the computer20is not limited to a general-purpose type computer that implements a plurality of functions by executing various types of programs, but may be an information processing apparatus dedicated to the processes explained in the first through third embodiments.