System for tracking object, and camera assembly therefor

Disclosed is an autonomous distributed system for detecting and tracking a falling object within a region being monitored by using a plurality of camera assemblies cooperating with each other, and disclosed is a camera assembly therefor. Each of the camera assembly, normally, operates in a falling object detecting mode, but, when detecting a falling object, shifts to a tracking mode. In the tracking mode, a degree of danger of the falling object is determined by referring to the size of the falling object in real space and the movement of a falling. In addition, when the falling object is further tracked and moved outside of the angle of view, a search request including information of the degree of danger as to the falling object is issued to surrounding camera assemblies, such that the multiple camera assemblies search for the falling object in cooperation.

TECHNICAL FIELD

The disclosure relates to a system for detecting and tracking an object, and a camera assembly therefor. The object includes a falling object on a road.

BACKGROUND

Conventionally, it is known that a system for detecting a falling object on a road by using image data output from a camera (see, for example, Patent Documents 1 to 5). In most of the conventional detecting systems, each camera arranged along the road has independently been performing an image processing for detecting a falling object on the road, respectively. When the camera has detected a falling object, the camera notifies a monitoring center of alarm signal including information that there is some falling object in a detecting area. Then, the monitoring center issues dispatch orders to a falling object correcting vehicle.

Further, it is known that a traffic control system for classifying an occurrence event such as congestion or an input event, into a cause event and a result event (for example, see Patent Document 6).

In addition, an intruding object monitoring method for tracking the object using a plurality of cameras is known as an image processing technique (for example, refer to Patent Document 7), or an image monitoring apparatus with converting from the acquired image coordinate system to a drawing coordinate system by calibrating camera parameters, is known as an image processing technique (for example, refer to Patent Document 8).

CITATION LIST

Patent Document

Non Patent Document

Non Patent Document 1: Takashi FUSE, two others: “Development of a Technique for Vehicle Maneuvers Recognition with Sequential Images from High Altitude Platforms” Journal of Infrastructure Planning and Management, Jul. 20, 2003, IV-60, No. 737, Page 159 to 173, Internet <URL: http://planner.t.u-tokyo.ac.jp/archive/web/research/fuse/vehicle_recognition.pdf>

SUMMARY

Conventional falling object detection system has two problems. First, when a falling object is detected, the degree of danger as to the falling object is not taken into consideration. For example, a heavy and large falling object such as an iron pipe or a drum should be estimated at high degree of danger, a light and small falling object like a plastic bag should be estimated at low degree of danger. In a situation that degree of danger as to a falling object is not considered, there is a risk that delay of the collecting work as to the dangerous falling object may lead to serious accidents, furthermore, a loss that collecting an object which does not need to be collected may be generated.

A second problem is that each camera connected to a network has independently been performing detecting a falling object, respectively. When each camera operates independently for detecting a falling object, the required total capability for image processing in the network may increase in proportion to the number of cameras connected to the network. Furthermore, it takes time to grasp a situation that the detected falling object moved to a monitoring range of the other camera, for example, as a result that the detected falling object came in contact with a vehicle.

The disclosure provides a system for detecting and tracking an object, and camera assembly therefor. The system can quickly initialize a collection operation as to the object including a falling object on a road by considering the degree of danger as to the object, and make the grasp of the situation more effectively by cooperating with peripheral cameras.

In one aspect of the disclosure, a system consists of at least a plurality of camera assemblies connected to a network, each camera assembly is arranged at an area to be monitored, respectively. A camera assembly can detect an object by referring to images within an angle of view, captured by an imaging device of the camera assembly. When the camera assembly detects an object, the camera assembly may issue a message notifying that the camera assembly detects the object to other device connected to a network. The object may include a falling object from a vehicle on a road. For example, in a normal state, each of the plurality of camera assemblies searches for a fallen object on the road to be monitored by controlling cyclically the photographing angle of view of the camera, respectively.

When a camera assembly detects a falling object, the camera assembly may transmit a message including a degree of danger as to the falling object to other device connected to a network, then a controlling procedure of the camera assembly may shift to Tracking mode to track the falling object.

The degree of danger may be evaluated at least by considering a moving state of the falling object, or a size of the falling object in real space. As the moving state, at least general movement and partial movement as to the falling object may be observed using the camera assembly. When the camera assembly determines that the falling object moved, the camera assembly may transmit a message including a reevaluated degree of danger to other device connected to a network. When the falling object is lost from the angle of view of the camera assembly, the camera assembly may transmit a message including a request for searching the object to the other camera assembly connected to the network.

The message may include area information about the area firstly look for the object. When the other camera assembly receives the message including a request for searching the object and the area information regarding the object, the camera assembly firstly searches the area specified by the area information for the object. For example, the area information may include the information about the place where the object really exists or the information about the possible place where the object may exist. For example, the area information may include at least one of the information regarding a location of the camera assembly which detects the object, the information regarding a current location of the object, or the information regarding a location that the falling object may exist on the global coordinate system, predicted by referring to a moving path of the falling object.

A timing to transmit the message including the request for searching the object to the other device connected to the network is not limited to the timing when the falling object is lost from the angle of view of the camera assembly. The message including the request for searching the object to the other device connected to the network may transmit at any timing including following timings:

(1) A timing when the camera assembly determines that the object cannot be captured clearly, for example, based at least in part on influence of bad natural environments.

(2) A timing when the camera assembly determines that the object is far away.

(3) A timing when the camera assembly predicts that the object may move to the outside of the angle of view soon.

(4) A timing when the camera assembly determines that the other device can capture the falling object more clearly.

The other device connected to the network, which might receive the message including the request for searching an object may correspond to a surrounding camera assembly. For example, a different camera assembly installed around the camera assembly, which transmitted the message including the request for searching the object, may correspond to the camera assembly, which should search the falling object continuously. When the different camera assembly receives a message including the request for searching the falling object, and the area information to be searched, the different camera assembly may change its operational mode to Request mode. In Request mode, the camera assembly may search for the falling object by turning initially an angle of view of an imaging device to a direction identified by the area information in received message.

According to one aspect of the disclosure, it is possible to collect a falling object having high degree of danger quickly, and to prevent the occurrence of serious accidents. Furthermore, when a falling object moves, tracking the object is performed in conjunction with the surrounding camera, so grasping the on-site situation can do effectively.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

In a normal situation, a falling object detecting system according to an embodiment of the disclosure searches for a certain object other than a vehicle, in other words, a falling object on the road, by changing a photographing angle of view of a camera cyclically, according to preset data for controlling the photographing angle of view of the camera. When the camera detects an object image within the photographing angle of view of camera, an alarm message including the information as to a degree of danger may be issued from the camera, and a controlling procedure of the camera shifts to Tracking mode. In Tracking mode, the degree of danger may be reevaluated by observing at least a position, a size or movement of the object. When it is presumed that the object was moved, for example, caused by coming in contact with another vehicle or the like, a new alarm may be issued with referring to the degree of danger, which has reevaluated in the system. For example, the new alarm issued by the camera may correspond to a message of Accident Outbreak Warning message.

FIG. 1is a diagrammatic illustration of a falling object detecting system according to a first embodiment. The falling object detecting system includes a plurality of camera assemblies, each camera assembly1arranged to along a road so as to photograph a monitoring area of the road, respectively, monitoring center3for monitoring the images received from the each camera assembly1and outputting instructions as necessary, and network4for connecting each camera assembly1with monitoring center3. For example, two camera assemblies consisted of camera assembly1and camera assembly2are representative in the embodiment, a plurality of camera assemblies, more than two camera assemblies, can be arranged with an appropriate distance each other and located along the road. Each camera assembly1or camera assembly2may include an electrical panning and tilting head, an electrical zooming lens, or an image recognition device for detecting and tracking an object. Each camera assembly1or camera assembly2may issue an alarm notification to monitoring center3via network4when detecting an object, respectively. Each camera assembly1or camera assembly2can be arranged corresponding to each Electronic Toll Collection System (ETC) roadside machine respectively, for the purpose of considering ETC 2.0 non-compliant vehicle.

A display device installed in monitoring center3is connected to camera assembly1via network4. The display device is configured to display an image of the falling object captured by camera assembly1, with information regarding the degree of danger as to the falling object. The information regarding the degree of danger as to the falling object to be displayed is determined by referring to at least the degree of danger identified by camera assembly1. The display device may have an attribute circuit configured to display the degree of danger as to the falling object at a screen, with various attribute corresponding to each degree of danger.

When a device installed in monitoring center3receives the alarm notification, a device installed in monitoring center3, including the display device can support an operation of an operator by displaying the degree of danger as to the falling object at a screen, with various attribute corresponding to each degree of danger. For example, the operation of the operator may include issuing dispatching orders to a falling object collecting vehicle. Since the image as to the falling object is tracked by image recognition process in camera assembly1, the image as to the falling object can be automatically displayed under zooming up appropriately. Furthermore, the operator can arbitrarily change the camera assembly from the camera assembly1currently capturing the falling object to another camera assembly, or change the display mode from current display mode to another display mode. At this time, when it is judged that an accident was broken out, it is good to be able to easily contact the nearest fire department (ambulance), police, etc. in addition to the falling object collecting vehicle. Network4is a communication medium capable of IP communication, for example, an optical network such as Gigabit Ethernet-Passive Optical Network (GE-PON) may be selected. The term, “Gigabit Ethernet-Passive Optical Network (GE-PON)” and the term, “Ethernet” are trademarks.

FIG. 2is a hardware block diagram of camera assembly1for the falling object detecting system in the embodiment. Each of camera assembly1or camera assembly2may include similar circuits, respectively. Each of camera assembly1or camera assembly2includes imaging device11, PTZ device12, video encoding device13, image recognition device14, and camera controlling device15, respectively.

FIG. 12shows imaging device11in accordance with an embodiment of the present disclosure.

Imaging device11may include a main body of camera having at least imaging elements112and zoom lens111for coordinating a focus with an object. The imaging elements112may include color or monochrome image elements.

Imaging device11may include circuit113to output a digital image signal captured by the imaging elements112, to each of PTZ device12, video encoding device13, or image recognition device14.

FIG. 13shows PTZ device12in accordance with an embodiment of the present disclosure.

PTZ device12may contain a mechanism for supporting and driving imaging device11. PTZ device12may include mechanism and circuit121configured to pan and tilt imaging device11. PTZ device12may include mechanism and circuit122configured to drive the zoom lens of imaging device11. PTZ device12may include circuit123configured to adjust an angle of view of imaging device11according to a specific preset position, which is identified by camera controlling device15.

PTZ device12may include memory circuit124to memorize a plurality of preset positions125. Each preset position may include a controlling value of panning operation of imaging device11to direct a specific target, a controlling value of tilting operation of imaging device11to direct the specific target, and a controlling value of zooming operation of imaging device11to focus the specific target, respectively.

Memory circuit124in PTZ device12may be configured to store a list of preset positions125containing a plurality of preset positions. The list of preset positions125may include at least one of following preset positions:

(1) Preset position1corresponding to an area around camera assembly1

(2) Preset position2corresponding to an area around another camera assembly, installed in the neighboring area

(3) Preset position3corresponding to one of the divided areas from a vicinity of camera assembly1to an adjacent camera assembly

(4) Preset position4corresponding to an area where searching for an object may be requested from the other camera assembly

Each preset position can be set so that imaging device11can capture each area at appropriate magnification, respectively.

PTZ device12may include a mechanism or circuit configured to pan and tilt imaging device11, or drive the zoom lens of imaging device11by referring to a controlling values of panning, tilting or zooming operation, directly identified from camera controlling device15, without referring to the preset position memorized in the memory circuit of PTZ device12.

FIG. 14shows Video encoding device13in accordance with an embodiment of the present disclosure.

Video encoding device13may include circuit131configured to encode a digital signal inputted from imaging device11to MPEG-code with a quality specified by camera controlling device15.

Video encoding device13may include circuit131configured to output the MPEG-coded image data with the specified quality as an image data captured by camera assembly1, to a device connected to network4through interface circuit1541.

FIG. 15shows Image recognition device14in accordance with an embodiment of the present disclosure.

Image recognition device14may include processing circuit141configured to perform an image data processing and an image recognition processing on the digital signal from imaging device11.

Image recognition device14may include circuit142configured to detect an object by referring to images within an angle of view, captured by imaging device11of the camera assembly1.

Image recognition device14may include circuit143configured to perform detection of an object on an area, or especially perform a detection of a falling object on a traffic lane or a peripheral area. Circuit143may include a circuit configured to recognize a traveling direction of the vehicle, or the traffic lane by optical flow techniques or the like.

Image recognition device14may include circuit144configured to perform recognition of a normally traveling vehicle, or read a vehicle's registered number from a traveling vehicle.

Image recognition device14may include circuit145configured to convert a coordinate of an object on the screen coordinate system to a coordinate of the object on the global coordinate system by referring to the controlling values of panning operation, tilting operation, or zooming operation in PTZ device12.

Image recognition device14may include circuit146configured to perform a method of template matching for an object, circuit147configured to perform a statistical learning method using feature quantities for distinguishing an object from a traveling vehicle or a shadow of the vehicle, circuit148configured to perform the image processing method using a background difference for reducing a target area for processing, or circuit149configured to perform the processing of Time Space Fast Fourier Transform (Time Space FFT) for identifying a partial movement of the object.

FIG. 16shows Camera controlling device15in accordance with an embodiment of the present disclosure.

Camera controlling device15may include controller1501configured to control the overall of the camera assembly1including a transition of an operational mode.

Camera controlling device15may include circuit1502configured to shift an operational mode.

Camera controlling device15may include circuit1503configured to control imaging device11so as to turn an angle of view of imaging device11to a specific direction by using PTZ device12, the specific direction including a direction identified from the other device including a different camera assembly, connected to a network.

Camera controlling device15may include circuit1504configured to search for an object on an area to be monitored by controlling cyclically the photographing angle of view of the camera. Circuit1504may include a detector configured to detect a specific object as a target object that is identified by other device. Circuit1504may include a detector configured to detect a specific object as a target object, the specific object having a characteristic value that is the same as a characteristic value of the target object that was notified of by the other device connected the network, or is close to the characteristic value of the target object. Circuit1504may include a detector configured to detect a specific object as a target object, the specific object having a degree of danger that is the same as a degree of danger of the target object that was notified of by the other device connected the network, or is close to the degree of danger of the target object. Circuit1504may include a controller configured to control the angle of view of imaging device11so as to shift to a traveling direction of the vehicle little by little to look for the object.

Camera controlling device15may include circuit1505configured to track an object by referring to a characteristic value of an object.

Camera controlling device15may include circuit1506configured to track an object by referring to a characteristic value of an object, for example, the characteristic value of the object may include a value received from a different camera assembly or other device connected to network4, via an interface circuit.

Camera controlling device15may include circuit1507configured to turn an angle of view of imaging device11by referring to area information1528received from a different camera assembly or other device connected to network4, via an interface circuit.

Camera controlling device15may include circuit1508configured to control at least PTZ device12so that an image of an object is captured within an angle of view of imaging device11by referring to an output of image recognition device14.

Circuit1508may include a circuit configured to control at least PTZ device12so that the image of an object is captured at near the center position of the angle of view of imaging device11, and at suitable magnification.

Camera controlling device15may include circuit1509configured to control video encoding device13. Circuit1509may include a circuit configured to specify quality rate of MPEG-coding to video encoding device13. The quality rate of MPEG-coding may include compressibility as to input image data captured by imaging device11. The quality rate of MPEG-coding may include the quality rate received from the other device connected to network4. The quality rate of MPEG-coding may be stored in video encoding device13or camera controlling device15. Circuit1509may include a circuit configured to flow the MPEG-coded image data with the specified quality, generated by video encoding device13to the other device connected to network4as an image of a monitoring camera.

Camera controlling device15may include circuit1510configured to control image recognition device14so that image recognition device14performs a desired image data processing and an image recognition processing on the digital signal from imaging device11.

Camera controlling device15may include transmitter1511configured to transmit coded video data captured by imaging device11to at least a device connected to network4, by using interface circuit1541.

Camera controlling device15may include transmitter1512configured to transmit a message including a request for searching a specific object to a different camera assembly or a device connected to network4, via interface circuit1541.

Camera controlling device15may include transmitter1513configured to transmit a message notifying that the camera assembly1detects the object to other device connected to network4, via interface circuit1541.

Camera controlling device15may include transmitter1514configured to transmit a message including at least a degree of danger as to an object to other device connected to network4, via interface circuit1541. Transmitter1514may include a circuit configured to transmit a message including a degree of danger depending on positional information of the object, identified by circuit143in image recognition device14. Circuit143may include a circuit configured to recognize a traveling direction of the vehicle, or the traffic lane by optical flow information.

Camera controlling device15may include transmitter1515configured to transmit a command for Search Request as a message including a request for searching an object to a different camera assembly or other device connected to network4, via interface circuit1541.

Camera controlling device15may include transmitter1516configured to transmit a message including area information1528to be searched by a different camera assembly, to the different camera assembly or other device connected to network4, via interface circuit1541.

Camera controlling device15may include transmitter1517configured to transmit a message including characteristic value of an object to a different camera assembly or other device connected to network4, via interface circuit1541.

Camera controlling device15may include transmitter1518configured to transmit a message including the request for searching an object to a different camera assembly or other device connected to network4, via interface circuit1541, at any timing including following timings:

(1) A timing when output of image recognition device14shows that the object cannot be captured appropriately.

(2) A timing when output of image recognition device14shows that the object is far away.

(3) A timing when output of image recognition device14shows that the object may move to the outside of the angle of view soon.

(4) A timing when output of image recognition device14shows that the other device can capture the object more appropriately.

Camera controlling device15may include receiver1519configured to receive a command as a message including the request for searching the specific object from a different camera assembly1or a device connected to network4, via interface circuit1541.

Camera controlling device15may include receiver1520configured to receive a message notifying that the different camera assembly1detects the object, from other device connected to network4, via interface circuit1541.

Camera controlling device15may include receiver1521configured to receive a message including at least a degree of danger as to a falling object from other device connected to network4, via interface circuit1541.

Camera controlling device15may include receiver1522configured to receive a command for Search Request as a message including a request for searching an object from a different camera assembly1or other device connected to network4, via interface circuit1541.

Camera controlling device15may include receiver1523configured to receive a message including area information1528to be searched, from a different camera assembly or other device connected to network4, via interface circuit1541.

Camera controlling device15may include receiver1524configured to receive a message including characteristic value of an object, from a different camera assembly or other device connected to network4, via interface circuit1541.

Camera controlling device15may include circuit1525configured to determine a degree of danger as to an object by considering moving information of the object, and a size of the object in real space.

Camera controlling device15may include calculator1526to identify the degree of danger, configured to calculate the size of the object in a real space by performing coordinate conversion from a screen coordinate system to a global coordinate system, using at least controlling values of panning, tilting, or zooming as to the object, for example, memorized in PTZ device12or camera controlling device15. Calculator1526may include a calculator configured to calculate the size of the object in a real space by comparing with a specific object that size is known, the specific object being captured within an angle of view same as the object. Calculator1526may include an estimator configured to estimate the degree of danger at high when the size of the object in the real space is larger than an average size or a predetermined size of an object regardless of the material of the object, or when the object does not have movement generally regardless of the size of the object. Calculator1526may include an estimator configured to estimate the degree of danger at high when a ratio of a moving portion is smaller than a predetermined value. Calculator1526may include an estimator configured to estimate the degree of danger at low when the size of the object in the real space is smaller than the average size or a predetermined size of the object, and the object moves generally.

Camera controlling device15may include calculator1527configured to calculate area information1528including coordinate information on the global coordinate system as to a target object by referring to at least position information of camera assembly1, and control values for panning, tilting, or zooming to the target object, for example, memorized in PTZ device12or camera controlling device15.

Area information1528may include information to be searched by a different camera assembly, area information1528including at least one of the following position information, the position information of the camera assembly, the current position information of the object, or the position information that the object may exist on the global coordinate system, predicted by referring to a moving path of the object.

Camera controlling device15may include generator1529configured to generate the message including area information1528to be searched at a different camera assembly.

Camera controlling device15may include calculator1530configured to calculate coordinate information on the global coordinate system as to a target object by referring to at least position information of camera assembly1, control values for panning, tilting, or zooming to the target object, for example, memorized in PTZ device12or camera controlling device15.

Camera controlling device15may include selector1531to select an angle of view of imaging device11in a different camera assembly, the angle of view being expected a detection of the target object.

Camera controlling device15may include calculator1532configured to calculate the characteristic value of an object. For example, calculator1532may include a calculator configured to calculate the characteristic value of an object, including information as to a shape of the object, or parameters in the Hue, Saturation, Value (HSV) space.

FIG. 3shows a controlling flowchart in camera controlling device15of camera assembly1, or camera controlling device15of camera assembly2which includes a same configuration as camera assembly1, according to this embodiment. In following description, camera assembly1represents camera assembly2or other camera assemblies. Controller1501disclosed inFIG. 16is configured to control the overall of the camera assembly1including a change of operational mode.

Initially, in Step S21, camera controlling device15checks whether camera assembly1was received a command (digital message data) for Search Request as to a falling object from another camera assembly. When camera assembly1was received the command for Search Request as to the falling object from another camera assembly, a controlling procedure of camera controlling device15is shifted to Request mode S26. This situation may occur when the other camera assembly transmitted the command for Search Request as to the falling object to camera assembly1as a neighboring camera assembly, or camera assembly1as a camera assembly which is more likely to detect the falling object, because the other camera assembly lost the sight of the falling object due to movement of the falling object, although the other camera assembly had detected the falling object once.

When camera assembly1does not receive the command for Search Request as to a falling object from the other camera assembly, camera controlling device15has image recognition device14search a falling object within the angle of view currently being photographed by camera assembly1in Step S22.

In a case that image recognition device14detects a falling object, image recognition device14may retain approximate time when detecting the falling object, or the vehicle information including the vehicle's image, color, size, type or the number of license plate as clues to identify the vehicle that dropped the falling object in Step S23. Thereafter, the operational mode is shifted to Tracking mode in Step S24, explained usingFIG. 5. In Tracking mode, camera controlling device15tracks the falling object with determining a degree of danger as to the falling object, or determining whether the position of the falling object was changed or not.

On the other hand, if camera controlling device15determines that there is no falling object in Step S22, camera controlling device15controls PTZ device12so as to shift the angle of view of camera assembly1to another angle of view corresponding to another preset position (Step S25), then the controlling procedure of camera controlling device15returns to the beginning of the flowchart. In addition, as frequent changes of the angle of view may cause a mechanical trouble of PTZ device12, the loop periods from Steps S21to S25, Steps S21to S26, or the number of preset positions can be appropriately set.

FIG. 4is a flowchart indicating one embodiment of Request mode S26inFIG. 3. When camera assembly1is received the command for Search Request as to the falling object from the other camera assembly, a controlling procedure of camera controlling device15is shifted to Request mode S26as illustrated inFIG. 3.

The command for Search Request may include area information1528where an angle of view of imaging device11should be directed to search for the falling object, area information1528including at least one of the position information regarding the camera assembly which issued the Search Request, the position information as to the falling object, which was identified by the camera assembly which issued the Search Request, or a future predictive position of the falling object on the global coordinate system, which was identified by the camera assembly which issued the Search Request. For example, the command for Search Request may include information identifying the position of a camera assembly which lost or may lose the image of falling object. In addition, the command for Search Request may include information identifying the position of the falling object, performed coordinate conversion from the screen coordinate system to the global coordinate system. The command for Search Request may include information identifying a future predictive position on the global coordinate system, the future predictive position may be supposed from the movement direction of the falling object up to the time.

The command for Search Request may include information identifying the degree of danger as to the falling object, based at least in part on a value which was determined when the falling object was detected.

The command for Search Request may include information identifying the characteristic value of the falling object to detect or track the falling object continuously. The characteristic value may include information as to a shape of the falling object, or parameters in the Hue, Saturation, Value (HSV) space, which was identified by the camera assembly which issued the Search Request.

Firstly, in Step S31, camera controlling device15controls the PTZ device12so as to turn the angle of view of imaging device11of camera assembly1to the direction identified by the area information1528. For example, if area information1528identify the position of the camera assembly which issued the Search Request, as a first place that should look for the falling object, camera controlling device15controls PTZ device12so as to turn the angle of view of imaging device11to the direction of the camera assembly which issued the Search Request, specified by area information1528of Search Request. For example, the area information received from the requesting camera assembly may include the information regarding the angle of view of camera assembly1, which should direct initially. The information may include the position information of the requesting camera assembly, the position information of the falling object, or the future predictive position information of the falling object. The position information may convert to the preset position to control PTZ device12by referring to the corresponded preset position among the list of preset positions stored in PTZ device12.

In a situation that the camera assembly which is requesting camera assembly1to search for the falling object is far away even if it is the neighboring camera, the preset position of imaging device11may correspond to an angle of view for the farthest point (vanishing point) of the road that can be seen, and the zooming magnification of image device11may be set nearly maximal.

Thereafter, image recognition device14performs searching the falling object in Step S32. A method for searching the falling object in Step S32includes, for example, selecting an object having a characteristic value close to the characteristic value of the falling object notified by the camera assembly, as the object which should be detected or tracked continuously. A method for searching the falling object in Step S32also includes, for example, selecting an object detected by a method of template matching, compared with an image before receiving Search Request, as the falling object.

If image recognition device14cannot detect the falling object in current angle of view of camera assembly1, the angle of view of camera assembly1can be shifted to another angle of view corresponding to another preset position not yet tried to search the falling object (Step S33). Thereafter, camera controlling device15determines whether all preset positions have been tried to search for the falling object (Step S34). The term “All preset positions” are not limited to completely all preset positions installed in camera controlling device15, the term “all preset positions” can be limited to a specific group of preset positions, configured by a plurality of specific preset positions, at least each preset position substantially expected the existence of the falling object. The specific group of preset positions may exclude a preset position where the existence of the falling object is not expected. Image recognition device14may perform searching a falling object by referring to at least the angles of view, which are limited to the angles where the falling object are likely to exist.

When Image recognition device14cannot detect the falling object with respect to possible preset positions, camera controlling device15can issue a command (message) for Lost Alarm to network4, and then ends the procedure of Request mode.

When a falling object is detected in Step S32, image recognition device14may perform a tracking process for the falling object in appropriate time intervals, for example, in a timing when a picture frame generated in imaging device11is input into image recognition device14in Step S36. The tracking process may be an iterative process of updating the position information as to the falling object, by detecting an object having an image similar to the object previously detected as a possible falling object, in the vicinity of the object previously detected. The camera controlling device15may also include a circuit configured to track an object by referring to the characteristic value of an object which calculated by the camera assembly issued Search Request. Meanwhile, camera controlling device15may control PTZ device12so as to display the falling object at suitable magnification, positioned near the center of the screen of monitoring center3. In addition, information regarding at least a size or movement of the falling object may be retained for a predetermined time, for using the information to estimate the degree of danger later. This iterative process is continued until it is determined that the falling object cannot be captured within the angle of view in Step S43, or until it is determined that a collection process has been completed in Step S39.

In Step S37, as a result of performing the tracking processes, which contains repeating processes from Step S43or S39, to Step S36, camera controlling device15determines whether the position of the falling object was changed, in other word, camera controlling device15may determine whether the position of the center of gravity, estimated based at least in part on the captured image of the falling object was changed in Step S37. If the position of the falling object does not change, camera controlling device15may transmit the position information of the falling object to monitoring center3in Step S38, the position information including the information that the falling object was not moved. In monitoring center3, an operation for transmitting the position information of the falling object to a falling object correcting vehicle may perform appropriately. Furthermore, in order to provide the information regarding the falling object as driving support information, the information of the falling object may provide to a server or a managing device of Electronic Toll Collection System 2.0 (ETC 2.0), through the ETC 2.0 roadside equipment installed along the road.

Next, when camera controlling device15determines that the collection for the falling object was completed, or the object is not a falling object, for example, the object is determined as a lane marking sign or construction work machine etc. in Step S37, such information may be notified to the falling object correcting vehicle, the server, or the managing device of ETC 2.0 via monitoring center3, then the procedure of Request mode in camera controlling device15may end. On the other hand, while the falling object has not been collected, the controlling procedure of camera controlling device15returns to Step S36for continuing the tracking process.

In Step S37, if camera controlling device15determines that the position of the falling object was changed, camera controlling device15may determine the degree of danger as to the falling objects, based at least in part on the degree of danger, which is received from the camera assembly issued Search Request, or the degree of danger, which is independently estimated in Step S32. The degree of danger is described by referring toFIG. 6later. At a high degree of danger, camera controlling device15may issue a command for Accident Outbreak Warning to convey the possibility of outbreak of the accident to network4in Step S41, because camera controlling device15determines that the falling object might be moved from the position, where the falling object had been detected by the camera assembly which issued Search Request, to the current position by coming in contact with another vehicle. At a low degree of danger, camera controlling device15may issue the command for Visual Confirmation Warning to network4in Step S42. The command for Visual Confirmation Warning induces someone to confirm the object visually

Next, in Step S43, according to the result of tracking procedure as to the falling object in Step S36, camera controlling device15may determine whether the falling object is captured within the angle of view, in other words, determine whether or not the tracking procedure as to the falling object has successfully been completed in Step S36. While the falling object is captured within the angle of view, the tracking procedure as to the falling object is continued by branching into Step S36. Normally, in a case that the tracking procedure as to the falling object has ended in failure in succession several times, in other words, in a case that image recognition device14cannot find a position where an image of an object currently captured substantially matches the image of the falling object previously captured, camera controlling device15can determine the image of the falling object is out of the current angle of view. There is a possibility that the falling object might be blown off by coming in contact with another vehicle. Therefore, when the falling object cannot be tracked temporarily, it may be effective for capturing the falling object that the angle of view is shifted to a traveling direction of the vehicle little by little. Circuit1504supports the angle of view of imaging device11so as to shift to a traveling direction of the vehicle little by little to look for the object.

When camera controlling device15determines that an image of the falling object is out of the angle of view, in other words, determines that the falling object was moved to the outside of the angle of view, or the falling object cannot be captured because the position of the object is too far away, camera controlling device15can issue a command for Search Request as to the falling object to another device connected the network, the other device includes a different camera assembly, for example camera assembly2, then the procedure for Request mode in camera controlling device15ends.

The command for Search Request may include the information of the degree of danger which was estimated in Step S40, or determined by circuit1525or calculated by calculator1526described inFIG. 16.

The command for Search Request may include area information1528calculated by calculator1527, or coordinate information on the global coordinate system as to a target object calculated by calculator1530.

The command for Search Request may include the characteristic value calculated by calculator1532.

The destination camera which should receive the command for Search Request may be camera assembly2, which is located around camera assembly1. The destination camera assembly may also be located at adjacent to the camera assembly1. In addition, the destination camera assembly, which should receive the command for Search Request can be narrowed down to at least a camera assembly located near the traffic lane where the falling object was existed, the camera assembly being located in a travelling direction of the traffic lane.

In the flowchart ofFIG. 4, each Step S36to S43can be executed by changing the order of execution of each step appropriately.

FIG. 5is a flowchart indicating one embodiment of Tracking mode, identified as Step S24inFIG. 3. Firstly, camera controlling device15may determine a degree of danger as to the falling object in response to a transition to Tracking mode in Step S51. At that time, image recognition device14may determine the degree of danger by referring to a time series image (moving image) of the falling object, acquired from imaging device11. At a high degree of danger, a command for Warning with a warning level of “high” is issued in Step S52. At a low degree of danger, a command for Warning with a warning level of “low” is issued in Step S53.

The subsequent procedures may be similar to the procedures Step S36to S44in Request mode, identified inFIG. 4. In other words, the position of the falling object is determined in Step S54. If the position of the falling object did not change, the position information is transmitted to a falling object collecting vehicle via network4in Step S55. When the collection as to the falling object was completed, Tracking mode is ended in Step S56. When camera controlling device15determines that the falling object was moved, camera controlling device15may transmit a warning in accordance with the degree of danger, determined in Step S51or Step S57. When the falling object, having a high degree of danger was moved, the command for Accident Outbreak Warning may be issued, and when the falling object, having a low degree of danger was moved, the command for Visual Confirmation Warning to confirm the object by viewing may be issued. Camera controlling device15may continue to track the falling object even after issuing each command of warning in Step S58. In a situation that the image of the falling object can be captured within the angle of view of camera assembly1, the tracking procedure may be continued. When image recognition device14determines that the image of the falling object moved outside the angle of view, or predicts to move outside the angle of view considering a current movement path, camera controlling device15may transmit a command for Search Request as to the falling object to other camera, then the control procedure as Tracking mode in camera controlling device15is ended in Step S59.

FIG. 6is a diagrammatic illustration indicating one embodiment for determining a degree of danger as to the falling object. When image recognition device14detects a falling object, camera controlling device15may estimate a degree of danger as to the falling object based at least in part on the size of the falling object or the movement information of the falling object in Step S40, or S51. The size of the object may be identified with the size of the object in the real space. The size of the object in the real space is calculated by using the controlling values of panning operation, tilting operation, or zooming operation in PTZ device12, provided to image recognition device14for performing coordinate conversion from the screen coordinate system to the global coordinate system. The size of the object may also be identified by comparing with a specific object that size is known, the specific object being captured within an angle of view same as the object.

When it is determined that the size of the falling object in the real space is larger than an average size or a predetermined size of the falling objects, the degree of danger may be estimated to be high regardless of the material of the object, or an attribute other than the size of the falling object. For example, the average size or the predetermined size of the falling object can be identified statistically, and memorized in camera controlling device15initially. In addition, in a case that there is no movement regardless of the size of the falling object, it is assumed that the object is a heavy object, the degree of danger as to the object may be estimated to be high. In addition, in a case that it is determined that a part of the object is moving, the degree of danger may be estimated to be high if a ratio of the moving part is larger than a predetermined ratio identified statistically, because it is assumed that a heavy object and a light object dropped together such as a situation that a blue sheet is spread under the drum can. In addition, the degree of danger as to a small object moving generally may be estimated to be low because the small object is assumed to be something like plastic bag. For example, the degree of danger may be calculated and retained as a numerical value, and determined in Step S40or S51.

As a method for detecting a falling object to distinguish an object from a traveling vehicle, or a shadow of the vehicle, in Step S22, S32, etc. for example, a statistical learning method using characteristic values (feature quantities) can be used. Circuit142configured to detect an object in image recognition device14may include a circuit configured to detect a falling object to distinguish an object from a traveling vehicle, or a shadow of the vehicle, by using characteristic values under the statistical learning method. For example, an orientation map obtained by quantizing the intensity gradient image in the HSV space (Hue, Saturation, Value (Brightness) space model), a histogram thereof, or the like can be used as the characteristic value (feature quantity, Value corresponding to the characteristic of an object). Image recognition device14can estimate that an area composed mostly of gradients (edges) with only saturation is an area corresponding to a shadow, or an area illuminated with lights. Thus, image recognition device14can detect other image, which was newly captured, as a falling object. In addition, characteristic values (feature quantities) extracted from traveling vehicles can be learned as incorrect data as to the falling object in the statistical learning method using characteristic value (feature quantities).

In addition, in order to reduce a target area for processing, a conventional method such as the image processing method using a background difference can combine with the statistical learning method using feature quantities. In that case, image recognition device14may retain an updated background image while searching a falling object at each preset position, then use the updated background image for calculating a background difference when returning to the same preset position again.

In addition, as it is considered that a falling object may be moved against a direction traveling a vehicle, image recognition device14can use a flow or edge information included in the image as the characteristic values (feature quantities) for the purpose of detecting a moment of falling. After falling, an object is considered to be almost stationary on the road once, so it is possible to determine that an object detected sometimes at the same position within a predetermined time as a falling object.

As a method for tracking a falling object in Step S36, S58, etc. a method of template matching can be used. In the method of template matching, an image of a falling object is cut out from a frame in which a falling object is detected, the image of the falling object can use as a template image for tracking procedure. As the image of the falling object may be often intercepted by passing vehicles in front of camera assembly1, a processing load for tracking the falling object using the method of template matching for the whole angle of view may be increased. Therefore, in order to reduce the processing load for tracking the falling object, some method including a particle filter for the active search or the stochastic search, or Mean Shift for performing local search can be used. Movement of a falling object may be classified in an entire motion of the falling object (translational movement), and a partial motion of the falling object. The entire motion of the falling object can be detected by using the method of template matching or the like, the partial motion of the falling object can be detected, for example, by a difference of images as to the falling objects being aligned, in other words, a residual value as a result of template matching, or by the processing of Time Space Fast Fourier Transform (Time Space FFT). The information as to both the entire and partial motions of the falling object can be used for estimating a degree of danger as to the falling object. In addition, the processing of Time Space FFT can be performed in two-dimensional space by converting image data in the two-dimensional coordinate system into column vectors. It is desirable that a value indicating a degree of the partial motion be normalized so as not to depend on the appearance or the size of the falling object.

FIG. 7,FIG. 8,FIG. 9, andFIG. 10schematically illustrate embodiments of operations of the falling object detecting system when a truck drops an object.FIG. 7is a diagrammatic illustration indicating one embodiment of the operation in a case that a falling object collided with a vehicle, the collided vehicle drags the falling object without noticing the falling object.FIG. 7shows the transitions of the relation between a vehicle and an object, according to progress of the time illustrated from the left side to the right side. For example, a light object such as a blue sheet may cause this kind of situation. Firstly, when image recognition device14detected a falling object, an operational mode of camera controlling device15was switched to Tracking mode as shown Step S22to S24inFIG. 3, and the position of the falling object was monitored as illustrated in most left side ofFIG. 7. In a case that a following vehicle came in contact with the falling object and the falling object was dragged by the following vehicle, camera controlling device15might issue the command for Accident Outbreak Warning as illustrated in the second position from the left side inFIG. 7. When the vehicle coming in contact with the falling object dragged the falling object, the camera which detected the falling object might lose sight of the falling object. At this time, camera controlling device15might transmit a command for Search Request as to the falling object to a neighboring camera as illustrated in the middle position inFIG. 7, but if the falling object could not be detected again by even the neighboring camera as illustrated in the fourth position from the left side inFIG. 7, the operational mode of each camera might return to Search mode as illustrated in most right side ofFIG. 7. The vehicle dragging the falling object might be identified by referring to the image, which captured at the timing when the command for Accident Outbreak Warning was issued.

FIG. 8is a diagrammatic illustration indicating one embodiment of the operation in a case that a falling object is moved to another position due to a collision with a vehicle.FIG. 8shows the transitions of the relation between a vehicle and an object, according to progress of the time illustrated from the left side to the right side. The position of the falling object is changed when the falling object collided with a vehicle. Camera controlling device15may issue the command for Accident Outbreak Warning, and the falling object is tracked as illustrated in the middle position inFIG. 8. Next, if there is a vehicle being stopped around the accident site, the vehicle can be also detected as an abnormally stopping vehicle as illustrated in most right side ofFIG. 8.

FIG. 9is a diagrammatic illustration indicating one embodiment of the operation in a case that a falling object is moved to another position due to a collision with a vehicle, and a vehicle has stopped in the monitoring range of another camera.FIG. 9shows the transitions of the relation between a vehicle and an object, according to progress of the time illustrated from the left side to the right side. When a falling object collided with a vehicle, the command for Accident Outbreak Warning is issued, and camera assembly1tracks the falling object as illustrated in the second position from the left side inFIG. 9. In camera2, when a stopped vehicle is detected, camera controlling device15of camera2may assume the stopped vehicle to be an abnormally stopped vehicle in consideration of a situation that the command for Accident Outbreak Warning was received from camera assembly1as illustrated in most right side ofFIG. 9.

FIG. 10is a diagrammatic illustration indicating one embodiment of the operation in a case that camera controlling device15cannot detect the falling object again because the falling object is moved to the outside of the road due to a collision with a vehicle.FIG. 10shows the transitions of the relation between a vehicle and an object, according to progress of the time illustrated from the left side to the right side. When the falling object collides with a vehicle, the command for Accident Outbreak Warning is issued as illustrated in the second position from the left side inFIG. 10. If even the neighboring camera cannot detect the falling object again, the command for Losing Sight Warning is issued as illustrated in the third position from the left side inFIG. 10, the operational mode of each camera may return to Search mode as illustrated in most right side ofFIG. 10. At this time, it is desirable that camera controlling device15transmits the information indicating some kind of falling object having been lost from sight of someone managing roads or facilities outside the jurisdiction.

As described above, this embodiment discloses an autonomous distributed cooperative system so as to track a falling object, detected by a plurality of cameras in cooperation with each other. In such a system, monitoring center3does not need a large-scale facility for intensive processing as long as it can receive or cancel an alarm, and it is easy to introduce it. In addition, Search request as to the falling object performed in Step S59is not limited to being performed from the camera, but may be performed from a host system such as a traffic control system that grasps the occurrence of traffic congestion or accident.

The scope of the disclosure can include the configuration of the embodiments explained so far, but is not limited thereto. For example, image recognition device14may detect road damage, an obstacle, or a vehicle stopped urgently etc. by applying the recognition/learning techniques disclosed above appropriately. In addition, in order to analyze the image which has not become high resolution on imaging device11, contrast correction, fluctuation correction, shake correction, super-resolution processing, etc., may be performed as preprocessing. Instead of measuring the speed of the vehicle, the number of vehicles passing per predetermined time may be measured, and the congestion start point may be searched from the change.

In addition, image recognition device14can recognize the traveling direction of the vehicle, or the traffic lane by optical flow techniques or the like, and can issue warnings with different degrees of danger levels in accordance with the positional relationship between the traffic lane and the falling object. For example, in a case that a falling object, previously detected in the roadside, except for the traffic lane, has currently moved to the traffic lane, a warning indicating a higher degree of danger may be reissued.

In addition, image recognition device14is not limited to being built in camera assembly1itself.FIG. 11shows an example of a falling object detection system, including image processing apparatus5having a function of image recognition device14, image processing apparatus5being installed in the latter stage of the network4in the case that image recognition device14cannot be installed in camera9. Normally, electrical equipment such as a junction box may be provided at the connection point between camera9and network4to supply electrical power to camera9etc., Therefore, image processing apparatus5can be provided therein.

REFERENCE SIGNS LIST