Source: https://patents.google.com/patent/CN105122270A/en
Timestamp: 2020-01-29 19:14:40
Document Index: 546814282

Matched Legal Cases: ['arts 225', 'arts 240', 'arts 240', 'arts 225', 'arts 225', 'arts 225', 'arts 240', 'arts 240', 'art 413', 'art.\n11']

CN105122270A - Method and system for counting people using depth sensor - Google Patents
Method and system for counting people using depth sensor Download PDF
CN105122270A
CN105122270A CN201380070934.6A CN201380070934A CN105122270A CN 105122270 A CN105122270 A CN 105122270A CN 201380070934 A CN201380070934 A CN 201380070934A CN 105122270 A CN105122270 A CN 105122270A
CN201380070934.6A
CN105122270B (en
2012-11-21 Priority to US13/683,886 priority Critical patent/US10009579B2/en
2012-11-21 Priority to US13/683,886 priority
2013-11-19 Application filed by 派尔高公司 filed Critical 派尔高公司
2013-11-19 Priority to PCT/US2013/070688 priority patent/WO2014081687A1/en
2015-12-02 Publication of CN105122270A publication Critical patent/CN105122270A/en
2019-03-08 Publication of CN105122270B publication Critical patent/CN105122270B/en
A sensor system according to an embodiment of the invention may process depth data and visible light data for a more accurate detection. Depth data assists where visible light images are susceptible to false positives. Visible light images (or video) may similarly enhance conclusions drawn from depth data alone. Detections may be object-based or defined with the context of a target object. Depending on the target object, the types of detections may vary to include motion and behavior. Applications of the described sensor system include motion guided interfaces where users may interact with one or more systems through gestures. The sensor system described may also be applied to counting systems, surveillance systems, polling systems, retail store analytics, or the like.
Use the method and system of depth transducer counting people
The application is the U. S. application the 13/683rd submitted on November 21st, 2012, the continuation application of No. 886.
The entire teachings more than applied for is incorporated in herein by reference.
The accurate metering of the people in scene is benefited from many application.Some in these application can combinationally use with heating ventilation and artificial atmosphere (HVAC) control system, video monitoring system, retail trade system and other field.Such as, the accurate metering of people can provide statistical information to judge their advertising results for retailer, and other application can use counting to detect trailing of security checkpoints place.Municipality also can use accurate scene population to count the quantity determining the visitor at specific facilities place.Out of true can be caused to count by using visible spectrum and thermal camera counting people.
When shade presents similar shape and size, visible spectrum camera is difficult to detect the object in scene.Thermal camera depends on temperature variation to determine or detects the motion in scene.When temperature is identical with the temperature of mobile object or close to time, thermal camera detection mobile object in encounter problems.The application accurately counting the depth transducer of people is described.Depth information may be used for deriving head sizes, and uses this information, determines the character of the detection window of head.Other object detecting methods application (one or more) multiple dimensioned (scale) detecting device.By removing multiple scale detecting via counterbody Size calculation, described technology is detected by single scale improves verification and measurement ratio significantly, provides the increase precision having and reduce computational complexity.
Embodiment described herein comprises a kind of method for counting people, it obtains depth data from depth transducer, in depth data, distinguish foreground object from background object, and from foreground object, determine the foreground object of the reference model mating target object based on depth data.
In one embodiment, depth transducer can be mounted to the vertical view providing scene.Depth transducer can use to determine that scene population counts further from the foreground object distinguished with visible spectrum camera combination.According to an embodiment, sensor (degree of depth or visible spectrum) can follow the tracks of the one or more foreground objects being confirmed as mating reference model from foreground object between frames.Embodiment also can be included in when predetermined hold-time section is not moved in scene and scene population counting is reset to zero.The method of counting people also can comprise and obtains visual picture from visible spectrum camera and use the information visible spectrum camera to check the precision of counting.
Information from visible spectrum camera can comprise movable information, time attribute, position and feature.In one embodiment, metadata can be sent to data center by sensor.Method can also comprise use depth transducer imaging scene and the quantity of speaker, statistics or supplementary data.In one embodiment, from foreground object, determine the reference model of background object coupling head part.
Embodiment can comprise or use the computer-readable medium with programmed instruction stored thereon, described programmed instruction can load and can be performed by processor, and when being executed by a processor, processor is caused to obtain depth data from depth transducer, from depth data identification foreground object, and applied for machines study application program is determined with the classification generating foreground object, and classification is kept to determine counting.Programmed instruction can also cause processor from depth data subtracting background information to identify foreground object, view data is obtained from visible spectrum camera, analysis of image data to produce image analysis result, and combination from the information of image analysis result produced and depth data information to increase the precision of identification foreground object.In one embodiment, the embodiment of machine learning application program can comprise neural network, support vector machine and/or clustering technique.Programmed instruction can learn application program with training dataset training machine and in the optimization of machine learning application program, select one or more feature to produce selectional feature vector.
A kind of system for counting people can comprise receive depth data depth transducer, communicate with depth transducer, the storer of storage depth data.System can also comprise the processor with memory communication, execution of program instructions.Programmed instruction can be configured to from depth data subtracting background information to distinguish foreground object, and applied for machines study application program is determined with the classification generating foreground object, and keeps classification to determine counting.Processor can also be configured to from camera receiving video data and generation classification is determined, use the video data from camera to increase the precision of machine learning application program.
Although those of ordinary skill in the art will understand multiple wave filter, Gaussian filter can be applied to the background information from depth data by embodiment.System also can produce identification selection feature with the depth characteristic collection analyzed from the depth data of depth transducer and produce identification selection feature to analyze from the video features collection of the video data of camera.In one embodiment, processor can be configured to feature set is applied to depth data to determine depth sorting and visual classification.In addition, system can be determined and upgrade classification to determine counting with the classification of the information calculating foreground object from depth sorting and visual classification.
Here is the description of embodiment.
Fig. 1 illustrates the schematic diagram of an embodiment of the system counted based on depth transducer 110.As shown in fig. 1, the system 100 counted based on depth transducer 110 is provided.System 100 comprises scene 105, and scene is the scope of the detectable vision of sensor/camera.As shown in scene, one or more people can be positioned at such as depth transducer 110 and the camera of (one or more) visible spectrum camera 115 or one or more visual fields 112,117 of imaging device.Visible spectrum camera 115 can comprise respectively with depth transducer 110 and associates visual field 117,112.Depth transducer 110 and/or camera can associate with data storage device 131, and described data storage device can be local, such as, may have access on public network or via wide area network 120.
Depth transducer 110 is from one or more scene sampling depth data.Depth data can be processed to remove background information and the focusing provided the object in scene with motion.In one embodiment, be detected as the object in scene with motion and be sent to one or more parts to analyze.The result analyzed can comprise and object to be detected is categorized as classification, such as people or inhuman.The error that analysis based on depth data is not generally vulnerable to the error that associates with the wrong report caused due to the temperature variation such as shown in dsc data or associates due to wrong report that such as shade shown in visible spectrum image causes.So depth data can return classifies more accurately, be wherein focus with the described easy acceptabilily reporting the error associated by mistake.In addition, by the sorter that will be supplied to through the data set of filtering and the selectional feature of target object for counting people, depth data detects can reduce computational complexity.
Continue with reference to figure 1, data storage device 131 can be the data storage device of any suitable type, such as, comprise storage chip, hard disk drive, USB flash disc drives, remote accessible database, cache memory and/or analog.Data storage device can communicate with depth transducer 110 or camera 115 or make it communicate with communication network 120 and can comprise wired and wireless network.
Communication network can be following any one or combination: direct interconnection; Internet; LAN (Local Area Network) (LAN); Metropolitan Area Network (MAN) (MAN); Operation task node (OMNI) on internet; Security customization connects; Wide area network (WAN); Wireless network (such as, use agreement, such as but not limited to WAP (wireless application protocol), and I pattern; And/or analog).Network interface can be regarded as the particular form of input/output interface.In addition, multiple network interface may be used for engaging with various communication network type.Such as, multiple network interface may be used for allowing to communicate in broadcast, multicast and/or unicast networks.
The data center 125 communicated with network operable is also shown in FIG.Data center can receive the data from depth transducer and/or visible camera.In one embodiment, the view data of one or more predetermined format can be transferred to data center 125 via network 120 by visible camera.Similarly, depth data can be sent to data center 125 so that process and/or analysis by depth transducer.In one embodiment, data center 125 can comprise one or more computer systems (client/server) (not shown) be distributed on network.These computer systems can be responsible for receiving, process, analyze, transmit and/or storing: view data, video data, depth data and/or class likelihood data.In one embodiment, data center 125 can comprise, store, instantiation, operation and/or the described in detail further below detecting and alarm of access (such as, by application programming interface (API)).
Fig. 2 is calcspar, and the software architecture according to an embodiment is shown.As shown in Figure 2, detecting and alarm 205 can comprise motion detecting component 210, video analysis parts 225, depth data analysis component 230 and counting deduction parts 240.Detecting and alarm 205 and network 220, data storage device 231 and/or server 265 can operation communications.Without limitation, detecting and alarm can comprise optional feature, such as, and API, memory unit, cloud service parts or analog.
Motion detecting component 210 can comprise one or more methods using the data received from one or more sensor device to carry out object of which movement detection.In one embodiment, motion detecting component is from the depth data inspected object received from data center.In addition, motion detecting component can comprise script or program with from the view data inspected object received from visible spectrum camera.Those of ordinary skill in the art will recognize that the data of other types also can receive from motion detecting component and be regarded as dropping in the scope of described embodiment.One type of the detection performed by detecting and alarm is the detection of motion.
Motion detection can identify where people appears in scene.Background model can be set up to detect motion, as Gauss model, gauss hybrid models and/or nonparametric background model by background subtraction.Can relative to the range calibration depth data from depth transducer to measure three-dimensional (3-D) information of object, as the shape characteristic (such as, compared to the seat of the seat at the top of the backrest at seat or the leg relative to the people at the top of the head of people) from the assigned address on object.Method of counting can based on depth segmentation and head detection.In one embodiment, depth segmentation initially can determine whether there is people in scene.Such as, if foreground object size is greater than a people, then can apply head testing process to measure the quantity of the head in scene.
As shown in the figure, the successive frame of video flowing 232 can be transferred to data storage device 231 or transmit from the node (not shown) network 220.Data storage device 231 can process video flowing 232 and analysis/trace information 236 is turned back to detecting and alarm 205.In one embodiment, motion detecting component 210 can combine in many ways, combines and/or distribute so that develop and/or issue.Such as, motion detecting component 210 can integrally or partly be stored in data storage device 231 or server 265.For this reason, (as required) can be integrated or dynamically be loaded in motion detection of code storehouse statically to data storage device 231 or server 265.
In one embodiment, detecting and alarm 205 can provide from the node receiver, video flow data 232 network 220 and from the one or more parts associated with detecting and alarm and analyze/trace information 238.The degree of depth/view data 244 also can be transferred to server 265 by detecting and alarm 205.Server can be positioned at described place of data center herein.Server can store and perform the one or more parts associated with detecting and alarm 205.In one embodiment, the analysis/trace information 242 based on the degree of depth/view data received can be transferred to detecting and alarm 205 by server.
In order to supplementary testing process, one or more lines of stumbling with optional direction can be arranged in visual field to measure the quantity entering and leave the people in region.In one embodiment, object to be detected may cross over line of stumbling, and can be sent to data center about the metadata (such as, stumble line counting, time, position etc.) of this event.
As mentioned above, detecting and alarm can carry out head detection process based on depth data.Depth data can be stored in two-dimensional array.Depth data can be calibrated to measure the physical location of the object in real world.After the calibration, depth data can provide the distance between depth transducer and object.From these data, the size of head to be detected can be determined.
In order to realize the size determining head, the distance from camera to head center can be expressed as X and picture traverse is expressed as M pixel by a formula.The angle that depth transducer covers can be represented by α.The physical extent that can cover at the distance sensor of X rice can be calculated as:
w = 2 * X t a n ( &alpha; 2 )
From this scope, can derive based on distance in the head sizes of pixel.Assumed average head radius is r rice, then it can be expressed as with pixel:
R = r M 2 * X t a n ( &alpha; 2 )
In one embodiment, by non-maximum suppression method, testing process can check that the moving window of specified size is with head.Apply non-maximum suppression to overlap to avoid multiple head.
Machine learning component can comprise one or more machine learning application program with based on the Data Detection object received from one or more sensor device.In one embodiment, machine learning component comprises for the behavioral value method from training data inspected object.In one embodiment, learning application program can be random forest, self-adaptation enhancing (adaboost), neural network, support vector machine or analog.Random forest is that the combination of tree-shaped fallout predictor makes each tree depend on the value of random vector of separately sampling and all trees in forest have same distribution.When the quantity of the tree in forest becomes large, the vague generalization error convergence of forest is to the limit.The vague generalization error of the forest of tree classifier depends on the intensity of independent tree in forest and the correlativity between them.
Self-adapting enhancement method gets training set (x1; Y1) ... (xm; Ym) as input, wherein each x ibelong to certain territory or instance space X, and each mark y iin certain label sets Y.Self-adaptation strengthens takes turns several t=1 a series of ... repeatedly call in T and specify weak or basic studies function.Self-adapting enhancement method keeps distribution or the set of the weight on training set.The weight of this distribution on the training example i on wheel number t can be expressed as D t(i).Initially, equally arrange weight, but take turns on number each, the weight increase of the example of mis-classification makes weak learner be forced to the difficult example focused in training set.The work of weak learner finds to be suitable for the D that distributes tweak hypothesis.The goodness of weak hypothesis is by its error measure.In one embodiment, the distribution D that can train thereon relative to weak learner tmeasuring error.Weak learner can use the weight D on training example t.In one embodiment, can according to D tthe subset of sample train example, and these (non-weighting) resampling examples may be used for training weak learner.
Neural network comprises simple process element (nerve), highly interconnected, the simple scalar message between treatment element and adaptive interaction.In one embodiment, neural network can be disposed for application-specific, such as, and pattern-recognition or Data classification.Neural network is learnt, organizes and represent for machine provides based on the data adaptive provided for training from the data of example, in real time operation and the ability being comprised fault tolerance by redundant information coding.Support vector machine (SVM) is an example of the machine learning application program for Data classification.SVM classifier generally will find separating hyperplance with maximum boundary in data-mapping to high-dimensional space.In one embodiment, machine learning component receives the expression of the object to be detected in depth data or image and is mapped to expression as on people or inhuman lineoid.
One or more machine learning application program can combinationally use with one or more clustering technique.Clustering technique generally comprises the set associative of object in one or more groups (classes).In one embodiment, the data received from detecting and alarm can by cluster to help to determine whether object just tracked or to be tracked should be classified as people or inhuman.Such as, candidate data object can be encoded for head detection process by motion detecting component 210.Candidate data object or target object can be sent to machine learning component (such as, SVM classifier) to classify by motion detecting component 210.Machine learning component can return the classification of target object, the additional function in its detection trigger engine, and such as, counting infers that parts are with more new scene population counting.
In order to calculate feature rapidly, an embodiment can use the volume image quantizing gradient orientation to carry out speed-up computation.In one embodiment, the incremental learning program proposed comprises: manually mark (one or more) head from training depth image; The directed concentration gradient (HODG) of application histogram extract head feature as positive feature and the feature extracting other positions as negative feature.Iteration variable initially can be set to zero by iterative loop, such as, arrange i=0.Iterative process can comprise applied for machines study application program (such as, backpropagation (BP) neural network) with appointment training dataset i Study strategies and methods.In one embodiment, process can comprise with labeled test image set i test learning model.Learning process can comprise and misclassification feature (wrong report) be added training data and concentrate.Iteration variable can increase progressively, such as, arrange i=i+1, and program can some in repetitive process until the stable and error in classification of learning model is lower than threshold value.
Depth data analysis component 230 can combine effectively calibrates head feature, swift nature extraction and efficient sorting technique to obtain real-time head detection.A specific advantages of the alignment features proposed is that roughly head sizes is known and need not detects in similar multiple dimensioned additive method (gradient (HOG) object detection as directed in histogram).Although multi-scale method can be realized, the computational complexity of testing process can be reduced by the single scale method of head detection.
Depth data analysis component 230 can generate histogram from the view data received and depth data.In one embodiment, the directed gradient (HOG) of histogram can from reception data genaration.In addition, the directed concentration gradient (HODG) of histogram can also be generated.In one embodiment, head detection can based on the HODG generated.The generation of the gradient orientation in the selected window of HODG technology counting.In one embodiment, gradient has symbol with the amplitude of the actual direction and/or gradient that indicate gradient.Use signed gradient, can infer that gradient is upwards or downwards.First window can be divided into unit, such as, and 3x3,4x4 and/or analog.In one embodiment, can in unit compute gradient directed.In one embodiment, as the overall function of weight application concentration gradient.The histogram of weighted gradient orientation may be used for representing unit.The histogram of weighting concentration gradient orientation can be combined to form the proper vector for head detection.
Depth data analysis component 230 can be selected and analytical characteristic from view data and/or depth data.In one embodiment, the set of the feature being used for analysis of image data is selected.Such as, when receiving the degree of depth/view data, it can be characterized to identify and/or extract the feature associated with depth data and/or view data.Characterization from the frame of image and/or depth data can depend on the type of classification.Such as, the classification of non-human object can from the frame identification of the degree of depth and/or view data and the different sets extracting feature.When identification or when extracting suitable feature, they can compare with the training dataset of just classifying and report preliminary making by mistake.From image and/or depth data feature relatively can determine whether the feature extracted mates the feature from housebroken data set by allowable depth data analysis component.Counting in the embodiment of people, the generation that people's type relatively can be caused to mark is determined to help classification.
If more than one people enters surveyed area, then depth data analysis component 230 can help head tracking.In one embodiment, head tracking data are sent to count and infer that parts 240 are acted on behalf of with notifications count by detecting and alarm.Method of counting can comprise the background model of study scene; The foreground pixel closer to depth transducer is detected by predetermined threshold use background subtraction; Use link analysis to be marked by foreground pixel and disconnect in object; Or calculate the size of object.
If the size of object is greater than the size of a people, then system can be applied head detection and can find more than one head to check whether.In one embodiment, (one or more) head detected can be tracked as independent object.In addition, line of stumbling can be employed to detect whether have any object traverse they.Can record to cross over and detect for entering/leaving counting.Such as, the difference " entered " between summation that the summation counted and " leaving " count is the quantity of the people in appointed area (such as, interested volume).Additional motion/vision sensor can be installed in a room to obtain further motion accuracy of detection and for preventing error count.If interior sensor does not detect any motion in the given time, then the current count in room can reset to zero (automatically or by the input of people manually).
Video analysis parts 225 can comprise the one or more parts being exclusively used in reception, process and analyzing the data received from camera.According to an embodiment, video analysis parts 225 can combinationally use with depth data analysis component the precision increasing detecting and alarm.Depend on the type of detection, depth data analysis component 230 can combinationally use the precision of the detection increased based on camera with video analysis parts 225.
Depth data analysis component 230 comprises the one or more parts detecting the object in scene based on the depth data received from data center and/or depth transducer.Depth transducer " can not see " shade and the reflection that can realize object segmentation in visible camera system.Depth transducer can supplement the dsc data from thermal camera, and reason is the temperature variation that the motion not realizing depth transducer detects.Detecting and alarm 205 can assign the task to depth data analysis component and miscellaneous part to help to count people.Component set can perform based on depth information the programmed instruction associated with head detection.
In one embodiment, depth data analysis component can work together with execution of program instructions with miscellaneous part (such as, motion detecting component) thus detect based on depth information Executive Head.The head detection performed can transfer data to counting and infer that parts are to help to keep scene population counting.In one embodiment, depth data can be calibrated to measure the physical location of the object in real world.After the calibration, depth data can provide the distance between depth transducer and object.Head detection process can combine effectively calibrates head feature, swift nature extraction, efficient sorting technique to obtain real-time head detection.An aspect of (one or more) alignment features proposed is to pre-determine the estimation head sizes of scene or visual field.Depth data analysis component can use slip two time scales approach to determine whether object to be detected mates appointment reference model.Such as, slip two time scales approach can determine the object that detects or object features whether in the boundary of slip yardstick.
Counting infers that parts 240 can keep the counting of the object to be detected in scene.In one embodiment, server 265 is from detecting and alarm 205 count pick up information 244.Then server 265 can derive the set of last count number, the quantity of such as, people at the appointed time in visual field, the quantity of the people within a period of time in visual field, etc.In the embodiment that position (such as, having the buildings of multiple entrance entrance hall, floor, staircase etc.) is monitored by detecting and alarm more than, the combined count of the subset of position that server can be derived buildings or be associated with buildings.Such as, when object enters and leave scene, counting infers that parts can keep scene population to count.In order to addressing is at similar time and the individuality that appears in physics approximated position in scene, counting infers that parts 240 can receive additional data from stumble line or the infrared motion detector for detecting motion.Counting infers that parts can make scene population enumeration data relevant to received or based on the information from depth sensor, visible spectrum camera and/or line of stumbling activity data.
In one embodiment, data center can store the one or more parts associated with detecting and alarm, receives and storage figure picture and/or depth data, and the Processing tasks that distribution associates with detecting and alarm.In one embodiment, data center receives image and/or depth data from one or more data storage device.Data center can comprise the detecting and alarm server be responsible for from the data identification received and extraction feature.Other distributed elements also can communicate with detecting and alarm thus help when classifying to determine whether the feature received from (one or more) sensor and/or (one or more) camera drops on one or more classification.
Fig. 3 illustrates the process flow diagram of the method based on sensor counting people according to an embodiment.As shown in Figure 3, method 300 comprises and obtains depth data from the depth transducer of the vertical view being mounted to provide scene.The vertical view of scene can be directly vertical (90 °) view from top.Vertical view also can be the view of roughly 90 ° (permissible errors 30 °) from top.Method also can distinguish foreground object 315 from background object 310 in depth data.One or more background subtraction technique also may be used for distinguishing foreground object from background object.In one embodiment, method can be determined to specify foreground object 315, based on depth transducer Data Matching head reference model from foreground object.
Fig. 4 is process flow diagram, and the method 400 counted based on depth transducer and visible spectrum camera according to an embodiment is shown.Method for counting can obtain depth data from depth transducer and obtain visual picture 405 from visible spectrum camera.In one embodiment, method can obtain from visible spectrum camera the information 409 comprising motion, time attribute, position and feature.Method for counting people can comprise metadata is sent to data center 409.Such as, metadata can comprise the information about the position of camera, timestamp, associated physical address (such as, building name).Method can distinguish foreground object 411 from background object in depth data and visual picture.In one embodiment, counting people can determine prospect from foreground object, and based on the reference model of depth data coupling target object, wherein target object is head part 413.The method of counting people can from foreground object determination scene population counting 415.In one embodiment, method can comprise use depth transducer imaging scene 417 and the quantity of speaker, statistics or supplementary data 419.
Such as, statistics can comprise the information of the averaging time that the frequency that occurs in scene about object or object occur in scene.Supplementary data can comprise the information about the non-inspected object in scene.Such as, the information describing the product that the retailer that is arranged in scene provides can be comprised about the information of the non-inspected object in scene.Supplementary data can comprise the determination of the whether attracting attention of appointed product of ad-hoc location.The determination whether appointed product attracts attention can comprise behavioral value, and such as, people stops predetermined amount of time to watch non-inspected object.
Method 400 for counting can check motion.Such as, additional motion/vision sensor can be installed in a room to obtain further motion accuracy of detection and for preventing error count.If interior sensor does not detect any motion in the given time, then the current count in room can reset to 0 423 (automatically or by the input of people manually).If motion detected, then method 400 can continue to obtain depth data and visual picture.
Fig. 5 comprises the sample frame of depth data.As shown in Figure 5, the first frame 505 illustrates the background model according to an embodiment.Sample frame 505 illustrates some white pixel, and wherein depth information may be unavailable and can be deducted by detecting and alarm.
In Figure 5 sample incoming frame 510 is also shown.Incoming frame 510 is the examples how mobile object can be represented by depth data.The center of frame 510 illustrates that what can yes or no people, and such as, eye can infer that the object at the center of image has the outward appearance at the top being similar to two shoulders and head part.Depth data in frame 510 does not also experience background subtraction, and the non-moving object wherein in frame is removed from frame at least in part.The mobile object that frame 520 detects after background subtraction is shown.As shown in Figure 5, compare with the object in frame 505 and 510, the mobile object detected in frame 520 has the more sharp contrast with background.As shown in sample frame 520, the object on the left side of the potential target at the center of image can be inferred as animal or pet by eye.
In one embodiment, mobile object is detected from background subtraction, as mentioned above.The spatial variations of amount can be represented by slope.Gradient represents steepness and the direction on this slope.In one embodiment, the histogram (HODG) of directed concentration gradient describes local objects outward appearance in depth image and shape by the distribution in (one or more) concentration gradient direction.Can by image is divided into the little join domain that is called as unit and for the gradient direction of the pixel in each unit compilation unit histogram and obtain this embodiment.Then these histogrammic combinations represent descriptor.Gradient calculation can apply 1-D centering, some discrete differential template in level and vertical direction.Particularly, with the depth data of following filtering core filtering image: [-1,0,1] and [-1,0,1] t.
Histogram parts can generation unit histogram.Such as, each pixel in unit can be weighted ballot based on the value found in alternative calculating to the histogram passage based on orientation.Unit self can be rectangle or radial in shape, and histogram passage is dispersed on 0 to 360 degree equably.About ballot weight, pixel contribution can comprise the function of gradient amplitude or gradient amplitude.
Frame 525 and 530 is frames that head window and respective depth gradient amplitude and orientation are shown.Frame 525 is sample frame that the head window not having respective depth gradient and amplitude is shown.Frame 530 is divided into the head window that four take advantage of Unit four.As shown in frame 530, the concentration gradient amplitude on peripheral unit and orientation have than being arranged in the concentration gradient amplitude of four internal elements and directed larger length.Compare with orientation with the shorter amplitude of the difference of the distance at the top represented from the side of head to head, this difference may represent the larger distance from the side of head to the shoulder adjacent cells.In one embodiment, each unit can based on gradient amplitude and directional structure histogram.Histogram can combine to form the proper vector describing head window.
Fig. 6 is calcspar, and the depth transducer system 600 according to an embodiment is shown.As shown in Figure 6, depth transducer system 600 comprises first sensor 605, second sensor 610, the 3rd sensor 615 and data center.In one embodiment, depth transducer can be installed downwards to detect motion and count the quantity entering and leave the people in region.Depth transducer can comprise the infrared laser projector with monochromatic complementary metal oxide semiconductor (CMOS) (CMOS) sensor combinations, and it catches 3D video data under environment light condition.
Sensor can comprise digital signal processor (DSP) and/or field programmable gate array (FPGA).DSP/FPGA be configured to treating depth data computing unit and can carry and be embedded with depth transducer by plate.Data center can be sent to via network processes and data mining by the count information of DSP/FPGA process.Depending on specific embodiment, the feature of depth transducer system can be obtained by realizing microcontroller.And in order to realize some feature of depth transducer system, some feature embodiments may depend on built in items, as: special IC (ASIC), DSP, FPGA or similar embedded technology.Such as, depth transducer system (distributed or other mode) can realize via microprocessor and/or via built in items; Such as, via ASIC, coprocessor, DSP, FPGA or analog.Alternatively, depth transducer system some embodiments can be configured to and for obtain various feature or signal transacting built in items realize.
In one embodiment, multiple depth transducer can be arranged on data collecting point, the door entering room entering as monitored and leave.If enter room have multiple door, then can for each fitting depth sensor.In one embodiment, the metadata about count information can be sent to data center to process further.The metadata of video analysis result can comprise the description of object and event.Metadata also can be the information about event, such as but not limited to object merging, division, appearance, disappearance etc.
Fig. 7 is the calcspar of the embodiment of the platform 700 that illustrates that people counts (PC).In this embodiment, PC platform may be used for assembling, process, store, search for, serve, identify, instruct, generate, mate and/or being convenient to and computer interactive.Typically, can be people and/or other system user can splice information technological system (such as, computing machine) so that information processing.Then computing machine uses processor processes information; Such processor can be called as CPU (central processing unit) (CPU).A kind of processor of form is called as microprocessor.CPU uses telecommunication circuit transmission to be used as the binary coded signal of instruction to allow various operation.These instructions can be operation and/or data command, and it comprises and/or the various processors of quoting storer (such as, register, cache memory, random access memory etc.) may have access to and other instruction and datas in operable area.Infosystem may be used for collecting data so that retrieval later, analysis and operation, and it can be promoted by database program 737.
In one embodiment, PC platform can be connected to entity and/or and entity communication, such as but not limited to: one or more user is from user input device (such as, flash memory/SD/SSD); Peripherals, such as watch-dog or camera 701; Optional cryptographic processing apparatus; And/or communication network 720.Network is believed to comprise interconnection and the interoperability of client, server and intermediate node in image topology usually.It should be noted that the term " server " used in this application usually represents process and the computing machine of the requirement of long-distance user on responding communication network, other equipment, program or their combination everywhere.Server 739 by its information service in request " (one or more) client ".The term " client " used in this article usually represent can process and make request and obtain and process from the computing machine of any response of the server on communication network, program, other equipment, user and/or their combination.
PC platform can based on one or more computer system, described computer system can comprise CPU (central processing unit) (" (one or more) CPU " and/or (one or more) processor), and (these terms are used interchangeably in the disclosure, unless stated to the contrary), storer (such as, ROM (read-only memory) (ROM), random access memory (RAM), cache memory etc.), and/or input/output end port, and can by having interconnects on one or more (mother) plate of conduction and/or other transmission circuit path and/or communication, instruction (such as, binary coded signal) can by described circuit paths transmission to realize communication, operation, store.
Processor and/or transceiver can as inner and/or external peripheral (such as, sensor) via I/O port interconnections.Then transceiver can be connected to (one or more) antenna, realizes radio communication and the reception of various communication and/or sensor agreement thus.CPU comprises at least one high-speed data processor being enough to the program element performed for performing user and/or system generation request.Usually, processor self will comprise various specialized processing units, such as but not limited to: integrated system (bus) controller, memory management control module, floating point unit, and even dedicated processes subelement, as Graphics Processing Unit, digital signal processing unit, and/or similar units.In addition, processor can comprise inner fast access addressable memory, and can map the storer exceeding processor self with addressing; Internal storage can include but not limited to: the cache memory of fast register, various rank (such as, 1,2,3 grade etc.), RAM etc.
Built in items can comprise certain combination of software solution, hardware solution and/or hardware/software solution.Memory interface (such as, data storage device 731) can receive, communicates and/or be connected to multiple memory device, such as but not limited to: memory device, moveable magnetic disc equipment, solid-state drive (SSD) and/or analog.Memory interface can use connection protocol, such as but not limited to: (surpassing) (serial) advanced technology attachment (packet interface) ((surpassing) (serial) ATA (PI)), (enhancing) integrated driving electronic equipment ((E) IDE)), Institute of Electrical and Electric Engineers (IEEE) 1394, optical-fibre channel, small computer system interface (SCSI), USB (universal serial bus) (USB), and/or analog.
(one or more) network card can receive, communicates and/or be connected to communication network 720.By communication network 720, user may have access to PC platform by Terminal Server Client (such as, having the computing machine of web browser).Network interface can use connection protocol, such as but not limited to: directly connect, Ethernet (thick, thin, twisted-pair feeder 10/100/1000BaseT and/or analog), token ring, wireless connections (as IEEE802.11a-x), and/or analog.Cloud service 725 can with PC Platform communication.Cloud service can comprise platform and namely serve (PaaS) model layer, and namely (IaaS) model layer that namely infrastructure serve and software serve (SaaS) model layer.SaaS model layer is generally included and to be managed by center and upgrade, issues on the internet and the software provided by access port.PaaS model layer usually provides service with exploitation, test, issues, trustship and safeguard application program in Integrated Development Environment.IaaS model layer generally includes virtual, virtual machine, such as virtual server, virtual desktop, and/or analog.
IO interface (I/O) can receive, communicates and/or be connected to user input device, peripherals, cryptographic processing apparatus and/or analog.The information that video interface compound is generated by computer system and based on composite information generating video signal in video memory frame.Another output device is the televisor from video interface Received signal strength.Typically, video interface (such as, receives the composite video connector of composite video cable by the video connecting interface of receiver, video display interface; Receive the DVI connector that DVI shows cable, etc.) composite video information is provided.
Peripherals can with I/O and/or other facility connection and/or communications, as network interface, memory interface, be directly connected to interface bus, system bus, CPU and/or analog.Peripherals can in the outside of PC platform, inside and/or the part as PC platform.Peripherals can comprise: antenna, audio frequency apparatus (such as, interior lines, outside line, microphone input, loudspeaker etc.), camera (such as, frequency still camera, video camera, shooting are first-class), cyberdog is (such as, for copy protection, Secure Transaction is ensured by digital signature, and/or analog), ppu is (for increasing ability; Such as, encryption device), force feedback equipment is (such as, vibrating motor), network interface, printer, scanner, memory device, transceiver is (such as, mobile phone, GPS etc.), video equipment (such as, safety goggles, monitor etc.), video source, video device, and/or analog.Peripherals usually comprises the type (such as, camera) of input equipment.
Usually, allow processor to realize the storage of information and/or any embodiment of retrieval and/or embodiment and be regarded as storer.Be to be understood that PC platform and/or computer system can use various forms of storer.In Typical Disposition, storer will comprise ROM, RAM and memory device.Memory device can be any conventional computer system memory storage.Memory device can comprise (fixing and/or removable) disc driver; MO drive (Magneto Optical drive); CD-ROM drive; The array (such as, Redundant Array of Independent Disks (RAID) (RAID)) of equipment; Solid storage device (USB storage, solid-state drive (SSD) etc.); Other processor readable storage mediums; And/or other similar devices.Therefore, computer system 703 usually needs and uses non-provisional and/or temporary storage.
User interface component 741 is the storage program parts performed by CPU.User interface can be provided by operating system 733 and/or operating environment, with operating system and/or operating environment or establish graphical user interface thereon.User interface can be allowed the display of program element and/or system facility, execution, alternately, be processed and/or operation by texture and/or figure facility.User interface provides facility, and user can by described facility impact, mutual and/or operation computer system.User interface can communicate with the miscellaneous part comprised in self component set and/or similar facilities.
Web browser parts are the storage program parts performed by CPU.Web browser can be that conventional hypertext checks application program.Secure web-page is browsed can provide 128 bit encryptions by HTTPS, SSL and/or analog.Web browser and similar information access tool can be integrated in mobile device.Web browser can communicate with the miscellaneous part comprised in self component set and/or similar facilities.Browser can with information server, operating system, integrated program parts (such as, plug-in unit) and/or analog communication; Such as, it can comprise, communicates, generates, obtains and/or provide program element, system, user and/or data communication, request and/or response.And, replace web browser and information server, combination application program can be developed with the similar operations of both execution.
Any one structure in PC platform engine group 705 and/or operation can be combined in many ways, combine and/or distribute so that developed and/or issue.Similarly, component set can combine in many ways so that develop and/or issue.For this reason, a kind of mode can in an integrated manner by component integration to common code storehouse or can in the facility of dynamically loading component as required.Engine group 705 parts can be combined and/or distribution by normal data process and/or development technique with numerous variations.Any one Multi-instance in program element in program element set 735 can in single node and/or on multiple node instantiation with by load balance and/or data processing technique improving SNR.In addition, single instance also can be distributed on multiple controller and/or memory device; Such as, database.All program element examples of collaborative work and controller can be done like this by the normal data process communication technology.
The configuration of PC platform will depend on the environment that system is issued.Include but not limited to budget, capacity, position and/or bottom hardware resource the factor of use may affect to issue and require and configuration.No matter whether configuration causes more associating and/or integrated program parts, causes the distribution serial of more program element and/or causes certain combination between associating with distribution configuration, can communicate, obtains and/or provide data.The example being joined to the parts common code storehouse from program element set can communicate, obtains and/or provide data.This can be realized by data processing communication technology in application program, such as but not limited to: data referencing (such as, indicator), inside story, object instance variable communication, shared storage space, variable transferring, and/or similar techniques.
In certain embodiments, program, equipment and method formation described herein comprises computer-readable medium (such as movable storage medium, as one or more DVD-ROM, CD-ROM, disk, tape etc.) computer program, it provides the software instruction of system at least partially.Be well known in the art, such computer program can be installed by any suitable software installation procedure.In another embodiment, also can being downloaded in cable, communication and/or wireless connections at least partially of software instruction.
Embodiment also can realize as the instruction be stored on the non-provisional machine readable media that can be read by one or more processor and be performed.Nonvolatile machine-readable media can comprise for storing with the readable form of machine (such as, computing equipment 703) or any mechanism of transmission information.Such as, nonvolatile machine-readable media can comprise ROM (read-only memory) (ROM); Random access memory (RAM); Magnetic disk storage medium; Optical storage media; Flash memory device; With other devices.
Although show with reference to its exemplary embodiment and describe the present invention, it should be appreciated by those skilled in the art that and can carry out the various change in form and details in this article and not depart from the scope of the present invention comprised by subsidiary claim.
Particularly describe apparent foregoing teachings from the following of the embodiment shown in accompanying drawing, wherein similar in different views Reference numeral represents identical part all the time.Accompanying drawing not necessarily in proportion, but emphasizes the embodiment that illustrates.
Fig. 1 is schematic diagram, and the system based on depth transducer for counting people is shown;
Fig. 2 is calcspar, and an embodiment of the software architecture based on depth transducer counting according to an embodiment is shown;
Fig. 3 is process flow diagram, and the method based on depth transducer counting according to an embodiment is shown;
Fig. 4 is process flow diagram, and the method counted based on depth transducer and visible spectrum camera according to an embodiment is shown;
Fig. 5 is the set of the sample frame of the depth data received from depth transducer according to the expression of an embodiment;
Fig. 6 is calcspar, and the depth transducer system according to an embodiment is shown; And
Fig. 7 is the schematic diagram of the depth transducer system according to an embodiment.
Obtain depth data from depth transducer, described depth transducer is mounted to the vertical view providing scene;
Foreground object is distinguished from background object in described depth data; And
From described foreground object, the appointment foreground object of the single scale reference model mating target object is determined based on described depth data.
2. the method for claim 1, it also comprises:
Calculate the counterbody size of described appointment foreground object;
Head sizes is derived based on depth data and counterbody size;
The character of detection window is limited based on described head sizes; And
Be applied to described appointment foreground object based on by described detection window, generating scene population counts.
3. method as claimed in claim 2, it is also included in when not moving in scene in predetermined amount of time and described scene population counting is reset to zero.
4. the method for claim 1, its be also included in described depth data frame by frame between follow the tracks of the one or more foreground objects being confirmed as mating described reference model from described foreground object.
5. the method for claim 1, it also comprises and obtains visual picture from visible spectrum camera, and uses the precision counted described in the information check in described visible spectrum camera.
6. method as claimed in claim 5, the information wherein from described visible spectrum camera comprises motion, time attribute, position and feature.
7. method as claimed in claim 6, it also comprises metadata is sent to data center.
8. the method for claim 1, it also comprises the described depth transducer imaging scene of use.
9. the method for claim 1, it also comprises the quantity of speaker, statistics or supplementary data.
10. the method for claim 1, wherein said target object is head part.
11. 1 kinds of computer-readable mediums with programmed instruction stored thereon, described programmed instruction can be performed by processor, and when loading and performed by described processor, causes described processor:
Depth data is obtained from depth transducer;
From described depth data identification foreground object;
Applied for machines study application program is determined with the classification generating described foreground object; And
Classification is kept to determine counting.
12. 1 kinds of systems, it comprises:
Depth transducer, described depth transducer is configured to the degree of depth of imaging object with sampling depth data;
The storer communicated with described depth transducer, described storer is configured to store described depth data;
With the processor of described memory communication, described processor is configured to execution of program instructions, and described programmed instruction causes described processor:
From described foreground object, the appointment foreground object of the reference model mating target object is determined based on described depth data.
13. systems as claimed in claim 12, wherein said processor is also configured to:
Head sizes is derived based on described depth data and counterbody size;
Based on described detection window being applied to described appointment foreground object generating scene population counting.
14. systems as claimed in claim 12, wherein said processor be also configured to described depth data frame by frame between follow the tracks of the one or more foreground objects being confirmed as mating described reference model from described foreground object.
15. systems as claimed in claim 13, reset to zero by described scene population counting wherein said processor is also configured to not move in scene within a predetermined period of time.
16. systems as claimed in claim 12, wherein said processor is also configured to obtain visual picture from visible spectrum camera and use the precision counted described in the information check described visible spectrum camera.
17. systems as claimed in claim 16, the information wherein from described visible spectrum camera comprises motion, time attribute, position and feature.
18. systems as claimed in claim 17, wherein said processor is also configured to metadata to be sent to data center.
19. systems as claimed in claim 12, wherein said processor is also configured to use described depth transducer imaging scene.
20. systems as claimed in claim 12, wherein said processor is also configured to the quantity of speaker, statistics or supplementary data.
21. systems as claimed in claim 12, wherein said target object is head part.
CN201380070934.6A 2012-11-21 2013-11-19 The method and system of people is counted using depth transducer CN105122270B (en)
US13/683,886 US10009579B2 (en) 2012-11-21 2012-11-21 Method and system for counting people using depth sensor
US13/683,886 2012-11-21
PCT/US2013/070688 WO2014081687A1 (en) 2012-11-21 2013-11-19 Method and system for counting people using depth sensor
CN105122270A true CN105122270A (en) 2015-12-02
CN105122270B CN105122270B (en) 2019-03-08
ID=49766153
CN201380070934.6A CN105122270B (en) 2012-11-21 2013-11-19 The method and system of people is counted using depth transducer
US (1) US10009579B2 (en)
EP (1) EP2923304A1 (en)
CN (1) CN105122270B (en)
WO (1) WO2014081687A1 (en)
WO2016064377A1 (en) * 2014-10-20 2016-04-28 King Abdullah University Of Science And Technology System and method for crowd counting and tracking
CN108701211A (en) * 2015-12-28 2018-10-23 罗伯特·博世有限公司 For detecting, tracking, estimating and identifying the system based on depth sense occupied in real time
ITUB20159615A1 (en) * 2015-12-29 2017-06-29 Universita’ Degli Studi Di Trieste Method for the detection of foot traffic in one space.
CZ2016453A3 (en) * 2016-07-27 2017-08-30 Vysoké Učení Technické V Brně A method of detecting unauthorized access attempts to protected areas
CN106228560B (en) * 2016-07-28 2019-05-14 长安大学 A kind of demographic method under complex scene
KR20190099216A (en) * 2016-12-22 2019-08-26 로베르트 보쉬 게엠베하 RGBD detection based object detection system and method
CN107103279A (en) * 2017-03-09 2017-08-29 广东顺德中山大学卡内基梅隆大学国际联合研究院 A kind of passenger flow counting method under vertical angle of view based on deep learning
CN107092871B (en) * 2017-04-06 2018-01-16 重庆市地理信息中心 Remote sensing image building detection method based on multiple dimensioned multiple features fusion
KR101967858B1 (en) * 2017-11-03 2019-04-11 전자부품연구원 Apparatus and method for separating objects based on 3D depth image
TWI628624B (en) * 2017-11-30 2018-07-01 國家中山科學研究院 Improved thermal image feature extraction method
CN102334142A (en) * 2009-02-24 2012-01-25 三菱电机株式会社 Human tracking device and human tracking program
JP3775683B2 (en) 2003-08-21 2006-05-17 松下電器産業株式会社 Person detection device and person detection method
FR2909205B1 (en) 2006-11-28 2009-01-23 Commissariat Energie Atomique Method for designation of an object in an image
JP4829141B2 (en) * 2007-02-09 2011-12-07 株式会社東芝 Gaze detection apparatus and method
DE102007028861A1 (en) 2007-06-22 2009-01-02 Albert-Ludwigs-Universität Freiburg Method for computer-aided prediction of intended movements
US8107676B2 (en) 2007-07-30 2012-01-31 International Business Machines Corporation Line length estimation
WO2010073929A1 (en) * 2008-12-22 2010-07-01 日本電気株式会社 Person-judging device, method, and program
US8885926B2 (en) 2009-04-15 2014-11-11 Massachusetts Institute Of Technology Image and data segmentation
JP2010266983A (en) 2009-05-13 2010-11-25 Sony Corp Information processing apparatus and method, learning device and method, program, and information processing system
JP2011172205A (en) 2010-01-20 2011-09-01 Canon Inc Video information processing apparatus and method
US8600108B2 (en) 2010-01-25 2013-12-03 Hewlett-Packard Development Compant, L.P. Data processing system and method
TW201220253A (en) * 2010-11-15 2012-05-16 Inst Information Industry Image calculation method and apparatus
US9001199B2 (en) 2011-04-29 2015-04-07 Tata Consultancy Services Limited System and method for human detection and counting using background modeling, HOG and Haar features
US9275472B2 (en) 2011-11-18 2016-03-01 Disney Enterprises, Inc. Real-time player detection from a single calibrated camera
US9076065B1 (en) 2012-01-26 2015-07-07 Google Inc. Detecting objects in images
US9226691B2 (en) * 2012-06-01 2016-01-05 Xerox Corporation Processing a video for tidal chest volume estimation
KR101932535B1 (en) * 2012-08-27 2018-12-27 한화테크윈 주식회사 Indoor surveillance system and method
WO2014037939A1 (en) * 2012-09-05 2014-03-13 Body Pass Ltd. System and method for deriving accurate body size measures from a sequence of 2d images
US9165190B2 (en) * 2012-09-12 2015-10-20 Avigilon Fortress Corporation 3D human pose and shape modeling
US9330297B2 (en) 2013-01-16 2016-05-03 Xerox Corporation Methods and systems for video-based chew counting via feature tracking
2012-11-21 US US13/683,886 patent/US10009579B2/en active Active
2013-11-19 EP EP13805981.1A patent/EP2923304A1/en active Pending
2013-11-19 CN CN201380070934.6A patent/CN105122270B/en active IP Right Grant
2013-11-19 WO PCT/US2013/070688 patent/WO2014081687A1/en active Application Filing
BYOUNG-KYU DAN 等: "Robust People Counting System Based on Sensor Fusion", 《IEEE TRANSACTIONS ON CONSUMER ELECTRONICS》 *
TAREK YAHIAOUI 等: "Real-time passenger counting in buses using dense stereovision", 《JOURNAL OF ELECTRONIC IMAGING》 *
US10009579B2 (en) 2018-06-26
CN105122270B (en) 2019-03-08
EP2923304A1 (en) 2015-09-30
US20140139633A1 (en) 2014-05-22
WO2014081687A1 (en) 2014-05-30
Norouzzadeh et al. 2018 Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning
DE112009000485T5 (en) 2011-03-17 Object comparison for tracking, indexing and searching
Ding et al. 2018 A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory
Ding et al. 2014 An experimental evaluation of novelty detection methods