Patent Description:
In the existing system capturing for vehicle horn, a coordinate transformation coefficient, between a central position coordinate of a horn sound source obtained by a microphone array and a high-definition image obtained by a high-definition camera, generally needs to be calibrated after field installation. The coordinate transformation coefficient relates to factors of the microphone array and the high-definition camera, such as installation position, angle, height, etc. The general procedure of conventional calibration for the coordinate transformation coefficient includes: using a test vehicle, respectively blowing the horn at different positions in the snapshot range; acquiring an audio signal and obtaining a central position coordinate of the horn sound source after calculation by the microphone array; finding a pixel point corresponding to the position where the license plate of the test vehicle is located when the horn is honked in a high-definition image; and calculating the coordinate transformation coefficient between the central position coordinate of the horn sound and the pixel point of the license plate by the linear coordinate transformation formula.

The calibration method for the coordinate transformation coefficient has the following three problems:.

<CIT> relates to an electronic traffic police system which comprises a whistle sound acquisition unit, a whistle sound recognition unit, a whistle sound source location unit, a position measurement unit, a whistle sound source tracking unit, an image acquisition unit and a license plate recognition unit, wherein the whistle sound acquisition unit, the whistle sound recognition unit, the whistle sound source location unit, the position measurement unit, the whistle sound source tracking unit, the image acquisition unit and the license plate recognition unit are based on a microphone array and are successively connected.

<CIT> discloses a method for real-time positioning of a whistling motor vehicle. The method for real-time positioning of a whistling motor vehicle comprises the following steps: obtaining real-time sound data; obtaining real-time image data; judging whether a whistling sound exists according to the real-time sound data; if the whistling sound exists, identifying the source direction of the whistling sound; and positioning the whistling motor vehicle according to the source direction of the whistling sound and the real-time image data.

<CIT> discloses an illegal honking snapshot system comprising a camera snapshot system and an omnidirectional passive sonar. The camera snapshot system comprises a host and a high-definition camera. The high-definition camera and the omnidirectional passive sonar are connected with the host. The distance between the high-definition camera and the omnidirectional passive sonar is not zero. The omnidirectional passive sonar comprises sixteen energy converters of which the openings are all exactly downward. Every four energy converters form a row. The host is internally provided with a Linux or Windows system.

<CIT> discloses a law enforcement and evidence taking system for car horning. The system is composed of a sonar system and a high-definition camera. The sonar system is responsible for collecting sound data, calculating a spectrum characteristic, and determining whether a horn sound is an effective one based on the spectrum characteristic; and if the horn sound is an effective one, the high-definition camera is triggered to capture a car image, licence plate information is identified, and car horning feature audio frequency information is recorded.

To solve the deficiencies in the prior art, the invention is as set out in the appended set of claims. The invention provides a system for capturing vehicle horn based on image pairing, which can automatically and accurately calibrate a coordinate transformation coefficient between a central position coordinate of a horn sound source obtained by a microphone array and a pixel point of a high-definition image obtained by a high-definition camera.

The invention is implemented by the following technical scheme.

The invention relates to a system for identifying a vehicle using a horn based on sound source positioning and image pairing and a corresponding method according to respectively claim <NUM> and claim <NUM> of the appended claims.

Other features of the embodiments are set out in the dependent claims. frame of low-resolution image which is acquired from the integrated microphone array and superimposed with the central position coordinate of the horn sound source when the horn is honked, and at the same time, acquires a frame of high-resolution image from the high-definition camera and outputs the high-resolution image to an image pairing system; and the image pairing system acquires an inter-image transformation coefficient between the low-resolution image and the high-resolution image, so as to obtain the corresponding position of the central position coordinate of the horn sound source in the high-resolution image, and provides the License information of the vehicle corresponding to the position via the image target recognition technology.

The camera and the microphone array in the integrated microphone array are of an integrated design. Once the assembly and calibration of the integrated microphone array itself are complete, the relative position of the camera and all microphones in the microphone array are unchanged; a sound source central position coordinate which is obtained by the integrated microphone array after calculation and a preset coordinate coefficient of the pixel point in the low-resolution image are unchanged; and the integrated microphone array outputs a low-resolution image which is superimposed with the central position coordinate of the horn sound source to the control unit.

The invention relates to an image pairing method implemented in the above system. The method includes: acquiring, by the control unit, two image frames at the same time from the integrated microphone array and the high-definition camera; acquiring, by the image pairing system, a feature point of an image and a descriptor corresponding to the feature point for each of the image frames; obtaining a feature point similarity according to the information of a feature point descriptor; ranking the feature point similarities; selecting a plurality of feature points with the highest feature point similarity and the fixed positions on the image frames as matching feature points of the two image frames; and calculating the inter-image transformation coefficient between the integrated microphone array camera and the high-definition camera by the linear coordinate transformation, so as to realize the position matching from the low-resolution image frame to the high-resolution image frame.

The acquiring of the feature point of the image refers to: performing Gaussian blur on the image to construct a Gaussian pyramid and obtain different scale spaces; obtaining local extreme points through extreme-point detection on different scale spaces; and performing preliminary and accurate positioning on the local extreme points, deleting an edge response point, and obtaining the feature point.

Preferably, the feature points are at least four pairs, and further preferably are six pairs.

The feature point descriptor is generated according to the gradient magnitude and direction of the feature point.

The feature point similarity refers to: calculating the similarity metric between any two feature point descriptors according tod <MAT>, and calculating the feature point similarity according to feature point similarity = <MAT> for the feature point descriptor Ri = (ri<NUM>, ri<NUM>,. , ri128) in the high-resolution image and the feature point descriptor Si = (si<NUM>, si<NUM>,. , si128)in the low-resolution image.

The invention relates to an image pairing system for realizing the above method, including an image preprocessing module, a feature point calculation module, a feature point descriptor calculation module, a feature point similarity calculation module, a ranking module and a linear coordinate transformation module; in which the image preprocessing module is connected with the feature point calculation module and outputs an image after grayscale processing to the feature point calculation module; the feature point calculation module is connected with the feature point descriptor calculation module and outputs feature point information to the feature point descriptor calculation module; the feature point descriptor calculation module is connected with the feature point similarity calculation module and outputs the feature point descriptor information to the feature point similarity calculation module; the feature point similarity calculation module is connected with the ranking module and outputs feature point similarity information to the ranking module; the ranking module is connected with the linear coordinate transformation module and outputs a matching feature point to the linear coordinate transformation module; and the linear coordinate transformation module obtains the inter-image transformation coefficient of the two frame of images after calculation.

Compared to the prior art, the invention adopts the integrated microphone array and the high-definition camera to monitor the horn use; the coordinate transformation between the integrated microphone array and the high-definition camera is completed based on the image pairing method; the inter-image transformation coefficient between the integrated microphone array and the high-definition camera can be calibrated quickly and accurately without the need to blow the test vehicle horn on the spot; and the calibration error is extremely small, and the invention is not influenced by the environment of the installation site. During system use, it can be selected whether to recalculate the inter-image transformation coefficient through image pairing when taking a snapshot of illegal horn use each time, so that the accuracy of each snapshot of illegal horn use each time can be ensured even if the relative position between the integrated microphone array and the high-definition camera changes due to natural, human and other factors.

The license plate number and the digital content in the drawings are both modified in the invention, and bear no relation to the actual vehicle with the same license plate.

As shown in <FIG>, the embodiment includes: an integrated microphone array, a high-definition camera, a control unit, and an image pairing system; in which the integrated microphone array includes a microphone array composed of a plurality of microphones and a camera, which are used for acquiring audio information and a static and/or dynamic low-resolution image in real time, and the high-definition camera acquires a high-resolution image in real time; a camera involved in the integrated microphone array is usually low in resolution, and obtains a low-resolution image; and the high-definition camera obtains the high-resolution image which can be used for identifying License information of the vehicle.

The image pairing system includes: an image preprocessing module, a feature point calculation module, a feature point descriptor calculation module, a feature point similarity calculation module, an ranking module and a linear coordinate transformation module; in which the image preprocessing module is connected with the feature point calculation module and outputs an image after grayscale processing to the feature point calculation module; the feature point calculation module is connected with the feature point descriptor calculation module and outputs feature point information to the feature point descriptor calculation module; the feature point descriptor calculation module is connected with the feature point similarity calculation module and outputs the feature point descriptor information to the feature point similarity calculation module; the feature point similarity calculation module is connected with the ranking module and outputs feature point similarity information to the ranking module; the ranking module is connected with the linear coordinate transformation module and outputs a matching feature point to the linear coordinate transformation module; and the linear coordinate transformation module obtains an inter-image transformation coefficient of the two frame of images after calculation.

As shown in <FIG> and <FIG>, the control unit determines whether the horn is honked, acquires a frame of low-resolution image which is superimposed with the central position coordinate of the horn sound source from the integrated microphone array; a frame of high-resolution image is acquired from the high-definition camera at the same time; and the image pairing system acquires the inter-image transformation coefficient between the low-resolution image and the high-resolution image, so as to obtain the corresponding position of the central position coordinate of the horn sound source in the high-resolution image, and provides the License information of the vehicle corresponding to the position in combination with the image target recognition technology.

As shown in <FIG>, the image pairing system automatically acquires two frame of images at the same time from the integrated microphone array camera and the high-definition camera, acquires a feature point of the image and a descriptor corresponding to the feature point for each of the images, obtains a feature point similarity according to the information of the feature point descriptor, ranks the feature point similarities, automatically selects a plurality of feature points with the highest feature point similarity and the fixed positions on the image as matching feature points of the two frame of images, calculates the inter-image transformation coefficient between the integrated microphone array camera and the high-definition camera through linear coordinate transformation, and calculates a corresponding pixel point coordinate on the image acquired by the high-definition camera from any pixel point coordinate on the image acquired by the integrated microphone array camera by the inter-image transformation coefficient. The specific process is as follows.

Step <NUM>, performing grayscale processing on the image.

Step <NUM>, constructing a Gaussian pyramid to obtain different scale spaces after performing Gaussian blur on the image; obtaining local extreme points through extreme-point detection on different scale spaces; obtaining the feature point after performing preliminary and accurate positioning on the local extreme points and deleting an edge response point; generating the feature point descriptor according to the gradient magnitude and direction of the feature point; obtaining the feature point similarity according to the information of the feature point descriptor; and ranking the feature point similarities. n pairs of feature points with the highest feature point similarity and the fixed positions on the image can be automatically or manually selected as the matching feature points of the two frame of images, in whichn ≥ <NUM>. In the embodiment, n = <NUM>, and n is not limited to value <NUM> in the present embodiment, as shown in <FIG>.

The Gaussian blur means that: a Gaussian function in an N-dimensional space is used to obtain a two-dimensional fuzzy template by calculation, and the two-dimensional fuzzy template is used to perform convolution operation with the original image, so as to blur the image.

The Gaussian function in the N-dimensional space is: <MAT>, in which σis the standard deviation of the normal distribution, the greater the value ofσ, the more blurred the image becomes; andris the blur radius which refers to the distance from the template element to the template center.

The convolution refers to: calculating the size of the Gaussian template matrix according to the value of σ, calculating the value of the Gaussian template matrix, performing convolution with the original image, and obtaining a Gaussian blur image of the original image. That is, the scale spaceL(x, y, σ)of one image is defined as the convolution of a Gaussian function G(x, y, σ) of varying scale and the original image I(x,y), L(x, y, σ) = G(x, y, σ) * I(x, y), in which * represents a convolution operation.

The size of the two-dimensional fuzzy template ism * n, and the Gaussian calculation formula corresponding to the elements (x, y) on the template is G(x, y) = <MAT>.

To ensure that the elements in the template matrix are between [<NUM>,<NUM>], it is preferable to normalize the two-dimensional fuzzy template matrix, and a <NUM>*<NUM> Gaussian template is shown in the table below:.

The number of layers of the Gaussian pyramid is determined according to the original size of the image and the size of the top image, and the number of layers n = log<NUM>{min(M, N)} - t, t ∈ [<NUM>, log<NUM>{min(M, N)}], in which M and N are the sizes of the original image, and t is the logarithmic value of the smallest dimension of the top image.

The Gaussian pyramid uses a difference of Gaussian (DOG) for extreme value detection, that is,D(x, y, σ) = [G(x, y, kσ) - G(x, y, σ)] * I(x, y) = L(x, y, kσ) - L(x, y, σ), and compares each pixel point with an adjacent point on the image domain and the scale domain to obtain a local extreme point.

The preliminary and accurate positioning means that: the Taylor expansion (fitting function) of the DOG in the scale space is: <MAT>, in which X = (x, y, σ)T; the offset of the extreme point is <MAT> through derivation; the value of the corresponding equation is <MAT>, and all extreme points in whichD(X) is less than <NUM> are deleted, so that a preliminary and accurate positioning result is obtained, and <NUM> is an empirical value.

Preferably, the preliminary and accurate positioning result obtained as above are further performed deletion on an unstable edge response point, which specifically refers to: constructing a Hessian matrix, <MAT>, in which:Tr(H) = Dxx + Dxy = α + β , Det(H) = DxxDyy - (Dxy)<NUM> = αβ , α represents a gradient in thexdirection,βrepresents a gradient in the ydirection, r is generally taken as <NUM> to detect whether the main curvature is at a certain domain valuer; and <MAT> is detected, and the feature point is kept when satisfied, or the feature point is deleted, so as to obtain an accurate positioning result.

The modulus and direction of the feature point gradient are: the pixel point gradient is denoted as <MAT>; the gradient magnitude is represented as m(x, y) = <MAT>; and the gradient direction is: <MAT>.

The feature point descriptor mean that: the feature point is described with a set of vectors to prevent the vectors from changing with illumination, viewing angle, and the like; the vector, or the feature point descriptor, include the feature point and the pixel point which are around the feature point and contribute to the feature point, and the feature point descriptor should have higher uniqueness in order to increase the probability that the feature point is correctly matched.

The feature point descriptor is obtained by calculation in the following manner:.

The feature point similarity refers to: calculating the similarity metric between any two feature point descriptors according to <MAT>, and calculating the feature point similarity according to feature point similarity = <MAT> for the feature point descriptor Ri = (ri<NUM>, ri<NUM>,. , ri128) in the high-resolution image and the feature point descriptorSi = (si<NUM>, si<NUM>,. , si128)in the low-resolution image.

The feature point similarity ranking refers to: ranking the feature point from high to low according to the feature point similarity; n pairs of feature points with the highest feature point similarity and the fixed positions on the image can be automatically or manually selected and defined as the matching feature points, in which n ≥ <NUM>. In the embodiment, n = <NUM>, and n is not limited to the value <NUM> in the present embodiment.

Step <NUM>, performing spatial coordinate transformation:.

The central position coordinate of the horn sound source is obtained by positioning the sound source of the vehicle with illegal horn use by means of the beamforming algorithm based on spherical wave except auto spectrum, and generating a sound pressure nephogram; and the coordinate corresponding to the maximum value of the sound pressure in the sound pressure nephogram is the central position coordinate of the horn sound source.

Specifically, the beamforming algorithm is: <MAT>, in whichV(k, w)is the mean square value of the beamforming, kis the focusing direction, wis the angular frequency, Mis the number of sensors of the microphone array,Cnmis the cross-spectrum of the sound pressure signal received by the mth microphone relative to the sound pressure signal received by the nth microphone,rmis the coordinate vector of the mth microphone, and rnis the coordinate vector of the nth microphone.

Claim 1:
A system for identifying a vehicle using a horn based on sound source positioning and image pairing, the system comprising:
an integrated microphone array configured to acquire audio and a low-resolution image frame in real time;
a high-definition camera configured to acquire a high-resolution image frame in real time;
a control unit (<NUM>); and
an image pairing system,
wherein the integrated microphone array includes a microphone array composed of a plurality of microphones (<NUM>) and a camera (<NUM>) configured to acquire the low-resolution image frame; the control unit (<NUM>) is configured to monitor the image frame of low-resolution which is acquired from the integrated microphone array and superimposed with a central position coordinate of a horn sound source when a horn is honked which is obtained by the integrated microphone array after calculation, and at the same time, is configured to acquire the image frame of high-resolution from the high-definition camera and is configured to output the high-resolution image frame to the image pairing system; and the image pairing system is configured to acquire an inter-image transformation coefficient between the low-resolution image frame and the high-resolution image frame, so as to obtain a corresponding position of the central position coordinate of the horn sound source in the high-resolution image frame, and is configured to provide License information of a vehicle corresponding to the position via an image target recognition technology.