Patent ID: 12254600

DETAILED DESCRIPTION OF THE EMBODIMENTS

The disclosure adopts a joint imaging system based on an unmanned aerial vehicle platform, as shown inFIG.1, which includes a flying unit1, a shutter control unit2, a posture control unit3, an airborne image transmission unit4, a load pan-tilt interface5, a ten-lens imaging unit6and an airborne computing unit7.

The flying unit1is used for carrying load equipment to fly according to a predetermined route;the shutter control unit2is used for controlling shooting parameters such as shutter time and exposure time of the load camera;the posture control unit3includes an IMU module and a GPS module, where the IMU module is used for measuring and recording the posture parameters of the flying unit and the position of the load camera, and the GPS module is used for measuring and recording the accurate geographic information position of the flying unit;the airborne image transmission unit4is used for transmitting thermal infrared images and visible light images to ground equipment in a wireless transmission mode;the airborne computing unit7includes an image processing module, which is used for receiving the image data of the load camera and the camera position and posture recorded by the posture control unit, and processing and fusing the received images in real time;the ten-lens imaging unit6is connected with the shutter control unit2, the posture control unit3and the airborne computing unit7through the load pan-tilt interface5. The ten-lens imaging unit6is used for color imaging and thermal infrared imaging of the target object, collecting image data and transmitting the image data to the airborne computing unit7.

In order to further illustrate the above-mentioned system of the present disclosure, it will be explained by the following specific embodiments.

As shown inFIG.2, the structure of the ten-lens imaging unit6includes visible light lenses601and thermal infrared lenses602.

The ten-lens imaging unit6includes five visible light lenses with the same specifications and five thermal infrared lenses with the same specifications.

The visible light lenses are RGB color cameras and the thermal infrared lenses are uncooled medium-wave infrared cameras.

The visible light lens and the thermal infrared lens are arranged in an equidistant cross layout, and are arranged at the center position and four corners equidistant from the center; the lens at the center position is arranged vertically downwards, and the lenses at the four corners are inclined to the center at the same angle, and the inclination angles are between 30 degrees and 50 degrees.

The imaging ranges of the visible light lens and the thermal infrared lens overlap in five directions, and the overlapping rate of the imaging ranges of the visible light lens and the thermal infrared lens in five visual angles is above 60%, and the imaging ranges of the thermal infrared lens are included in the imaging ranges of the visible light lens.

The visible light lens and the thermal infrared lens are adjacently arranged at the center position and four corners respectively, and the adjacent visible light lenses and the thermal infrared lens have the same inclination direction and imaging visual angle, and are arranged in a parallel optical axis structure, as shown inFIG.3.

As shown inFIG.4, the method for realizing image enhancement fusion by using the system include the following steps.

Step 1, collecting thermal infrared images and visible light images of the same object in the same time domain and the same space domain.

Step 2, inputting the internal parameter matrix and distortion coefficient of the thermal infrared camera and the visible light camera into the image processing module respectively, and correcting the radial and lateral distortion of the thermal infrared camera and the visible light camera according to the following formulas:

Zc[uv1]=(fx0uo0fyvo001)*(RT0T1)[XWYWZW1]⁢def——⁢KCPWr2=x2+y2{xdistortion=x⁡(1+k1⁢r2+k2⁢r4+k3⁢r6)+2⁢p1⁢xy+p2⁢(r2+2⁢x2)ydistortion=y⁡(1+k1⁢r2+k2⁢r4+k3⁢r6)+2⁢p2⁢xy+p1⁢(r2+2⁢y2),

where: fx, fy, u0and v0are internal parameter coefficients; R is an external parameter rotation matrix; T is an external parameter translation vector; K is an internal parameter matrix; ZCis a Z-axis coordinate of a calibration point in a camera coordinate system; C is a spatial transformation matrix from a world coordinate system to the camera coordinate system; XW, YWand ZWrespectively represent an X-axis coordinate, a Y-axis coordinate and a Z-axis coordinate of the calibration point in the world coordinate system; u and v respectively represent an abscissa and an ordinate of the calibration point in a pixel coordinate system; x and y are an X-axis coordinate and a Y-axis coordinate of the calibration point on a normalized plane of a camera, respectively; r is a polar coordinate form of the calibration point in the normalized plane; k1, k2, k3, p1, and p2are distortion coefficients; and xdistortion, ydistortionare distortion values of the camera in a direction of x, y.

Step 3, respectively identifying the same feature point in the visible light image and the thermal infrared image, measuring the coordinate values of the point in the visible light pixel coordinate system and the thermal infrared pixel coordinate system, and according to the solved transformation matrix between the visible light Ppixel_Vand the thermal infrared pixel coordinate system Ppixel_Iand the world coordinate system, and according to the formula:

CI-1(fx_I-10u1⁢fx_I-10fy_I-1vI⁢fy_I-1001)⁢Ppixel_I=CV-1⁢(fx_V-10uV⁢fx_V-10fy_V-1vV⁢fy_V-1001)⁢Ppixel_V,

where: CI−1and CV−1are an inverse matrix of the internal parameter matrix of the thermal infrared camera and an inverse matrix of the internal parameter matrix of the visible light camera respectively, and Ppixel_Iand Ppixel_Vare an abscissa and an ordinate of the calibration point in the pixel coordinate system; fx_I−1and fy_I−1are reciprocals of the internal parameter coefficients of the thermal infrared camera; fx_V−1and fy_V−1are reciprocals of the internal parameter coefficients of the visible light camera; uIand vIare coordinates of the thermal infrared camera in the pixel coordinate system; and uVand vVare coordinates of visible light in the pixel coordinate system.

The rotation matrix and translation matrix of pixel coordinates of visible light images and thermal infrared images are capable of being obtained by linear transformation, and the spatial geometric registration is completed.

The difference values between abscissas and ordinates of pixels of two adjacent calibration points in the visible light images and the thermal infrared images are calculated respectively, where the image scale factor σ is a ratio of a distance between the two adjacent calibration points in visible light images and a distance between the two adjacent calibration points in thermal infrared images;

σScalefactor=Xinfraredn-Xinfraredn-1Xvisiblen-Xvisiblen-1,

where n is the number of checkerboard calibration points; the offset vector is the column vector of the same calibration point in the visible light pixel coordinate system and the column vector in the thermal infrared pixel coordinate system of a same calibration point:

[xshiftyshift]=[xv⁢isiblenyvisiblen]-[xinfrarednyinfraredn]

Spatial information registration of thermal infrared images and visible light images is realized by combining offset vector.

Step 4, down-sampling the visible light images, unifying imaging specifications and imaging ranges of the thermal infrared images and the visible light images: unifying resolution of the images, unifying the imaging ranges to be thermal infrared image imaging ranges and unifying spatial scale to be scale parameters of the thermal infrared images, taking the imaging ranges of the thermal infrared images as the interest areas, multiplying the visible light image V(x, y) by the image scale factor to scale to the spatial scale L(u, v) of the thermal infrared image, and combining with the offset vector to realize the spatial information registration of the thermal infrared images and the visible light images:

L⁡(u,v)=V⁡(x,y)*σScalefactor;

according to the image scale factor and offset vector, the visible light images are corrected, matched and the interest areas are reserved by using the image processing function.

Step 5, using edge detection operators such as Canny operator and Sabel operator to detect the edge of the down-sampled visible light images, and the edge skeleton maps of the visible light images are extracted and binarized.

Step 6, adding the thermal infrared pseudo-color image matrix and the edge feature binary image matrix by adopting a pixel-level fusion method to obtain thermal infrared images with enhanced texture details, thereby improving the number of feature points of the thermal infrared images and the recognition capability.

Embodiment

The present disclosure will be further described in detail with an embodiment.

(1) The unmanned aerial vehicle is equipped with a ten-lens joint imaging system to image the target, and a group of thermal infrared images and visible light images are obtained, where the thermal infrared image resolution is 640×512, the focal length is 25 mm, the visible light image resolution is 4000×3000, and the focal length is 8 mm.

(2) By extracting the coordinate values of two same calibration points in the thermal infrared image and the visible light image in the respective pixel coordinate systems, the image scale factor σscalefactor=0.34 and the offset vector

[xshiftyshift]=[372261]
are calculated.

(3) Taking the imaging ranges of the thermal infrared images as the interest areas, the visible light images are down-sampled to the scale of the thermal infrared images to obtain the down-sampled visible light images with the image resolution of 640×512, and the thermal infrared images are moved into the down-sampled visible light images according to the offset vector to realize spatial registration.

(4) Edge detection is carried out by gradient descent method, and the detailed feature information of the down-sampled visible light image is extracted and binarized to obtain a binary image of texture feature information.

(5) Using the image fusion method based on pixel level, the binary image of texture feature information is fused with the thermal infrared images, and finally the thermal infrared images with texture details are obtained.

The disclosure provides a joint imaging system based on an unmanned aerial vehicle platform and an image enhancement fusion method. There are many ways and means to realize the technical scheme. The above is only the preferred embodiment of the disclosure. It should be pointed out that for ordinary technicians in the technical field, several improvements and retouching can be made without departing from the principle of the disclosure, and these improvements and retouching should also be regarded as the protection scope of the disclosure. All components that are not clear in this embodiment can be realized by existing technology.