Patent ID: 12211259

DETAILED DESCRIPTION OF THE EMBODIMENTS

The technical scheme of the present invention is explained below in combination with the attached drawings, but is not limited to this. Any modification or equivalent replacement of the technical scheme of the present invention without deviating from the spirit and scope of the technical scheme of the present invention shall be included in the protection scope of the present invention.

The present invention provides a deep learning based segmentation and identification method of RCSP and extraction of multidimensional characterization parameters. By building a binocular image acquisition system of recycled concrete sand, a binocular camera is used to obtain multidimensional image data of RCSP. By training U-Net semantic segmentation network model and combining with dynamic image processing method, the image segmentation and morphology characterization parameters of RCSP can be quickly extracted. Specific optimization examples are as follows:

Step 1. Binocular Camera Calibration:

(1) In the experiment, “Zhang's calibration method” was used to calibrate the binocular camera: the checkerboard was calibrated by printing black and white. The size of the checkerboard was 10×7, and the size of the single checkerboard was a cube with side length of 11 mm.(2) After the completion of the calibration board, two industrial cameras were used to take checkerboard photos of different positions and angles, and the internal and external parameters and distortion parameters of binocular camera were obtained by combining the calibration toolbox of MATLAB software, which were used for the subsequent correction of binocular image and generation of depth image by combining OpenCV.FIG.1AandFIG.1Bare the specific camera parameter information calibrated by MATLAB, and table 1 shows the obtained internal and external parameters and distortion parameters of binocular camera.

TABLE 1Internal and external parameters and distortion parameters of binocular cameraDistortion_leftDistortion_rightCam_matrix_leftCam_matrix_right(k1, k2, p1, p2, k3)(k1, k2, p1, p2, k3)Rotation_matrixTranslation_matrix335200339400−0.3−0.20.90.0−0.5−264.7−2.033420733980−1.9−5.10.01.00.010.663062411081566127.750.00.50.00.968.00.00.00.00.0
Step 2. Binocular Image Collection:(1) Experiment 500 g of RCSP in a vibrating feeder bin conveyor belt, by adjusting the frequency control transfer speed, make the RCSP evenly dispersed to the conveyor belt to end and make free fall down into image acquisition area.(2) Adjust the acquisition Angle, position and light source intensity of the two industrial cameras. Table 2 shows the specific setting parameters of the camera and light source.

TABLE 2Specific setting parameters of camera and light sourceSet parametersUnitThe numericalThe frame rate capturedFps150Time of exposuremm800Gamma valueAPI1.18BrightnessNits50Area source voltageV3.6Distance between the camerasmm264.66Left/right camera Anglemm45°/90°Distance from camera to light sourcemm52.56(3) In the experiment, the falling RCSP are collected synchronously in the same frame and transmitted synchronously to the PC terminal for image processing. Finally, when the RCSP are completely collected and fall into the recycling box, the image of this batch of recycled concrete sand experimental particles is collected completely.
Step 3. Processing and Division of Data Set:(1) Image samples containing RCSP were selected according to the collected data set.(2) A total of 116 original images of recycled concrete sand were collected in this experiment. Image enhancement technology was used to cut, mirror, rotate and enlarge the original images, so that the original image data was enhanced to 3244.(3) For the training set and verification set in the data set, Labelme was used to mark the regenerated sand particles in the image as red, while other backgrounds were marked as black, so as to facilitate the subsequent image substitute into the network for training.
Step 4. Construction of Network Model:(1) The experiment is based on Pytorch open source neural network framework, and uses U-Net neural network structure to construct semantic segmentation model of RCSP. Table 3 shows the selected network parameters.

TABLE 3Network parametersModelChoiceActivation functionReLULoss functionThe cross entropyLearning rate adjustmentPiecewise constant attenuationOptimizerAdam(2) After the original image is input into the U-Net network model, the size of the convolution image is changed to 256×256 by two convolution kernels with a size of 3×3, and the edge padding=1 is used to keep the size of the convolution image unchanged, and then activated by a ReLU function. Finally, the size of the convolution image is changed to 256×256 by a size of 2×2 maximum pooling. This process is a complete down-sampling. The following three down-sampling operations are the same as above. In the down-sampling process, the number of channels of the image is continuously doubled from 64 channels to 1024 channels. After the down-sampling is completed, splicing and up-sampling are carried out synchronously. (Splicing refers to the fusion of shallow information acquired in feature extraction and deep information in up-sampling, that is, the features of shallow information are fused and splicing during up-sampling to improve the learning accuracy of the network. The up-sampling part is also called extended network, which enlarges the size of the image continuously to extract deep information.) Four up-sampling parts are used continuously, In the process of up-sampling, the number of channels in the image is halved continuously, which is contrary to the change of channel number in the process of feature extraction. In the experiment, the size of the final segmentation image is 512×512, which is consistent with the size of the input image, and the number of channels is 2 (background image and recycled concrete sand segmentation target),FIG.2shows the structure of U-Net network used
Step 5. Model Training and Verification:(1) After the model was built, the experiment divided the enhanced 3200 image data into data sets according to the ratio of 9:1, among which 2880 were training sets, 320 verification sets and 44 test sets. Firstly, 2880 images of the training set and their corresponding label images are substituted into the training network to obtain the optimal weight.FIG.3AandFIG.3Bshow the change process of Loss function over time in the training set and verification set in 100 training cycles, when the learning rate is 1E-4, 1E-5 and the experimental design scheme.(2) 320 verification sets were substituted into the neural network to verify the performance of the model, and 44 test sets were finally substituted into the neural network. The present invention introduces an evaluation index to evaluate the accuracy of network model prediction and recognition results.FIG.4shows the changes of the average union ratio of various indicators in the validation set under the optimal learning rate adjustment scheme, including M-IOU, Recall, F1-Score and Accuracy ACC.
Step 6. Extraction of Particle Characterization Parameters:(1) Invent the use of open source OpenCV computer vision library for feature extraction of segmented images. Firstly, gaussian filter is used to eliminate some small noises in the early segmentation process, and then the image is grayscale and binarization to facilitate subsequent image processing operations.(2) After the binarization operation is completed, the size of the kernel is controlled, and the central cavity of some particles in the image is filled by the image processing method of expansion before corrosion. Finally, the edge detection is carried out by watershed algorithm to separate the regenerated sand particles in contact with each other in the image.(3) After the image processing of the segmentation result graph is completed, the contour extraction function FindContours( ) in the OpenCV library is used to extract the edge contour of the separated RCSP. Finally, the geometric morphology parameters of RCSP are extracted by different algorithms. Table 4 is the characterization parameters of recycled concrete sand particles determined in the experiment, Table 5 shows geometric parameters and corresponding algorithm implementation.

TABLE 4Characterization parameters of recycled concrete sandAlgorithmParameter nameSymbolimplementationLength-diameter ratioHmin/Wmin/CircularityRR = 4πS/p2ConvexityC0C0= S/ShVolumeVV = πTWminHmin/6Degree of sphericitySPSP= Ds/Dp(4) After completing the training of the segmentation network model of recycled concrete sand, different batches of RCSP were segmented and extracted. Finally, geometric morphology parameters and characterization parameters were extracted from the segmented images of RCSP. Table 6 shows the pixel values of geometric morphology parameters of extracted part of RCSP, and Table 7 shows the pixel values of characterization parameters of corresponding RCSP.(5) The parameters extracted in the experiment are all in pixels. In the final image processing process, the calibration results of the previous binocular camera are used to obtain the proportion relationship between pixel size and actual size through the known chess grid size, and finally achieve the conversion of pixel value to the actual size. Table 8 shows the actual size of geometric parameters and characterization parameters of RCSP after scaling.

TABLE 5Geometric parameters and algorithm realizationof recycled concrete sandParameter nameSymbolAlgorithm implementationAreaSCV.ContourAreaPerimeterPCV.ArclengthMinimum width of theWminDrawCounTour is used to draw theenclosing rectangleminimum outer rectangle graph firstand then calculateMinimum height ofHmin/the enclosing rectangleMaximum width of theWmaxCV.BoundingRectouter rectangleMaximum height ofHmaxCV.BoundingRectthe outer rectangleMinimum area of theCminCV.MinEnclosingCirclecircumscribed circleEllipse fitting areaECV.FitEllipseConvex hull areaShCV.ConvexHullEqual area circleDS/diameterIsoperimetric circleDP/diameterThicknessT3D coordinate difference calculationConstant volume ballSb/diameter

TABLE 6Pixel values of geometric morphology parameters of RCSPNumberSPWminHminWmaxHmaxCminEShDSDpTSb1305.573.214.027.015.028.0592.7343.3320.019.723.310.07.82409.087.620.032.019.034.0874.5441.0438.022.827.919.611.63274.067.518.021.019.022.0547.2309.5285.518.721.510.37.94246.070.318.024.018.025.0565.2347.1267.017.722.440.012.95563.095.317.036.028.031.0879.1606.9584.526.830.312.79.96497.595.51.042.030.030.01069.2534.7525.525.230.417.44.57258.562.915.022.016.023.0408.2259.2266.018.120.018.99.28439.086.418.031.023.030.0773.2458.2458.523.627.528.712.69351.573.719.024.020.025.0531.8370.3365.521.223.526.311.410895.0120.433.036.034.037.01352.7990.3941.033.838.34.78.9. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .. . .464383.076.813.030.024.024.0524.4401.2403.522.124.422.410.3The average498.888.418.331.423.830.3849.4536.7521.524.628.124.910.9

TABLE 7Pixel values of characterization parameters of RCSPNumberWmin/HminT/HminRCoVSp10.540.360.720.891986.440.8520.560.580.670.936564.020.8230.860.470.760.892034.430.8740.721.600.630.719039.300.7950.900.410.780.934067.070.8861.000.580.690.93382.120.8370.700.820.821.003271.130.9180.770.960.740.968371.940.8690.801.050.810.956265.450.90100.920.130.780.902931.860.88. . .. . .. . .. . .. . .. . .. . .4641.000.930.820.954562.720.90The average0.800.900.770.936560.110.88

TABLE 8Conversion between pixel size and actual size of RCSPThe parameterThe average of pixelThe actual numericalS/mm2305.54.11P/mm73.28.78Wmin/mm14.01.68Hmin/mm27.03.24Wmax/mm15.01.80Hmax/mm28.03.36Cmin/mm2592.77.71E/mm2343.34.46Sh/mm2320.04.16DS/mm19.72.36Dp/mm23.32.80T/mm10.01.20Sb/mm7.80.94V/mm34562.727.08

The deep learning-based recycled concrete sand image recognition and particle characterization extraction method of the present invention can be widely used in specific engineering projects. In the scheme, a binocular camera is used to obtain multi-dimensional recycled concrete sand particle image data. Training U-Net semantic segmentation network model combined with dynamic image processing method can realize segmentation of recycled concrete sand particle image and quick extraction of morphology characterization parameters.

The above are preferred embodiments of the present invention, which does not limit the patent scope of the present invention. Any equivalent structure or process transformation made by using the description of the present invention and the attached drawings, or directly or indirectly applied in the relevant technical field, is also included in the patent protection scope of the present invention.