Patent ID: 12223702

Symbols in the figures:1—branch module,1.1—shared base network,1.2—private base network,1.3—detection module,2—result merging module,3.1—network block,3.2—optional network block,4.1—first branch,4.2—second branch,4.3—third branch,5—embedded device,5.1—target detection logic,5.2—local service logic,5.3—sample collection logic,6—server,6.1—sample annotation module,6.2—model correction module,7—sample library,8—network model parameter,9—faster-RCNN network, and10—SSD network10.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The disclosure will be further described in the following with reference to the specific embodiments. It should be understood that these embodiments are only used to illustrate the disclosure and not to limit the scope of the disclosure. In addition, it should be understood that those skilled in the art can make various changes or modifications to the disclosure after reading the content taught by the disclosure, and these equivalent forms also fall within the scope defined by the appended claims of the disclosure.

Please refer toFIG.1, a target detection system suitable for an embedded devices in the disclosure comprises an embedded device5and a server6. A remote service logic is running on the server6; a target detection logic5.1and a local service logic5.2operate on the embedded device5. The target detection logic5.1comprises a deep learning network model.

The target detection system suitable for an embedded device in the disclosure further comprises an online model self-calibration system, which may be used to solve the problem of reduced learning capability because the small model reduces the number of parameters in order to reduce the amount of calculation. The online self-calibration system comprises a sample collection logic5.3running on the embedded device5, and a sample annotation module6.1and a model correction module6.2running on the server6; On the embedded device5, all the actually collected images enter the target detection logic5.1, and the detection results of the target detection logic5.1are sent to the local service logic5.2and the sample collection logic5.3, respectively. The local service logic5.2completes service-related logic, and the sample collection logic5.3serves as a part of the online self-calibration system, where samples are collected in a controlled manner and added into a sample library7in preparation for subsequent calibration.

The samples in the sample library7may be transmitted to the server6by various means such as Bluetooth, Wi-Fi, etc.

After the sample library7is uploaded to the server6, the duplicate images are deleted by calculating the similarity between the images, and enters the sample annotation module6.1. The annotated samples are used as training set and test set, and enter the model correction module6.2to train new target detection network model parameters8, and then the updated network model parameters8are deployed in the embedded device5.

Please refer toFIG.2, the deep learning network model in the target detection logic consists of a multi-layer structure comprising a plurality of branch modules1, and a result merging module2. The network consists of several branch modules1: M1, M2 . . . Mx. Each branch module1corresponds to one or more anchors. For example, the following design may be provided: (1) the number of branch modules is 2, namely M1, M2; (2) M1 corresponds to an anchor size of 16×16; (3) M2 corresponds to two anchor sizes (32×32, 64×56), and finally the model can detect objects approximately of the set anchor sizes.

Each branch module1is composed of three major components: a shared base network1.1, a private base network1.2, and a detection module1.3.

1) Shared base network1.1is formed by stacking of MobileNet network blocks. MobileNet is a network structure suitable for mobile devices, which greatly reduces the amount of calculation and the number of parameters compared with CNN, while having the “scaling” feature of CNN at the same time. Here, the design of the shared base network1.1(backbone_1) of the first layer is different from that of the shared base networks1.1of other layers: the first layer of network uses CNN in order to prevent MobileNet from losing too many features.

The function of the shared base network1.1is mainly to determine the scaling ratio of the branch module through a stride. Taking the design of backbone_1 as an example, the stride is multiplied to 8, that is, the feature map obtained by the branch module is ⅛ of the original image in size. When the detected object is relatively large, a larger stride may be used, which can quickly diminish the size of the feature map and reduce the number of parameters and the amount of calculation.

The shared base networks1.1of the shallower layers share parameters with the shared base networks1.1of the deeper layers, reducing overall parameters and the amount of calculation of the network. For example, the output of backbone_1 becomes the input of backbone_2, and the output of backbone_2 becomes the input of backbone_3, and so on.

2) The private base network1.2is also formed by stacking of MobileNets. Unlike the shared base network1.1, the parameters of the private base network1.2are only valid for the current module and are not affected by other modules.

The private base network1.2can also be increased or decreased based on the actual detection effect. When the expressiveness is too poor, the network layers can be appropriately increased to improve the expressiveness; when the expressiveness is acceptable, the network can be appropriately reduced to increase the speed.

3) The detection module1.3improves the detection effect of the model by fusing the feature maps of different receptive fields.

The result merging module2of the target detection logic gathers the detection boxes predicted by all branch modules, and removes the redundant detection boxes through NMS to obtain the final prediction result.

Please refer toFIG.3. The shared base network is formed by stacking a plurality of network blocks3.1, in which the convolution corresponding to the dotted box is an optional network block3.2. The optional network block3.2can be increased or decreased depending on the difficulty of detecting the detected object. If the detected object is difficult to detect, or there are many false detections, the optional network blocks3.2can be added; otherwise, remove the optional network blocks3.2.

Please refer toFIG.4. The input feature map enters from the input end of the detection module, with information of C dimensions. After entering the module, the feature map will be divided into the first branch4.1, the second branch4.2, and the third branch4.3. For the feature map on the second branch4.2, after passing through two MobileNet modules, the number of dimensions of the feature map is increased from C to2C. The receptive field of the second branch4.2is between the upper and lower branches, and the number of its dimensions is increased to make it the main feature information. The features of the first branch4.1and the third branch4.3serve as auxiliary information. Finally, the information of the three branches is joined together to form a new feature map. The new feature map is obtained by passing through different 1×1 convolutions respectively to obtain scores and detection boxes. An additional 1×1 convolution is added to obtain a key point, if the key point is required.

Please refer toFIG.5. The sample collection logic running on the embedded device is triggered and activated by a customized condition. For example, it can be triggered at regular intervals where the sample collection logic may be activated once an hour, or it may be triggered by service, for example, when the device is performing face entry, if an image of “no object detected” is shown, there is high probability of missed detection, then the sample collection logic is activated. The workflow of the sample collection logic includes the following steps:Step501) the sample collection logic is triggered.Step502) sending the detection result of each frame to the “detection result queue”, and calculating a number Z of consecutively failed frames, which specifically includes:Step502.1) starting with the last time of object detected;Step502.2) recording the number of frames where no object is detected;Step502.3) ending with the next time of object detected, and counting the total number of frames where no object is detected.Step503) setting threshold Zthreshold, when Z is greater than Zthreshold, it is judged that there is no object in the Z frames of images, the sample collection logic ends; when Z is less than Zthreshold, it is judged that object detection is missed in the Z frames of images, go to step504.Step504) extracting 1 frame from the Z frames missed in the detection.Step505) save this frame of image into the sample library, and the sample collection logic ends.

Here, the size of the sample library will be limited, and when the limit is exceeded, new samples will replace the oldest samples. It is ensured that it does not take up too much storage resource and ensure the freshness of the sample data, which can better reflect the recent environmental conditions.

Please refer toFIG.6. The sample annotation module running on the server performs automatic or manual annotation on each frame of collected image in the sample library, with the specific following steps:Step601) each frame of image in the sample library enters the sample annotation module;Step602) the image samples are sent into a plurality of super-large networks, such as YOLO, SSD, Faster-RCNN, etc.Step603) obtaining the results L1, L2to LXrespectively.Step604) combining the results of the plurality of super-large networks (L1, L2to LX), and calculating the image difficulty factor λ.Step605) if the difficulty factor λ is less than or equal to a difficulty threshold λthreshold, go to step606; if the difficulty factor λ is greater than the difficulty threshold λthreshold, go to step608.Step606) completing automatic annotation of the images is by combining the target identification results of the plurality of super-large networks.Step607) classifying the image as a second-level difficult sample and adding it into the annotated sample library, go to step610.Step608) submitting the image for manual processing to complete manual annotation of the image.Step609) classifying the image as a first-level difficult sample and adding it into the annotated sample library.Step610) forming a dataset.

In this way, it is made possible to quickly collect difficult sample data sets, and at the same time ensure the correctness of sample annotation. In the end, the dataset contains both automatically annotated and manually annotated image samples.

Here, in step604, the specific process of calculating the sample difficulty factor is to group first, and then obtain the result based on the grouping information. Here, the steps of grouping include:Step701) obtaining the target identification results of each super-large network.Step702) selecting the target identification result of one of the super-large networks as the benchmark group (that is, each detection box is used as the benchmark detection box of a group), and classifying the target identification results of the remaining super-large networks as to be classified.Step703) selecting an super-large network to be classified, taking its target identification result, and calculating the IoU values between a plurality of detection boxes and the benchmark detection box thereof.Step704) selecting the detection box with a largest IoU value among the plurality of detection boxes to be classified. If the IoU value of this detection box is greater than the threshold Cthreshold, the current detection box is incorporated into the group where the benchmark detection box is located. The detection boxes that cannot be grouped are grouped individually.Step705) if there is still an unprocessed super-large network, go to step703. Otherwise, end the process.

Please refer toFIG.7for a specific grouping example. In this example, the result of the Faster-RCNN network9is used as the benchmark group. The IoU of the detection box 1 of the SSD network10and the detection boxes 1 to 5 of the Faster-RCNN network9are calculated, respectively. Finally, it is found that the IoU of the detection box 2 of the Faster-RCNN network9is the largest and greater than Cthreshold, so the detection box 1 of the SSD network10and the detection box 2 of the Faster-RCNN network9are grouped together, and so on. The detection box 5 of the SSD network10cannot be grouped, so it becomes an independent group.

After the grouping is completed, the number of detection boxes in each group is counted, annotating them as N1to Nk. The difficulty factor λ is calculated according to the following formula:

λ=∑i=1k❘"\[LeftBracketingBar]"Ni-N^❘"\[RightBracketingBar]"(N^-1)×∑i=1kNi,λ∈[0,1]in which {circumflex over (N)} is the number of super-large networks. TakingFIG.7as an example, it can be obtained that λ=0.1.

In step606, the specific process of automatic annotation of the image is to firstly discard the detection boxes of the independent groups, and then use the average value of the detection boxes of the non-independent groups as the final annotation of the image sample. The expression is as follows:

x^=∑i=1N^xiN^,y^=∑i=1N^yiN^,w^=∑i=1N^wiN^,h^=∑i=1N^hiN^in which {circumflex over (N)} is the number of super-large networks, and x, y, w, and h respectively represent the abscissa and ordinate of the upper left corner of the detection box, the width and height of the detection box.

Please refer toFIG.8. The original model is fine-tuned by utilizing the annotated samples, in order to fit the current environment. The dataset generated by the annotated samples is divided into an actual training set and an actual validation set, and the public dataset is used as the public validation set. The training data is in batch as the smallest unit.

The calibration process includes the following steps:Step801) preparing an original model (the model after the last correction, or the initial model if the correction is performed for the first time) and calculating the Loss value of the original model on the public validation set and the actual validation set, L0and I0.Step802) prepare a batch of actual training sets, and go to step803. If all the samples in the actual training set have been traversed, stop training, and jump to step806.Step803) starting training.Step804) after each batch of training, calculate the Loss value of the post-training model on the public validation set and the actual validation set, L and I.Step805) if L0−L>LThresholdand I0−I>IThreshold, it is regarded as a valid training, update the network parameters of the model, and jump to step801; otherwise, stop the iteration and go to step806.Step806) the calibration is completed, and new model network data is generated.On the embedded device, the first initial model may be built using open source datasets. Open source datasets usually cover a variety of scenarios with high richness. The model trained with such data can adapt to each scenario relatively evenly. This initial model will be deployed to the device first. During service operation, the embedded device utilizes the online model self-calibration system to update image samples to the server from time to time, and the model network parameters corrected by the online self-calibration system are sent back to the embedded device by the server through Bluetooth, Wi-Fi and other means to update the network parameters in the device.

The above described are only preferred embodiments of the disclosure, and are not intended to limit the scope of the disclosure. Any equivalent structure or equivalent process transformation made by utilizing the contents of the description and drawings of the disclosure, or directly or indirectly applied to other related technical fields are all similarly included in the scope of patent protection of the disclosure.