YOLOv2 Object Detection from ONNX Model in MATLAB | by Neha Goel | Towards Data Science

RoboSub is a competition run by RoboNation where students build an autonomous underwater vehicle to perform simulated search & rescue tasks. The basic task is to identify and avoid the Autonomous Underwater Vehicle (AUV) through the objects under water.
The image data used for this story is provided by Robotics Association at Embry-Riddle’s RoboSub Team. The images are captured during their practice runs with a live camera mounted on their AUV. The complete dataset can be downloaded from this google drive.
All the images are resized to (416 x 416 x 3) and divided into three folders of train, test and validation. Images are then labelled using the custom automation algorithms in the Ground Truth Labeler app in MATLAB. To learn more about the complete labeling process please refer to this YouTube video.
Data from the Ground Truth Labeler app is exported into MATLAB in the form of groundTruth data object. It is then further converted into tables using the function objectDetectorTrainingData. The detailed code file is located here.
Note: Download the data and run this script of dataPreProcessing before proceeding to next steps.
This section will go through steps for importing the ONNX model and then perform transfer learning based on the classes of the data set.
Step 1 : Import the Tiny YOLOv2 onnx model from the ONNX Model zoo.
The file gets downloaded and stored as model.onnx in the folder.
modelFile = fullfile('Utilities','model.onnx');
importONNXNetwork function imports the pre-trained network from onnx. As the function arguments, I passed the modelFile and the ‘OutputLayerType”
net = importONNXNetwork(modelFile,'OutputLayerType','regression');
tiny YOLOv2 model imported was a network of total 34 layers, combining of : input image, 8 sets of convolution, batchnorm, activation and pooling layers with the final layer as convolution8 layer. I visualized the network architecture using the analyzeNetwork function.
analyzeNetwork(net);
Step 2: Calculate the number of classes to be detected and anchor boxes depending on the training data.
In this project, I had 4 classes/objects: nav gate, green buoy, red buoy and yellow buoy.
numClasses = size(trainingData,2)-1;
Anchor boxes are calculated using the clustering method. The detailed process followed is linked here.
Anchors = [43 59    18 22    23 29    84 109];
Step 3: Redesign and Assemble the YOLOv2 network.
I then redesigned the imported network according to the classes to be detected. I implemented it using the function yolov2Layers. It adds a YOLOv2 network at the end of the extracted layers.
The parameters given to the function are :
Input size: Refers to the minimum input size to be given to the network. Here i gave (128 x 128 x 3).Number of Classes (numClasses) and Anchors calculated in Step 2 aboveImported network (net) from Step 1 aboveFeature extraction layer: The features extracted from this layer is given as input to the YOLOv2 object detection sub-network. You can specify any network layer except the fully connected layer. Here I used activation4 named layer of the imported network net.
Input size: Refers to the minimum input size to be given to the network. Here i gave (128 x 128 x 3).
Number of Classes (numClasses) and Anchors calculated in Step 2 above
Imported network (net) from Step 1 above
Feature extraction layer: The features extracted from this layer is given as input to the YOLOv2 object detection sub-network. You can specify any network layer except the fully connected layer. Here I used activation4 named layer of the imported network net.
lgraph = yolov2Layers([128 128 3], numClasses, Anchors, net, 'activation4');
Below image shows how the new network looks like. Observe the highlighted sub network of yolov2layers added at the end after the feature layer activation4.
analyzeNetwork(lgraph);
Step 4: Train the new network.
Based on the size of the data set I trained the network with the solver — stochastic gradient descent for 100 epochs with an initial learning rate of 0.001 and mini-batch size of 10. I performed lower learning rate to give more time for training considering the size of data and adjusting the epoch and mini-batch size. Documentation which I used to select the options is: trainingOptions.
options = trainingOptions('sgdm', ..."LearnRateSchedule","piecewise", ...'LearnRateDropFactor',0.5,..."LearnRateDropPeriod",5,...'Verbose' ,true, 'MiniBatchSize',10,'MaxEpochs',100, ...'Shuffle','every-epoch', 'VerboseFrequency',50, ...'DispatchInBackgrpund',true,...'ExecutionEnvironment','auto');
I then used trainYOLOv2ObjectDetector function to train my network.
[detectorTinyYolo2, info] = trainYOLOv2ObjectDetector(trainingData, lgraph,options);
Once I had the trained network, I then evaluated its performance on the Test Data as per the below steps.
Step 1: Create a table to hold the results and initialize a Deployable Video Player to view the image stream.
results = table('Size',[height(TestData) 3],...'VariableTypes',{'cell','cell','cell'},...'VariableNames',{'Boxes','Scores', 'Labels'});depVideoPlayer = vision.DeployableVideoPlayer;
Step 2: Loop through all the images in the Test set.
for i = 1:height(TestData)    % Read the image    I = imread(TestData.imageFilename{i});        % Run the detector.    [bboxes,scores,labels] = detect(detectorTinyYolo2,I);    if ~isempty(bboxes)         I = insertObjectAnnotation(I,'Rectangle',bboxes,cellstr(labels))                         depVideoPlayer(I);         pause(0.1);    else         depVideoPlayer(I);    end    % Collect the results in the results table    results.Boxes{i} = floor(bboxes);    results.Scores{i} = scores;    results.Labels{i} = labels;endthreshold =0.3;expectedResults = TestData;
Step 3: Evaluate the precision metrics.
I used evaluateDetectionPrecision function to calculate the data points for plotting the precision-recall curve using the given input arguments and the threshold value.
[ap, recall, precision] = evaluateDetectionPrecision(results, expectedResults(:,2:end),threshold);
Plot the precision metrics for each class.
plot(recall{1,1},precision{1,1},'g-','LineWidth',2, "DisplayName",'greenBuoy');hold on;plot(recall{2,1},precision{2,1},'b-','LineWidth',2, "DisplayName",'navGate');hold on;plot(recall{3,1},precision{3,1},'r-','LineWidth',2, "DisplayName",'redBuoy');hold on;plot(recall{4,1},precision{4,1},'y-','LineWidth',2, "DisplayName",'yellowBuoy');hold off;xlabel('Recall');ylabel('Precision');title(sprintf('Average Precision = %.2f\n', ap)legend('Location', 'best');legend('boxoff');
Step 4: Evaluate miss rate metric for object detection.
Using the function evaluateDetectionMissRate I calculated the log average -miss rate of the results, compared to the groundTruthData and data points.
[am,fppi,missRate] = evaluateDetectionMissRate(results, expectedResults(:,2:end),threshold);
Plot the log miss rate metrics for each class to false positives per image.
loglog(fppi{1,1}, missRate{1,1},'-g','LineWidth',2, "DisplayName",'greenBuoy');hold on;loglog(fppi{2,1}, missRate{2,1}, 'b','LineWidth',2,"DisplayName",'navGate');hold on;loglog(fppi{3,1}, missRate{3,1},'-r','LineWidth',2, "DisplayName",'redBuoy');hold on;loglog(fppi{4,1}, missRate{4,1},'-y','LineWidth',2, "DisplayName",'yellowBuoy');hold off;xlabel('False Positives Per Image');ylabel('Log Average Miss Rate');title(sprintf('Log Average Miss Rate = %.2f\n', am))legend('Location', 'best');legend('boxoff');
As observed due to the imbalanced classes the metrics are more accurate for the “navGate” class. One can solve the unbalancing of the classes using oversampling and under-sampling techniques.
This story’s main purpose was to walk through an approach of importing onnx models and retraining on customized data in MATLAB. You can watch the video of this story here.
As a part of next step I deployed this network on a NVIDIA Jetson and detected objects on a live camera. To learn more about the entire project you can check this GitHub repository : https://github.com/mathworks-robotics/deep-learning-for-object-detection-yolov2.