5.6.6 ディープラーニング：点群データのサポート

# 目次へのリンク


[MATLABによる画像処理・コンピュータービジョン入門目次](https://qiita.com/tohruk/items/e5f7939b622bcf2e3c4c)


# 概要


ディープラーニングによる点群の分類(PointNet)を行います。 




実行には下記のToolboxが必要になります。 



   -  MATLAB R2020a 以降 
   -  Computer Vision Toolbox 
   -  Deep Learning Toolbox 
   -  Parallel Computing Toolbox(NVIDIA GPUでの学習・推論の高速化） 



対応ファイル



   -  [I5_06_6_pointNet.m](https://github.com/mathworks/MATLAB-IPCV-Eval-Kit-JP/blob/master/demo_files/I5_06_6_pointNet.m) 

# 必要なサポート関数へのパスを追加
```matlab:code
addpath(fullfile(matlabroot,'\examples\deeplearning_shared\main'));
```
# データのダウンロード
```matlab:code
datapath = downloadSydneyUrbanObjects;
```
```text:output
Downloading Sydney Urban Objects data set...
```
# 学習用と検証用にデータ分割
```matlab:code
foldsTrain = 1:3;
foldsVal = 4;
dsTrain = sydneyUrbanObjectsClassificationDatastore(datapath,foldsTrain);
dsVal = sydneyUrbanObjectsClassificationDatastore(datapath,foldsVal);
```
# データの一つを確認
```matlab:code
data = read(dsTrain);
ptCloud = data{1,1};
label = data{1,2};

figure
pcshow(ptCloud.Location,[0 0 1],"MarkerSize",40,"VerticalAxisDir","down")
xlabel("X");ylabel("Y");zlabel("Z");
title(label)
```


![figure_0.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/14336/a7bd4721-9ffd-92bb-50e2-5a8b3ae8244b.png)


# クラス毎のデータ数の分布を確認
```matlab:code
dsLabelCounts = transform(dsTrain,@(data){data{2} data{1}.Count});
labelCounts = readall(dsLabelCounts);
labels = vertcat(labelCounts{:,1});
counts = vertcat(labelCounts{:,2});

figure
histogram(labels)
```


![figure_1.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/14336/e190626f-79f4-b8c1-d828-0e95294cc78f.png)

# 頻度の少ないクラスのデータを複製
```matlab:code
rng(0)
[G,classes] = findgroups(labels);
numObservations = splitapply(@numel,labels,G);
desiredNumObservationsPerClass = max(numObservations);
files = splitapply(@(x){randReplicateFiles(x,desiredNumObservationsPerClass)},dsTrain.Files,G);
files = vertcat(files{:});
dsTrain.Files = files;
dsTrain.Files = dsTrain.Files(randperm(length(dsTrain.Files)));
```
# ミニバッチサイズを指定
```matlab:code
dsTrain.MiniBatchSize = 128;
dsVal.MiniBatchSize = 128;
```
# データの水増し（回転・点群の間引き・ノイズ追加）
```matlab:code
dsTrain = transform(dsTrain,@augmentPointCloud);
```
# 前処理
```matlab:code
% 固定数の点群を抽出
numPoints = 1024; % 1024点の抽出
dsTrain = transform(dsTrain,@(data)selectPoints(data,numPoints));
dsVal = transform(dsVal,@(data)selectPoints(data,numPoints));

% 点群データの正規化
dsTrain = transform(dsTrain,@preprocessPointCloud);
dsVal = transform(dsVal,@preprocessPointCloud);
```
# PointNetモデルの定義（モデルの初期化）


各ネットワークの層を定義し、初期化します。


```matlab:code
% 1.入力変換モデル
inputChannelSize = 3;
hiddenChannelSize1 = [64,128];
hiddenChannelSize2 = 256;
[parameters.InputTransform, state.InputTransform] = initializeTransform(inputChannelSize,hiddenChannelSize1,hiddenChannelSize2);

% 2.Shared MLPモデル
inputChannelSize = 3;
hiddenChannelSize = [64 64];
[parameters.SharedMLP1,state.SharedMLP1] = initializeSharedMLP(inputChannelSize,hiddenChannelSize);

% 3.特徴変換モデル
inputChannelSize = 64;
hiddenChannelSize1 = [64,128];
hiddenChannelSize2 = 256;
[parameters.FeatureTransform, state.FeatureTransform] = initializeTransform(inputChannelSize,hiddenChannelSize,hiddenChannelSize2);

% 4.Shared MLPモデル
inputChannelSize = 64;
hiddenChannelSize = 64;
[parameters.SharedMLP2,state.SharedMLP2] = initializeSharedMLP(inputChannelSize,hiddenChannelSize);

% 分類モデル
inputChannelSize = 64;
hiddenChannelSize = [512,256];
numClasses = numel(classes);
[parameters.ClassificationMLP, state.ClassificationMLP] = initializeClassificationMLP(inputChannelSize,hiddenChannelSize,numClasses);
```
# 学習オプションの設定
```matlab:code
numEpochs = 40;
learnRate = 0.001;
l2Regularization = 0.01;
learnRateDropPeriod = 15;
learnRateDropFactor = 0.5;

gradientDecayFactor = 0.9;
squaredGradientDecayFactor = 0.999;
```
# 学習の実行


カスタム学習ループを使って点群データの学習を行います。


```matlab:code
avgGradients = [];
avgSquaredGradients = [];

doTraining = false;% true:学習の実行　false:学習後のモデルを使用

if doTraining
    
    % 学習の進捗確認用のプロッター作成
    [lossPlotter, trainAccPlotter,valAccPlotter] = initializeTrainingProgressPlot;
    
    numClasses = numel(classes);
    iteration = 0;
    start = tic;
    for epoch = 1:numEpochs
        
        % データセットのリセット
        reset(dsTrain);
        reset(dsVal);
        
        while hasdata(dsTrain)
            iteration = iteration + 1;
            
            % データの読み込み
            data = read(dsTrain);
            
            % バッチデータの作成
            [XTrain,YTrain] = batchData(data);
            
            % 勾配と損失の計算
            [gradients, loss, state, acc] = dlfeval(@modelGradients,XTrain,YTrain,parameters,state);
            
            % L2正則化
            gradients = dlupdate(@(g,p) g + l2Regularization*p,gradients,parameters);
            
            % ネットワークの重みを更新
            [parameters, avgGradients, avgSquaredGradients] = adamupdate(parameters, gradients, ...
                avgGradients, avgSquaredGradients, iteration,...
                learnRate,gradientDecayFactor, squaredGradientDecayFactor);
            
            % プロッターを更新
            D = duration(0,0,toc(start),"Format","hh:mm:ss");
            title(lossPlotter.Parent,"Epoch: " + epoch + ", Elapsed: " + string(D))
            addpoints(lossPlotter,iteration,double(gather(extractdata(loss))))
            addpoints(trainAccPlotter,iteration,acc);
            drawnow
        end
        
        % 検証用データでモデルを評価
        cmat = sparse(numClasses,numClasses);
        while hasdata(dsVal)
            
            % 次のデータを読み込み
            data = read(dsVal);
            
            % バッチデータの作成.
            [XVal,YVal] = batchData(data);
            
            % クラスの推論
            isTraining = false;
            YPred = pointnetClassifier(XVal,parameters,state,isTraining);
            
            % スコアが一番高かったクラスを抽出.
            [~,YValLabel] = max(YVal,[],1);
            [~,YPredLabel] = max(YPred,[],1);
            
            % 混同行列の集計
            cmat = aggreateConfusionMetric(cmat,YValLabel,YPredLabel);
        end
        
        % 平均の分類精度をプロッターに追加
        acc = sum(diag(cmat))./sum(cmat,"all");
        addpoints(valAccPlotter,iteration,acc);
        
        % 学習率の更新
        if mod(epoch,learnRateDropPeriod) == 0
            learnRate = learnRate * learnRateDropFactor;
        end
        
        % データセットのリセット
        reset(dsTrain);
        reset(dsVal);
    end

else
    % 学習済みモデルのダウンロードと読み込み
    pretrainedURL = 'https://www.mathworks.com/supportfiles/vision/data/pretrainedPointNet.mat';
    
    pretrainedNetwork = fullfile(pwd,'pretrainedPointNet.mat');
    if ~exist(pretrainedNetwork,'file')
        disp('Downloading pretrained network (5 MB)...');
        websave(pretrainedNetwork,pretrainedURL);
    end
    
    pretrainedResults = load('pretrainedPointNet.mat');
    parameters = pretrainedResults.parameters;
    state = pretrainedResults.state;
    cmat = pretrainedResults.cmat;
    
    % GPUがある場合はgpuArrayに変換
    parameters = prepareForPrediction(parameters,@(x)dlarray(toDevice(x,canUseGPU)));
    state = prepareForPrediction(state,@(x)toDevice(x,canUseGPU));

end

% 混同行列の表示
figure
chart = confusionchart(cmat,classes);
```


![figure_2.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/14336/0a03fc6c-bfb4-f402-ee93-b5dde24a05ca.png)


```matlab:code

acc = sum(diag(cmat))./sum(cmat,"all")
```
```text:output
acc = 0.6000
```
# 学習したモデルで分類
```matlab:code
ptCloud = pcread("car.pcd");
X = preprocessPointCloud(ptCloud);
dlX = dlarray(X{1},"SCSB");

YPred = pointnetClassifier(dlX,parameters,state,false);
[~,classIdx] = max(YPred,[],1);

figure
pcshow(ptCloud.Location,[0 0 1],"MarkerSize",40,"VerticalAxisDir","down")
title(classes(classIdx))
```


![figure_3.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/14336/c3366fbc-a4ae-bdb2-5914-b6b21d1892ec.png)


# まとめ


点群データの分類にディープラーニングを活用した例を紹介しました。


# 参考

   -  [Point Cloud Classification Using PointNet Deep Learning](https://jp.mathworks.com/help/vision/examples/point-cloud-classification-using-pointnet-deep-learning.html) 


# 謝辞


本記事は @eigs さんの[livescript2markdown](https://github.com/minoue-xx/livescript2markdown)を使わせていただいてます。



# 目次へのリンク


[MATLABによる画像処理・コンピュータービジョン入門目次](https://qiita.com/tohruk/items/e5f7939b622bcf2e3c4c)