01 HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation• CVPR 2021
• Contribution
Scene understanding plays a crucial role in semantic segmentation,Providing the network with additional adaptivity by hypernet.

01 HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation• Meta-learning technique(Referred to as dynamic networks or hypernet)- learn ‘how to learn a new knowledge’?（学会学习）• Conv：
conv

Imagenn.Conv2d(in_channels, out_channels, kernel_size)

Feature map• Hyper Conv:

F.conv2d(x, w, b, stride=, dilation=, padding=,groups=)
Xwbconv

ImageFeature map𝜽controller→→→feature map(batchsize, channel, h, w)
weight of conv(out_channel, in_channel, kh, kw)
bias of conv(out_channel)

01 HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation• Meta-learning technique(Referred to as dynamic networks or hypernet)• Hyper Conv:

Few channels feature map

Feature inconv

Feature map1

Feature map2
𝜽controller(Shape)

Batchsize×Channel × 1 × 1
Channel = out_channel × in_channel × kh × kw

Positional embedding

Positional embedding:(5x5)tensor([[[[-1.0000, -0.5000, 0.0000, 0.5000, 1.0000],[-1.0000, -0.5000, 0.0000, 0.5000, 1.0000],[-1.0000, -0.5000, 0.0000, 0.5000, 1.0000],[-1.0000, -0.5000, 0.0000, 0.5000, 1.0000],[-1.0000, -0.5000, 0.0000, 0.5000, 1.0000]],[[-1.0000, -1.0000, -1.0000, -1.0000, -1.0000],[-0.5000, -0.5000, -0.5000, -0.5000, -0.5000],[ 0.0000, 0.0000, 0.0000, 0.0000, 0.0000],[ 0.5000, 0.5000, 0.5000, 0.5000, 0.5000],[ 1.0000, 1.0000, 1.0000, 1.0000, 1.0000]]]])

01 HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation• Overall network architecture• Encoder(hypernetwork): EfficientNet/ ResNet18/ PSPNet50• Context head(weight mapper)• Decoder(meta block)01 HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation• Backbone（EfficientNet / ResNet 18 / PSPNet50）• Stride1 ->RGB map• Stride2、Stride4、Stride8、Stride16 -> 1×1conv -> feature map• Stride32 -> weight• Position embedding,01 HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation• Context head(stride = 32)• Employ 2×2 convolutions with a stride of 2
• Computationally cheaper than 3 × 3 convolutions• Padding for the low-resolution feature maps significantly increase the spatial resolution

stride = 32

stride = 64

stride = 128

Signal feature

01 HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation• Meta block• point-wise convolution(pw) + depth-wise convolution(dw) + point-wise convolution(pw)——from MobileNet v2

Channel = out_channel × in_channel × kh × kw

01 HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation• Weight mapper• Employing the full signal in each mi is inefficient, because φ (Signal feature)is
directly mapped into a large number of weights.

Signal feature channel: 1280
Weight_num for each stage: [7298, 4070, 760, 270, 156]

01 HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation• Weight mapper• divide the channels of φ into parts, Cφ0 , . . . , Cφn, which are relative in size
to the required number of weights of each meta.
• the number of groups gwi , controls the amount of computations and trainable
parameters invested in producing the weights for mi.

Weight_num for each stage: [7298, 4070, 760, 270, 156]
Feature channel: 1280

[ 736400641664]

01 HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation• Dynamic patch-wise convolution• Each color represents weights corresponding to a specific patch and ’*’ is theconvolution operation.
conv

Feature map1

Feature map2
𝜽controller

beforeBatchsize×Channel × 1 × 1
Channel = out_channel × in_channel × kh × kw

afterBatchsize×Channel × ph x pw
Channel = out_channel × in_channel × kh × kw

01 HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation• Dynamic patch-wise convolution

Dynamic Neural Representational Decoders for High-Resolution Semantic Segmentation

01 HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation• Result-resolution

01 HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation• Result-backbone/ gride size