Don't rely on the layers matching in the def and in the pretrained network.