Do we really need per-unit connectivity masks? It seems like this could be quite a large data structure. For example, if you had a convolutional layer, the size of this mask would be (input image size * hidden layer image size), i.e. it scales quadratically in input image size, which is what a convolutional layer is meant to avoid.