karpathy/convnetjs — pull request #3 by ajtulloch (2014-04-19 09:50:03+00:00)
Title: [ConvNetJS] Implement DropConnect for regularizing large fully-connected layers

# Summary
This diff implements DropConnect, following the paper in
http://cs.nyu.edu/~wanli/dropc/dropc.pdf.

"When training with Dropout, a randomly selected subset of activations
are set to zero within each layer. DropConnect instead sets a randomly
selected subset of *weights* within the network to zero. Each unit thus
receives input from a random subset of units in the previous layer."

DropConnect has been shown to achieve significant improvements in model
performance over DropOut.

## Test Plan
I'm not sure on how to proceed here - do you want me to add a demo
demonstrating DropOut vs DropConnect on an otherwise identical
dataset/architecture?

I'd like to add unit tests but it's unclear how to add them into this
project.

I verified that existing demonstrations (MNIST, autoencoder, etc) work,
and that adding a drop-connect layer allows training to continue as expected.