adding maxout activation neurons. They take a set of input neurons and max over groups of them. In a convnet, this happens only along depth preserving spatial dimensions x,y. The group size can be adjusted as a parameter. Feel free to try them out instead of sigmoid or relu! maxouts have very nice properties in that they are safe to train, compared to relus that can die, and sigmoids that can take very long to train.