@jeffdonahue why don't you use `caffe_set` again? Also I don't think you need to initialize `top_mask` since later you are going to copy `max_idx_` over it.