BVLC/caffe — pull request #85 by kloudkl (2014-02-08 17:25:12+00:00)
Title: Replace atlas/cblas routines with Eigen in the math functions

This pull request meets the requirement of issue #84. Layerwise runtime analysis of imagenet.prototxt using @sguada's detailed net_speed_benchmark from #83 was a little disappointing for my work today. Running with the max number of physical cores (4 on my machine), Eigen is much slower than OpenBLAS and MKL. It requires expert knowledge of [how Eigen evaluates expressions](http://eigen.tuxfamily.org/dox/TopicWritingEfficientProductExpression.html) to fully exploit its high performance. In contrast, OpenBLAS is a very low-hanging fruit. You only need to install the multi-threaded package (#80, #81) and link it with your application (#82). Then everything works like a charm. MKL also does a good job but at a high price.

I would appreciate any other independent benchmark very much.

Eigen (boost-eigen branch head + this PR)
OMP_NUM_THREADS=4 ../build/examples/net_speed_benchmark.bin local_imagenet.prototxt 1 CPU
```
***	Benchmark	begins	***
data	forward:	0.02	seconds.
conv1	forward:	4.96	seconds.
relu1	forward:	0.22	seconds.
pool1	forward:	0.17	seconds.
norm1	forward:	1.21	seconds.
pad2	forward:	0.02	seconds.
conv2	forward:	8.05	seconds.
relu2	forward:	0.14	seconds.
pool2	forward:	0.11	seconds.
norm2	forward:	0.79	seconds.
pad3	forward:	0.01	seconds.
conv3	forward:	5.06	seconds.
relu3	forward:	0.06	seconds.
pad4	forward:	0.02	seconds.
conv4	forward:	4.57	seconds.
relu4	forward:	0.05	seconds.
pad5	forward:	0.02	seconds.
conv5	forward:	2.74	seconds.
relu5	forward:	0.03	seconds.
pool5	forward:	0.03	seconds.
fc6	forward:	0.85	seconds.
relu6	forward:	0.01	seconds.
drop6	forward:	0.01	seconds.
fc7	forward:	0.4	seconds.
relu7	forward:	0	seconds.
drop7	forward:	0.01	seconds.
fc8	forward:	0.14	seconds.
loss	forward:	0.01	seconds.
Forward	pass:	29.71	seconds.
loss	backward:	0	seconds.
fc8	backward:	0.18	seconds.
drop7	backward:	0.01	seconds.
relu7	backward:	0	seconds.
fc7	backward:	0.75	seconds.
drop6	backward:	0.01	seconds.
relu6	backward:	0	seconds.
fc6	backward:	2.54	seconds.
pool5	backward:	0.07	seconds.
relu5	backward:	0.01	seconds.
conv5	backward:	7.05	seconds.
pad5	backward:	0.02	seconds.
relu4	backward:	0.02	seconds.
conv4	backward:	8.31	seconds.
pad4	backward:	0.02	seconds.
relu3	backward:	0.02	seconds.
conv3	backward:	10.41	seconds.
pad3	backward:	0.01	seconds.
norm2	backward:	0.7	seconds.
pool2	backward:	0.28	seconds.
relu2	backward:	0.05	seconds.
conv2	backward:	16.63	seconds.
pad2	backward:	0.02	seconds.
norm1	backward:	1.06	seconds.
pool1	backward:	0.42	seconds.
relu1	backward:	0.08	seconds.
conv1	backward:	9.97	seconds.
data	backward:	0	seconds.
Backward	pass:	58.64	seconds.
Total	Time:	88.35	seconds.
***	Benchmark	ends	***
```

OpenBLAS (boost-eigen branch head & git cherry-pick 969d0ab6)
```
***	Benchmark	begins	***
data	forward:	0.08	seconds.
conv1	forward:	3.89	seconds.
relu1	forward:	0.44	seconds.
pool1	forward:	0.18	seconds.
norm1	forward:	1.22	seconds.
pad2	forward:	0.02	seconds.
conv2	forward:	4.94	seconds.
relu2	forward:	0.36	seconds.
pool2	forward:	0.11	seconds.
norm2	forward:	0.78	seconds.
pad3	forward:	0.01	seconds.
conv3	forward:	2.76	seconds.
relu3	forward:	0.19	seconds.
pad4	forward:	0.08	seconds.
conv4	forward:	2.3	seconds.
relu4	forward:	0.18	seconds.
pad5	forward:	0.08	seconds.
conv5	forward:	1.81	seconds.
relu5	forward:	0.12	seconds.
pool5	forward:	0.12	seconds.
fc6	forward:	0.47	seconds.
relu6	forward:	0.01	seconds.
drop6	forward:	0.04	seconds.
fc7	forward:	0.2	seconds.
relu7	forward:	0.01	seconds.
drop7	forward:	0.03	seconds.
fc8	forward:	0.07	seconds.
loss	forward:	0.01	seconds.
Forward	pass:	20.51	seconds.
loss	backward:	0	seconds.
fc8	backward:	0.11	seconds.
drop7	backward:	0.02	seconds.
relu7	backward:	0	seconds.
fc7	backward:	0.42	seconds.
drop6	backward:	0.03	seconds.
relu6	backward:	0	seconds.
fc6	backward:	0.92	seconds.
pool5	backward:	0.3	seconds.
relu5	backward:	0.02	seconds.
conv5	backward:	3.47	seconds.
pad5	backward:	0.07	seconds.
relu4	backward:	0.06	seconds.
conv4	backward:	4.68	seconds.
pad4	backward:	0.05	seconds.
relu3	backward:	0.06	seconds.
conv3	backward:	5.18	seconds.
pad3	backward:	0.04	seconds.
norm2	backward:	0.87	seconds.
pool2	backward:	0.27	seconds.
relu2	backward:	0.04	seconds.
conv2	backward:	9.84	seconds.
pad2	backward:	0.06	seconds.
norm1	backward:	1.24	seconds.
pool1	backward:	0.4	seconds.
relu1	backward:	0.06	seconds.
conv1	backward:	6.11	seconds.
data	backward:	0	seconds.
Backward	pass:	34.32	seconds.
Total	Time:	54.83	seconds.
***	Benchmark	ends	***
```

MKL (master head)
```
Initial	loss:	7.63095
***	Benchmark	begins	***
data	forward:	0.07	seconds.
conv1	forward:	3.32	seconds.
relu1	forward:	0.81	seconds.
pool1	forward:	0.2	seconds.
norm1	forward:	0.13	seconds.
pad2	forward:	0.08	seconds.
conv2	forward:	5.18	seconds.
relu2	forward:	0.51	seconds.
pool2	forward:	0.43	seconds.
norm2	forward:	0.18	seconds.
pad3	forward:	0.04	seconds.
conv3	forward:	2.5	seconds.
relu3	forward:	0.18	seconds.
pad4	forward:	0.09	seconds.
conv4	forward:	2.3	seconds.
relu4	forward:	0.19	seconds.
pad5	forward:	0.08	seconds.
conv5	forward:	2.25	seconds.
relu5	forward:	0.12	seconds.
pool5	forward:	0.1	seconds.
fc6	forward:	0.68	seconds.
relu6	forward:	0.01	seconds.
drop6	forward:	0.02	seconds.
fc7	forward:	0.22	seconds.
relu7	forward:	0.01	seconds.
drop7	forward:	0.02	seconds.
fc8	forward:	0.06	seconds.
loss	forward:	0	seconds.
Forward	pass:	19.78	seconds.
loss	backward:	0	seconds.
fc8	backward:	0.11	seconds.
drop7	backward:	0.01	seconds.
relu7	backward:	0.01	seconds.
fc7	backward:	0.43	seconds.
drop6	backward:	0.01	seconds.
relu6	backward:	0	seconds.
fc6	backward:	0.96	seconds.
pool5	backward:	0.25	seconds.
relu5	backward:	0.07	seconds.
conv5	backward:	3.31	seconds.
pad5	backward:	0.04	seconds.
relu4	backward:	0.07	seconds.
conv4	backward:	4.38	seconds.
pad4	backward:	0.05	seconds.
relu3	backward:	0.06	seconds.
conv3	backward:	4.87	seconds.
pad3	backward:	0.03	seconds.
norm2	backward:	0.23	seconds.
pool2	backward:	1.05	seconds.
relu2	backward:	0.04	seconds.
conv2	backward:	10.69	seconds.
pad2	backward:	0.06	seconds.
norm1	backward:	0.28	seconds.
pool1	backward:	1.59	seconds.
relu1	backward:	0.07	seconds.
conv1	backward:	5.92	seconds.
data	backward:	0	seconds.
Backward	pass:	34.6	seconds.
Total	Time:	54.38	seconds.
***	Benchmark	ends	***
```