> GPs usually do not scale but there are some interesting works that use exotic kernels or approximations that make them useable with large data.

That's interesting! 

> Simple GPs work very well with few number of samples and high dimensions (curse of dimensionality), that is at least what I found them to be most useful.

Not everybody has high n, low p data, indeed. I just want to avoid
something that is slow if p or n are high.